Multimodal RAG + GraphRAG over TikTok video content — real-time search and grounded intelligent generation. Vector retrieval guarantees generalizable memory; graph retrieval guarantees precise reasoning; the fused result provides the answer (Hybrid Intelligence).
Built in Python 3.12 with Milvus (vector), Neo4j (graph), and a queue-decoupled ingestion pipeline targeting 5-minute freshness (p95).
- Real-time multimodal search — natural-language queries over visual frames, ASR transcripts, captions, hashtags, and metadata in one index (CLIP-style unified space).
- Graph-powered exploration — multi-hop creator–topic–product–hashtag queries with
evidence paths (max depth 3,
LIMIT 50guardrails). - Grounded generation — answers, summaries, short-video script drafts, and trend forecasts, every claim cited to video + timestamp; refuses to fabricate when evidence is insufficient.
- Fast ingestion — queue-decoupled stages (crawl → dedup → transcode → ASR → chunk → embed → index), GPU-batched, with a politeness-aware crawler (full-jitter backoff, per-domain circuit breaker, daily volume cap — no evasion).
- Dual-write integrity — every chunk written to Milvus is synchronously materialized in Neo4j via a transactional outbox, with reconciliation (constitution T1).
- Public-grade serving — API keys + JWT, RBAC (viewer/analyst/administrator),
per-account token-bucket rate limiting (fail-open), per-request
trace_id, 30-day media retention, and an idempotent takedown/purge saga. - Quality gates — TDD (Red-Green-Refactor), >80% unit coverage, and an evaluation harness with hard red lines (context recall > 0.85, answer relevancy > 0.80, faithfulness > 0.90).
┌──────────────────────────────────────────────────────────────┐
│ INGESTION (queue-decoupled) │
│ crawl → dedup(phash) → transcode → ASR → chunk → embed → │
│ vector write (GPU-batched) │
└───────────────────────────┬──────────────────────────────────┘
│ outbox event (T1 dual-write)
┌───────────────────▼───────────────────┐
│ Milvus ┌───────────────┐ │
│ IVF_RABITQ/SQ8 │ Neo4j │ │
│ chunks │ Entity+Rel │ │
└─────────┬─────────┘ ▲ materializer┘
│ │ (LLM extraction + disambiguation)
┌─────────▼─────────────┴──────────┐
│ RETRIEVAL: vector ANN + graph │
│ paths → RRF fusion → rerank │
└─────────┬─────────────────────────┘
┌─────────▼─────────────────────────┐
│ GENERATION: grounded answers, │
│ scripts, forecasts + citations │
└─────────┬─────────────────────────┘
┌─────────▼─────────────────────────┐
│ SERVING: FastAPI · auth/RBAC · │
│ rate limit · trace · admin · │
│ takedown · retention │
└───────────────────────────────────┘
Code follows strict 4-layer isolation (Ingestion → Indexing → Retrieval → Generation);
cross-layer calls flow only through interface/service boundaries. See
.specify/memory/constitution.md for the governing
principles.
| Layer | Choice |
|---|---|
| Language | Python 3.12 (uv-managed, committed uv.lock) |
| API | FastAPI + uvicorn |
| Vector store | Milvus ≥ 2.6, index IVF_RABITQ (refine_type=SQ8, COSINE) |
| Graph store | Neo4j ≥ 5.2 (async driver), vector index on Entity.embedding |
| Transactional store | PostgreSQL (SQLAlchemy async + Alembic) |
| Queue / rate limiting | Redis (arq workers, token-bucket Lua) |
| Object storage | S3-compatible (MinIO) |
| ASR | faster-whisper (GPU, batched, Silero VAD, zh/en) |
| Embedding | CLIP-style dual encoder (vision + text, unified space) |
| Eval | RAGAS / DeepEval harness |
Full run/validation guide: specs/001-tiktok-video-rag/quickstart.md.
uv sync --locked # install from uv.lock
cp .env.example .env # fill in URIs/credentials
docker compose -f docker-compose.dev.yml up -d # Milvus 2.6+, Neo4j, Postgres, Redis, MinIO
uv run alembic upgrade head # create schema
uv run python -m src.pipelines.bootstrap # Milvus collection + Neo4j indexes
uv run python -m src.serving.auth.seed # roles + admin + API key
uv run uvicorn serving.app:app --port 8000 # API
uv run arq src.pipelines.ingest.workers.WorkerSettings # ingestion workersRequires the ml extra (uv sync --extra ml) and a GPU node before the embedder/ASR run
in production.
A minimal React + Vite SPA under frontend/ covers login (seeded admin), uploading a
video by file or URL, and searching with playable result cards (thumbnail + inline
video). The Vite dev server proxies /v1 to the API, so no CORS config is needed in dev.
uv sync --extra ml # real CLIP embedding for search
uv run python -m src.serving.auth.seed # creates admin@example.com / changeme-now
uv run uvicorn src.serving.app:app --port 8000
cd frontend && npm install && npm run dev # http://localhost:5173Notes:
- Sign in with the seeded admin (
admin@example.com/changeme-now). - File uploads are stored in MinIO and ingested through the real pipeline; URL-ingested
videos are persisted the same way, so both are playable via
GET /v1/media/{key}. milvus_vector_dimis512(matches the real CLIP base embedder); the collection istiktok_chunk_512. The E2E suite still runs with the fake ML tier behindRUN_E2E=1.
src/
├── core/ # Pydantic v2 entities, ABCs, tracing, config, DB layer
├── ingestion/ # crawler (politeness), dedup, transcoder, ASR, chunkers, moderation
├── indexing/ # Milvus + Neo4j clients, CLIP embedder, graph extractor/materializer
├── retrieval/ # vector ANN, graph paths, RRF fusion, cross-encoder rerank
├── generation/ # grounded generator, citations, script drafts, forecasts
├── serving/ # FastAPI: auth/RBAC, rate-limit, routes (ingest/search/graph/generate/…)
└── pipelines/ # ingest workers + outbox, reconcile, retention, takedown, community
tests/
├── unit/ # TDD unit tests (coverage > 80%)
├── contract/ # API contract + auth + rate-limit tests
├── integration/ # Milvus + Neo4j (RUN_INTEGRATION=1)
├── eval/ # RAGAS/DeepEval harness + eval_dataset.json
└── load/ # SC-008/009 load + availability (RUN_LOAD=1)
- Quantization is mandatory. Uncompressed Float32 indexes are forbidden in production;
IVF_RABITQwith an SQ8 refiner satisfies both recall and the no-FP32 rule. - Outbox dual-write. Vector + graph stores cannot share one ACID transaction, so a transactional outbox makes graph materialization a mandatory consumer of every chunk — compensation is re-materialization, never deletion.
- Recall-first, rerank-second. The recall stage stays simple (ANN + templated paths) and noise is removed by fusion + rerank, keeping recall-stage latency flat.
- Rate limiter fails open. A limiter outage must never take the API down (99.9% availability target); per-account isolation means one abuser can't degrade others.
- No crawler evasion. Blocks are handled with backoff, circuit breakers, and graceful freshness degradation — no IP/proxy rotation (ToS/legal posture).
uv run pytest # 84 unit/contract tests, 6 infra-guarded
uv run coverage run -m pytest # coverage report (gate: > 80%)
uv run ruff check src tests
uv run mypy src --strict
uv run pytest tests/integration # with RUN_INTEGRATION=1 + live services
uv run pytest tests/load # with RUN_LOAD=1 + running service
RUN_E2E=1 uv run pytest tests/e2e # end-to-end: real video → ingestion → retrieval
# (Tier 1 fake ML; add RUN_GPU=1 for real ASR+CLIP)The end-to-end suite (tests/e2e/) generates a deterministic test video, drives it through
the real ingestion pipeline, and verifies retrieval returns the ingested video with
timestamps and a persisted trace trail. Tier 1 uses deterministic fake ASR/embedder (runs
in any CI); Tier 2 (RUN_GPU=1) uses real faster-whisper + CLIP on a GPU runner. See
openspec/changes/e2e-ingestion-retrieval/ for the design.
The ingestion + search slice is production-shaped; the items below are the known open seams.
| Area | Status / gap |
|---|---|
| Real data source | Crawler is a polite downloader; needs a ToS-validated TikTok feed/webhook + compliance sign-off before production (FR-009) |
| Graph fusion end-to-end | RRF exists; search still returns vector-only results — fuse graph paths + rerank into /v1/search to unlock the Hybrid Intelligence payoff (constitution T2) |
| Live infra validation | RUN_INTEGRATION / RUN_LOAD suites + quickstart V0–V7 need running Milvus/Neo4j/GPU to prove SC-001/002/006/008/009 |
| Text2Cypher (G3) | Templated Cypher only; natural-language → Cypher with max_depth=3 + LIMIT 50 guardrails is designed, not built |
| Streaming generation | /v1/generate is non-streaming; constitution SLA calls for first streamed token ≤ 2.5 s E2E |
| Evaluation quality | tests/eval/eval_dataset.json is a synthetic 100-row scaffold — needs real labeled queries before red-lines are trustworthy |
| ML runtime | uv sync --extra ml (torch/transformers) + GPU for CLIP embedder / faster-whisper |
| Admin & MFA | RBAC + admin endpoints exist; mfa_enabled is a field, TOTP + admin UI are not |
| Observability | trace_id threading + freshness alert exist; no metrics export or alerting channel |
| Reconciliation depth | Outbox path is real; full Milvus↔Neo4j diff is a placeholder (T1 backstop beyond happy path) |
| Moderation depth | Keyword rules only; FR-011 needs richer policy config |
| Milvus bulk backfill | Streaming upserts implemented; bulk-import path for a seed corpus is not |
| Multi-language | zh/en only (FR-014); expansion path designed |
| License | Apache 2.0 (see LICENSE) |
- Specification:
specs/001-tiktok-video-rag/spec.md - Plan & research:
plan.md·research.md - Data model:
data-model.md - Contracts:
contracts/api.md - Constitution:
.specify/memory/constitution.md
Content acquisition via public crawling is subject to platform Terms of Service and copyright review. This system is intended to be operated on a licensed/owned corpus; validate compliance before production deployment.