Skip to content

About

Distributed microservice-based RAG platform for multimodal ingestion, semantic retrieval, and scalable AI-powered querying. Features async embedding pipelines, vector search, reranking, remote worker scaling, and multi-provider LLM orchestration. Built with Next.js, FastAPI, PostgreSQL, Redis, Qdrant, Docker, Ollama, Groq, and Gemini APIs.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Distributed RAG

ci

A self-hosted Retrieval-Augmented Generation system where the expensive work (extracting text from PDFs, OCR, transcribing audio and video, embedding chunks) is spread across any number of worker machines that pull jobs from a shared queue. Ask questions over your documents and get answers from Groq, Ollama or Gemini, grounded in the retrieved chunks.

It is split into 5 FastAPI services plus a Next.js UI so each part scales on its own:

Service Job Scale it when
gateway Uploads, scraping, query proxy, metrics Many concurrent users
ingestion-worker Extract + chunk (PDF, OCR, Whisper, vision) Upload backlog grows (CPU heavy)
embedding-worker Embed chunks, upsert vectors; also embeds queries Embedding backlog grows (API/IO bound)
vector-store Sole owner of Qdrant reads and writes Search latency rises
query-service Hybrid (dense + keyword) retrieval, fusion, rerank, LLM call Query QPS rises

Architecture

flowchart LR
    UI[Next.js UI] --> GW[gateway]
    GW -- XADD --> IS[(rag:ingestion<br/>Redis Stream)]
    IS -- XREADGROUP --> ING1[ingestion-worker A]
    IS -- XREADGROUP --> ING2[ingestion-worker B<br/>other machine]
    ING1 & ING2 -- chunks + job rows<br/>one transaction --> PG[(Postgres)]
    ING1 & ING2 -- XADD --> ES[(rag:embedding<br/>Redis Stream)]
    ES -- XREADGROUP --> EMB[embedding-worker x N]
    EMB -- upsert, id = hash(chunk_id) --> VS[vector-store] --> QD[(Qdrant)]
    EMB -- mark job done --> PG
    GW --> QS[query-service]
    QS -- embed question --> EMB
    QS -- search --> VS
    QS -- answer --> LLM[Groq / Ollama / Gemini]
Loading

Why the queue is a Redis Stream, not a list

The first version used LPUSH / BRPOP. BRPOP deletes the job the moment a worker takes it, so a worker that crashes, is OOM-killed or loses its network mid-job loses that document for good: it sits in extracting forever. With workers on several machines, that is a normal day, not an edge case.

Now both queues are Redis Streams read through consumer groups (shared/rqueue.py):

Concern How it is handled
Worker crashes mid-job Message stays in the group's pending list until XACK. Other workers run XAUTOCLAIM and take over anything idle longer than the visibility timeout.
Long job (a 40-minute video) mistaken for a crash The worker renews ownership every third of the timeout (XCLAIM on its own message), so it is never stolen while it is still running.
Transient failure (rate limit, vector store restarting) Re-published with an attempt counter and retried after a short backoff.
Poison job (corrupt file, bad API key) After max_deliveries it goes to rag:<stage>:dead and the document is marked failed. One bad file cannot stall a worker.
Two workers grab the same job Consumer groups deliver each message to exactly one consumer.
Consumers offline for days XADD ... MAXLEN ~ caps stream memory.

Delivery is at-least-once. A job can run twice (for example, a worker finishes and crashes before it acks). So every write is idempotent:

  • Chunk ids are deterministic: uuid5(doc_id, chunk_index). A re-run produces the same ids, and INSERT ... ON CONFLICT (id) DO UPDATE overwrites instead of duplicating. Leftover chunks from a previous run that produced more chunks are deleted in the same transaction.
  • Embedding jobs have a unique index on chunk_id.
  • Qdrant point ids are a hash of chunk_id, so a repeated upsert replaces the same point.

Why exactly-once is not the goal

True exactly-once across Redis, Postgres and Qdrant would need a distributed transaction. Idempotent consumers on an at-least-once queue give the same visible result (no duplicates, nothing lost) with far less machinery.

Document completion without races

The old code inserted each embedding_jobs row when a worker picked up that chunk, then marked the document done when no rows were pending. The first chunk to finish saw zero pending rows (the rest had not been inserted yet) and marked the whole document done early.

Now the ingestion worker writes all chunk rows and all job rows in one transaction, and only then publishes the embedding jobs. "Zero not-done rows" therefore really means every chunk is embedded. If two workers finish the last two chunks at the same time, each commits its own row before counting, so whichever counts last sees zero. Marking done twice is harmless.

Retrieval pipeline

Pure logic lives in shared/retrieval.py, so the service and the benchmark run the same code.

dense search (one per query variant) ─┐
                                      ├─ RRF fusion ─ dedupe ─ rerank ─ top k ─ abstain?
Postgres full-text search ────────────┘
  1. Dense search. The question (plus a couple of light rewrites, e.g. time words dropped from "when did" questions) is embedded and searched in Qdrant, concurrently.
  2. Keyword search (optional, off by default). With KEYWORD_SEARCH_ENABLED=1, the same question also runs against a generated tsvector column on chunks with a GIN index, fused at a lower weight (KEYWORD_RRF_WEIGHT=0.3). The tsquery ORs the question's terms (plain plainto_tsquery ANDs them, so one missing word drops a chunk) and ts_rank_cd rewards chunks that match more terms, closer together. This finds exact strings embeddings blur: error codes, names, version numbers. It is off by default because the benchmark below measured it as a loss on natural-language questions.
  3. Reciprocal Rank Fusion (k=60). Lists are merged by rank, so a cosine similarity and a ts_rank_cd score never have to share a scale.
  4. Dedupe. Chunks with identical text (case and whitespace ignored) keep only their best-ranked copy, so the same paragraph from two uploads can't fill several of the top-k slots.
  5. Rerank the fused shortlist: w_rrf * fused rank + w_lex * keyword overlap + w_sem * cosine. Cosine is only a small bonus because keyword-only hits don't have one. Weights were tuned on a training split (see Benchmark).
  6. Abstain unless there is a strong dense match or a chunk containing most of the question's content words, instead of letting the LLM improvise.
  7. Answer with Groq, Ollama or Gemini. Audio and video chunks carry timestamps, so answers can cite "at 12:40".

Every response includes retrieval stats: query variants, dense hits, keyword hits, fused count and duplicates dropped.

Duplicate documents

  • Same file uploaded twice. The gateway hashes the bytes (SHA-256) while streaming the upload. If a non-failed document with that hash exists, it returns that document ("duplicate": true) and deletes the new copy instead of ingesting it again. A partial unique index on documents(content_sha256) makes the check race-safe: two identical uploads at the same moment both pass the lookup, exactly one INSERT wins, and the other falls back to the winner. Failed documents are excluded, so a failed upload can be retried. Website scrapes are hashed the same way.
  • Same text in different files. Handled at query time by the dedupe step above.

Existing databases get the new columns and indexes on gateway startup (idempotent ALTER ... IF NOT EXISTS), the same statements as the tail of shared/init.sql.

Benchmark

eval/retrieval_benchmark.py measures what each stage is worth, running the production code: the service's query variants and keyword SQL against a real Postgres built from shared/init.sql, and the fusion, dedupe and rerank from shared/retrieval.py.

  • Data: BEIR SciFact, 5,183 scientific abstracts with labelled answers.
  • Honest split: rerank weights tuned on the 809 training questions; every number below is on the 300 test questions, read once.
  • Embedder: BAAI/bge-small-en-v1.5 locally, as a stand-in for the production Gemini embedder. The comparison is between pipeline stages under one fixed embedder.
Pipeline (300 test questions) nDCG@10 Recall@5 Recall@10 MRR@10
Dense search, original question only 0.706 0.764 0.830 0.673
Previous pipeline (dense variants + old rerank) 0.706 0.768 0.818 0.676
Keyword search only (Postgres full text) 0.328 0.405 0.486 0.281
Dense + keyword, fused, no rerank 0.642 0.703 0.809 0.599
Dense + keyword (full weight) + rerank 0.682 0.731 0.818 0.648
Dense + keyword (weight 0.3) + rerank 0.717 0.778 0.840 0.687
Dense + dedupe + tuned rerank (default) 0.724 0.779 0.840 0.696

Duplicates. With 30% of the corpus re-uploaded under new ids, duplicate copies took 17.1% of the top-5 slots; with query-time dedupe (dedupe() in shared/retrieval.py, keyed on a hash of the normalised chunk text), 0%. This measures duplicates that reach the index anyway, such as the same text in two different files. The upload-time SHA-256 check is a separate guard that stops the identical file being ingested twice; the benchmark does not exercise it, because it re-ingests under new ids.

What this changed: the default pipeline is dense search with dedupe and the tuned rerank, up from 0.706 to 0.724 nDCG@10 and from 0.818 to 0.840 Recall@10 over the previous pipeline. Postgres full-text ranking (ts_rank_cd) is a much weaker keyword signal than BM25 (0.33 nDCG@10 alone), and fused at full weight it pulled good dense results down, so it ships off. A proper BM25 index is the next thing to try.

Reproduce (about 15 minutes on a laptop CPU):

docker run -d -p 5433:5432 -e POSTGRES_HOST_AUTH_METHOD=trust -e POSTGRES_DB=ragtest postgres:16-alpine
pip install -r eval/requirements.txt
python eval/retrieval_benchmark.py          # full run: tune, test, duplicate experiment
python eval/retrieval_benchmark.py --quick  # reuse tuned weights, re-score the test strategies

Keeping queries fast during an ingest spike

  • Ingestion is fully async. The upload request returns as soon as the file is saved and the job is published.
  • Query-time embedding has its own concurrency limit (QUERY_EMBED_CONCURRENCY), separate from the background embedding workers (WORKER_CONCURRENCY). A backlog of 10,000 chunks waits in the stream; it does not take query embedding slots.
  • Workers can be moved to other machines (docker-compose.worker.yml), so heavy Whisper/OCR work leaves the box that serves queries.

Tests

pip install -r tests/requirements.txt
# needs Redis on :6379 and Postgres on :5433 (or set TEST_REDIS_URL / TEST_POSTGRES_URL)
pytest -q

They run against real Redis and Postgres (CI starts both as service containers). External AI calls are stubbed.

Test Proves
test_crashed_consumer_job_is_reclaimed_by_another A crashed worker's job is finished by another worker
test_keepalive_prevents_steal_of_long_job A job running 3x the visibility timeout is not stolen
test_each_message_goes_to_exactly_one_live_consumer 200 jobs, 4 workers: each processed once
test_failed_job_is_retried_then_dead_lettered / ..._keeps_crashing_workers_... Bounded retries, then dead-letter
test_happy_path_marks_done_only_after_last_chunk The early-"done" bug is fixed
test_ingestion_worker_crash_is_recovered_without_duplicates Re-running a document leaves chunks = jobs = vectors
test_transient_embedding_error_is_retried / test_permanent_..._fails_document Rate limits recover; bad keys fail the doc
test_rrf_*, test_dedupe_*, test_rerank_*, test_abstains_* Fusion, dedupe, rerank and abstain rules
test_keyword_search_* (real Postgres) The service's exact keyword SQL: exact terms found, OR semantics, unfinished documents skipped
test_same_file_cannot_be_registered_twice / test_failed_upload_can_be_retried_... The duplicate-upload guard

Quick start

cp .env.example .env        # set GEMINI_API_KEY; optional GROQ_API_KEY, OLLAMA_URL
docker compose up -d --build

Add worker machines

On another machine with access to the uploads folder:

# .env: PUBLIC_HOST=<LAN IP of the main machine>, SHARED_UPLOADS_DIR=<path that mirrors /uploads>
docker compose -f docker-compose.worker.yml up -d --build

They join the same consumer groups automatically. Consumer names are host-pid, so every process is distinct.

More detail: SETUP.md.

Known limits

  • Shared filesystem. Remote ingestion workers read the uploaded file from a shared folder. Object storage (S3/MinIO) would remove that requirement; it is the next change I would make.
  • LAN-trust deployment. Redis and Postgres are exposed to the LAN with default credentials so remote workers can connect. Fine on a home or lab network; on anything else, set passwords, enable Redis ACLs and put the workers on a private network or VPN.
  • Stale vectors on shrink. If a re-run of a document produces fewer chunks than the first run (rare: only if extraction output changes between attempts), the extra Postgres chunks are deleted but their Qdrant points stay until the document is deleted. Fix: delete points for the removed chunk ids in the same step.
  • Rerank is a weighted formula, not a model. A cross-encoder would likely rank better, at the cost of a model call per query.
  • Benchmark embedder is a stand-in. Production embeds with Gemini; the benchmark uses bge-small-en-v1.5 locally to compare pipeline stages, so absolute numbers will differ.
  • One benchmark dataset. SciFact questions are natural-language claims. A corpus of logs or tickets full of exact codes may favour keyword search; measure before switching it on.

Repository layout

gateway/            API ingress, uploads, scrape, query proxy, metrics
ingestion-worker/   extract + chunk, publishes embedding jobs
embedding-worker/   embeds chunks and queries, upserts vectors
vector-store/       owns Qdrant
query-service/      retrieval orchestration, keyword SQL, LLM providers
shared/retrieval.py fusion, dedupe, rerank, abstain (pure, tested)
eval/               retrieval benchmark on BEIR SciFact
shared/rqueue.py    at-least-once stream queue used by all three producers/consumers
shared/init.sql     schema
frontend/           Next.js UI
tests/              queue, pipeline and retrieval tests

About

Distributed microservice-based RAG platform for multimodal ingestion, semantic retrieval, and scalable AI-powered querying. Features async embedding pipelines, vector search, reranking, remote worker scaling, and multi-provider LLM orchestration. Built with Next.js, FastAPI, PostgreSQL, Redis, Qdrant, Docker, Ollama, Groq, and Gemini APIs.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages