A self-hosted Retrieval-Augmented Generation system where the expensive work (extracting text from PDFs, OCR, transcribing audio and video, embedding chunks) is spread across any number of worker machines that pull jobs from a shared queue. Ask questions over your documents and get answers from Groq, Ollama or Gemini, grounded in the retrieved chunks.
It is split into 5 FastAPI services plus a Next.js UI so each part scales on its own:
| Service | Job | Scale it when |
|---|---|---|
gateway |
Uploads, scraping, query proxy, metrics | Many concurrent users |
ingestion-worker |
Extract + chunk (PDF, OCR, Whisper, vision) | Upload backlog grows (CPU heavy) |
embedding-worker |
Embed chunks, upsert vectors; also embeds queries | Embedding backlog grows (API/IO bound) |
vector-store |
Sole owner of Qdrant reads and writes | Search latency rises |
query-service |
Hybrid (dense + keyword) retrieval, fusion, rerank, LLM call | Query QPS rises |
flowchart LR
UI[Next.js UI] --> GW[gateway]
GW -- XADD --> IS[(rag:ingestion<br/>Redis Stream)]
IS -- XREADGROUP --> ING1[ingestion-worker A]
IS -- XREADGROUP --> ING2[ingestion-worker B<br/>other machine]
ING1 & ING2 -- chunks + job rows<br/>one transaction --> PG[(Postgres)]
ING1 & ING2 -- XADD --> ES[(rag:embedding<br/>Redis Stream)]
ES -- XREADGROUP --> EMB[embedding-worker x N]
EMB -- upsert, id = hash(chunk_id) --> VS[vector-store] --> QD[(Qdrant)]
EMB -- mark job done --> PG
GW --> QS[query-service]
QS -- embed question --> EMB
QS -- search --> VS
QS -- answer --> LLM[Groq / Ollama / Gemini]
The first version used LPUSH / BRPOP. BRPOP deletes the job the moment a worker takes it, so a worker that crashes, is OOM-killed or loses its network mid-job loses that document for good: it sits in extracting forever. With workers on several machines, that is a normal day, not an edge case.
Now both queues are Redis Streams read through consumer groups (shared/rqueue.py):
| Concern | How it is handled |
|---|---|
| Worker crashes mid-job | Message stays in the group's pending list until XACK. Other workers run XAUTOCLAIM and take over anything idle longer than the visibility timeout. |
| Long job (a 40-minute video) mistaken for a crash | The worker renews ownership every third of the timeout (XCLAIM on its own message), so it is never stolen while it is still running. |
| Transient failure (rate limit, vector store restarting) | Re-published with an attempt counter and retried after a short backoff. |
| Poison job (corrupt file, bad API key) | After max_deliveries it goes to rag:<stage>:dead and the document is marked failed. One bad file cannot stall a worker. |
| Two workers grab the same job | Consumer groups deliver each message to exactly one consumer. |
| Consumers offline for days | XADD ... MAXLEN ~ caps stream memory. |
Delivery is at-least-once. A job can run twice (for example, a worker finishes and crashes before it acks). So every write is idempotent:
- Chunk ids are deterministic:
uuid5(doc_id, chunk_index). A re-run produces the same ids, andINSERT ... ON CONFLICT (id) DO UPDATEoverwrites instead of duplicating. Leftover chunks from a previous run that produced more chunks are deleted in the same transaction. - Embedding jobs have a unique index on
chunk_id. - Qdrant point ids are a hash of
chunk_id, so a repeated upsert replaces the same point.
True exactly-once across Redis, Postgres and Qdrant would need a distributed transaction. Idempotent consumers on an at-least-once queue give the same visible result (no duplicates, nothing lost) with far less machinery.
The old code inserted each embedding_jobs row when a worker picked up that chunk, then marked the document done when no rows were pending. The first chunk to finish saw zero pending rows (the rest had not been inserted yet) and marked the whole document done early.
Now the ingestion worker writes all chunk rows and all job rows in one transaction, and only then publishes the embedding jobs. "Zero not-done rows" therefore really means every chunk is embedded. If two workers finish the last two chunks at the same time, each commits its own row before counting, so whichever counts last sees zero. Marking done twice is harmless.
Pure logic lives in shared/retrieval.py, so the service and the benchmark run the
same code.
dense search (one per query variant) ─┐
├─ RRF fusion ─ dedupe ─ rerank ─ top k ─ abstain?
Postgres full-text search ────────────┘
- Dense search. The question (plus a couple of light rewrites, e.g. time words dropped from "when did" questions) is embedded and searched in Qdrant, concurrently.
- Keyword search (optional, off by default). With
KEYWORD_SEARCH_ENABLED=1, the same question also runs against a generatedtsvectorcolumn onchunkswith a GIN index, fused at a lower weight (KEYWORD_RRF_WEIGHT=0.3). The tsquery ORs the question's terms (plainplainto_tsqueryANDs them, so one missing word drops a chunk) andts_rank_cdrewards chunks that match more terms, closer together. This finds exact strings embeddings blur: error codes, names, version numbers. It is off by default because the benchmark below measured it as a loss on natural-language questions. - Reciprocal Rank Fusion (
k=60). Lists are merged by rank, so a cosine similarity and ats_rank_cdscore never have to share a scale. - Dedupe. Chunks with identical text (case and whitespace ignored) keep only their best-ranked copy, so the same paragraph from two uploads can't fill several of the top-k slots.
- Rerank the fused shortlist:
w_rrf * fused rank + w_lex * keyword overlap + w_sem * cosine. Cosine is only a small bonus because keyword-only hits don't have one. Weights were tuned on a training split (see Benchmark). - Abstain unless there is a strong dense match or a chunk containing most of the question's content words, instead of letting the LLM improvise.
- Answer with Groq, Ollama or Gemini. Audio and video chunks carry timestamps, so answers can cite "at 12:40".
Every response includes retrieval stats: query variants, dense hits, keyword hits, fused count and
duplicates dropped.
- Same file uploaded twice. The gateway hashes the bytes (SHA-256) while streaming the upload. If a
non-failed document with that hash exists, it returns that document (
"duplicate": true) and deletes the new copy instead of ingesting it again. A partial unique index ondocuments(content_sha256)makes the check race-safe: two identical uploads at the same moment both pass the lookup, exactly one INSERT wins, and the other falls back to the winner. Failed documents are excluded, so a failed upload can be retried. Website scrapes are hashed the same way. - Same text in different files. Handled at query time by the dedupe step above.
Existing databases get the new columns and indexes on gateway startup (idempotent ALTER ... IF NOT EXISTS),
the same statements as the tail of shared/init.sql.
eval/retrieval_benchmark.py measures what each stage is worth, running the
production code: the service's query variants and keyword SQL against a real Postgres built from
shared/init.sql, and the fusion, dedupe and rerank from shared/retrieval.py.
- Data: BEIR SciFact, 5,183 scientific abstracts with labelled answers.
- Honest split: rerank weights tuned on the 809 training questions; every number below is on the 300 test questions, read once.
- Embedder:
BAAI/bge-small-en-v1.5locally, as a stand-in for the production Gemini embedder. The comparison is between pipeline stages under one fixed embedder.
| Pipeline (300 test questions) | nDCG@10 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| Dense search, original question only | 0.706 | 0.764 | 0.830 | 0.673 |
| Previous pipeline (dense variants + old rerank) | 0.706 | 0.768 | 0.818 | 0.676 |
| Keyword search only (Postgres full text) | 0.328 | 0.405 | 0.486 | 0.281 |
| Dense + keyword, fused, no rerank | 0.642 | 0.703 | 0.809 | 0.599 |
| Dense + keyword (full weight) + rerank | 0.682 | 0.731 | 0.818 | 0.648 |
| Dense + keyword (weight 0.3) + rerank | 0.717 | 0.778 | 0.840 | 0.687 |
| Dense + dedupe + tuned rerank (default) | 0.724 | 0.779 | 0.840 | 0.696 |
Duplicates. With 30% of the corpus re-uploaded under new ids, duplicate copies took 17.1% of the top-5
slots; with query-time dedupe (dedupe() in shared/retrieval.py, keyed on a hash of the normalised chunk
text), 0%. This measures duplicates that reach the index anyway, such as the same text in two different
files. The upload-time SHA-256 check is a separate guard that stops the identical file being ingested twice;
the benchmark does not exercise it, because it re-ingests under new ids.
What this changed: the default pipeline is dense search with dedupe and the tuned rerank, up from 0.706 to
0.724 nDCG@10 and from 0.818 to 0.840 Recall@10 over the previous pipeline. Postgres full-text ranking
(ts_rank_cd) is a much weaker keyword signal than BM25 (0.33 nDCG@10 alone), and fused at full weight it
pulled good dense results down, so it ships off. A proper BM25 index is the next thing to try.
Reproduce (about 15 minutes on a laptop CPU):
docker run -d -p 5433:5432 -e POSTGRES_HOST_AUTH_METHOD=trust -e POSTGRES_DB=ragtest postgres:16-alpine
pip install -r eval/requirements.txt
python eval/retrieval_benchmark.py # full run: tune, test, duplicate experiment
python eval/retrieval_benchmark.py --quick # reuse tuned weights, re-score the test strategies- Ingestion is fully async. The upload request returns as soon as the file is saved and the job is published.
- Query-time embedding has its own concurrency limit (
QUERY_EMBED_CONCURRENCY), separate from the background embedding workers (WORKER_CONCURRENCY). A backlog of 10,000 chunks waits in the stream; it does not take query embedding slots. - Workers can be moved to other machines (
docker-compose.worker.yml), so heavy Whisper/OCR work leaves the box that serves queries.
pip install -r tests/requirements.txt
# needs Redis on :6379 and Postgres on :5433 (or set TEST_REDIS_URL / TEST_POSTGRES_URL)
pytest -qThey run against real Redis and Postgres (CI starts both as service containers). External AI calls are stubbed.
| Test | Proves |
|---|---|
test_crashed_consumer_job_is_reclaimed_by_another |
A crashed worker's job is finished by another worker |
test_keepalive_prevents_steal_of_long_job |
A job running 3x the visibility timeout is not stolen |
test_each_message_goes_to_exactly_one_live_consumer |
200 jobs, 4 workers: each processed once |
test_failed_job_is_retried_then_dead_lettered / ..._keeps_crashing_workers_... |
Bounded retries, then dead-letter |
test_happy_path_marks_done_only_after_last_chunk |
The early-"done" bug is fixed |
test_ingestion_worker_crash_is_recovered_without_duplicates |
Re-running a document leaves chunks = jobs = vectors |
test_transient_embedding_error_is_retried / test_permanent_..._fails_document |
Rate limits recover; bad keys fail the doc |
test_rrf_*, test_dedupe_*, test_rerank_*, test_abstains_* |
Fusion, dedupe, rerank and abstain rules |
test_keyword_search_* (real Postgres) |
The service's exact keyword SQL: exact terms found, OR semantics, unfinished documents skipped |
test_same_file_cannot_be_registered_twice / test_failed_upload_can_be_retried_... |
The duplicate-upload guard |
cp .env.example .env # set GEMINI_API_KEY; optional GROQ_API_KEY, OLLAMA_URL
docker compose up -d --build- UI: http://localhost:3000
- Gateway health: http://localhost:8000/health
- Queue backlog per stage:
GET /api/stats/queuereturnslag(not yet delivered),pending(being worked on or held by a crashed worker) anddead.
On another machine with access to the uploads folder:
# .env: PUBLIC_HOST=<LAN IP of the main machine>, SHARED_UPLOADS_DIR=<path that mirrors /uploads>
docker compose -f docker-compose.worker.yml up -d --buildThey join the same consumer groups automatically. Consumer names are host-pid, so every process is distinct.
More detail: SETUP.md.
- Shared filesystem. Remote ingestion workers read the uploaded file from a shared folder. Object storage (S3/MinIO) would remove that requirement; it is the next change I would make.
- LAN-trust deployment. Redis and Postgres are exposed to the LAN with default credentials so remote workers can connect. Fine on a home or lab network; on anything else, set passwords, enable Redis ACLs and put the workers on a private network or VPN.
- Stale vectors on shrink. If a re-run of a document produces fewer chunks than the first run (rare: only if extraction output changes between attempts), the extra Postgres chunks are deleted but their Qdrant points stay until the document is deleted. Fix: delete points for the removed chunk ids in the same step.
- Rerank is a weighted formula, not a model. A cross-encoder would likely rank better, at the cost of a model call per query.
- Benchmark embedder is a stand-in. Production embeds with Gemini; the benchmark uses
bge-small-en-v1.5locally to compare pipeline stages, so absolute numbers will differ. - One benchmark dataset. SciFact questions are natural-language claims. A corpus of logs or tickets full of exact codes may favour keyword search; measure before switching it on.
gateway/ API ingress, uploads, scrape, query proxy, metrics
ingestion-worker/ extract + chunk, publishes embedding jobs
embedding-worker/ embeds chunks and queries, upserts vectors
vector-store/ owns Qdrant
query-service/ retrieval orchestration, keyword SQL, LLM providers
shared/retrieval.py fusion, dedupe, rerank, abstain (pure, tested)
eval/ retrieval benchmark on BEIR SciFact
shared/rqueue.py at-least-once stream queue used by all three producers/consumers
shared/init.sql schema
frontend/ Next.js UI
tests/ queue, pipeline and retrieval tests