[YBazanPro]

Summary

Implements spec 019. Both stages ship switched off: the evaluation gate (AC-20 to AC-22) is deferred until after spec 020.

Based on #18 (nightly workflow fix and spec 018 calibration). Its two commits appear here until #18 is merged.

Hybrid search (Documents)

• Migration HybridSearch:
  • the unaccent extension and the documents.french_unaccent text search configuration;
  • passages.search_config (english, documents.french_unaccent or simple) and a generated, GIN-indexed search_vector.
• Database triggers set each passage's configuration from its document language on insert, and update it in the same transaction when the language changes, so imports, edits and language detection all stay consistent.
• Keyword pool: websearch_to_tsquery in the question's language (SearchFilters.QueryLanguage, detected question language, never the requested answer language), ranked with ts_rank_cd. It uses the same filters and ReadableBy as the vector pool and is fetched while the query is embedded.
• Fusion: reciprocal rank fusion (Documents:Retrieval:Hybrid:RrfK, default 60). Without reranking, cosine still gates, so a keyword-only hit needs cosine ≥ MinScore.

Reranking

• Contract: IPassageReranker in SharedKernel/Contracts/Retrieval. The API implements it with a typed HttpClient against Text Embeddings Inference serving BAAI/bge-reranker-v2-m3 (ADR 0010, image cpu-1.9.4).
• Flow: the top Candidates (30) go in one call with a 2 s timeout. They are then reordered and gated by the rerank score (Reranking:MinScore / ConfidentScore), and near-duplicate reduction and topK apply after.
• Fallback: a timeout, error, unconfigured reranker or wrong score count gives rerank: skipped and cosine gating, plus a documents.rerank.skipped counter. A question never fails because of the reranker.
• AppHost: the reranker container exists only with the new https-rerank launch profile, or --Parameters:reranking=true.

Diagnostics

• Per-passage stages: vector and keyword ranks, cosine, ts_rank_cd, fused and rerank scores, admittedBy.
• Where they show: the search endpoint, the request trace (migration RetrievalStages) and the admin request inspector. The activity panel and the SSE retrieval.completed event show mode, rerank and admittedBy only.
• Search endpoint overrides: mode, rerank and queryLanguage.
• Admin configuration: the hybrid and rerank settings are on the Documents configuration, which owns them, rather than the Chat one the spec names. Chat cannot read Documents options across the module boundary.
• Metrics: documents.rerank.duration, documents.rerank.skipped, documents.search.keyword_hits, and the Chat metrics tagged retrieval.mode.

Harness

• run --variants measures retrieval for vector, hybrid and hybrid + rerank on the same queries.
• Reports record the stage settings, and a run mixing retrieval modes is marked invalid.
• sweep calibrates rerank thresholds on their own grid.
• The nightly workflow now runs with --variants; without TEI on the runner, the rerank variant is skipped.

Docs

ADR 0010, CLAUDE.md (module table, stack, commands) and the rag-pipeline skill.

Measured so far (not the gate)

• TEI cpu-1.9.4 on the reference machine: 18–24 s to rerank 30 corpus-length passages, against the 800 ms budget. Relevant passages scored 0.99, unrelated ones 0.0000. With the 2 s default timeout, reranking is skipped on a CPU.
• Postgres image: unaccent ships with the pinned pgvector/pgvector:0.8.7-pg18-trixie image, as the spec required.

Test plan

• dotnet build with 0 warnings, and dotnet format --verify-no-changes clean
• Full suite: 1,288 tests, all passing after updating ChatMetricsTests for the new low-cardinality retrieval.mode label
• New tests:
  • HybridRankingTests (AC-9, ordering, stages, rerank order);
  • HybridSearchTests (AC-1 to AC-8, AC-15, AC-16);
  • RerankingTests (AC-10, AC-12 to AC-14);
  • RetrievalStagesTests (AC-11, AC-12 trace, AC-17, keyword configuration of the question's language);
  • TeiPassageRerankerTests.
• AC-19: the AppHost started with https-rerank (API and Keycloak ready), but whether TEI came up and the API reached it was not checked before the run was stopped
• AC-20 to AC-22: evaluation gate and the spec's Calibration section, deferred until after spec 020; until then Hybrid:Enabled and Reranking:Enabled stay false

🤖 Generated with Claude Code