fix(retrieval): OR keyword query and coverage-gated floor exemption (spec 019)

CLAUDE.md

@@ -12,7 +12,7 @@ Stack decisions are recorded in [ADR 0001](docs/adr/0001-ui-ai-streaming-stack.m
 
 | Module | Owns | Schema |
 |--------|------|--------|
-| `Documents` | Upload/ingest documents, text extraction, chunking, embedding generation, vector storage, similarity search. Exists: data model, `POST /api/documents`, `GET /api/documents/{id}` ([spec 006](docs/specs/006-documents-data-model-and-import.md)); background ingestion (`IngestionWorker`, chunker, batched embedding), `POST /api/documents/{id}/ingestion/retry` and the `GET …/ingestion/events` SSE stream ([spec 007](docs/specs/007-background-ingestion-and-progress.md)); listing with filters and paging, facets, versions and passages of a version (`GET /api/documents`, `/facets`, `/{id}/versions`, `/{id}/versions/{n}/passages`) and the Documents UI ([spec 008](docs/specs/008-browse-corpus-and-documents-ui.md)); edit metadata (`PUT`, `If-Match`/`ETag`), new version, reprocess, archive and restore, with the matching detail-page actions ([spec 009](docs/specs/009-document-lifecycle.md)); PDF import up to 100 MiB (streamed upload into a content-addressed file store, text-layer extraction with page numbers on passages, scanned PDFs rejected at upload), `GET …/versions/{n}/file` for the original and `progress` events inside long ingestion stages ([spec 010](docs/specs/010-pdf-import.md)); retrieval through `IDocumentSearch` (vector search over active, indexed versions with language and tag filters, evidence thresholds, near-duplicate reduction, cached query embeddings) and the maintainers' diagnostic `GET /api/documents/search` ([spec 011](docs/specs/011-retrieval.md)); private and shared documents: `Visibility` on `Document`, owner = `CreatedBy`, one `ReadableBy(viewerSub)` rule applied on every read path, so a document the caller cannot read is a 404 and `IDocumentSearch`/`IDocumentCatalog` take a required `viewerSub` ([spec 016](docs/specs/016-private-and-shared-documents.md)); maintainers' read side under `/api/admin/documents` (corpus health, currently failed versions, a metadata-only list of private documents that deliberately bypasses `ReadableBy`, configuration) and `IDocumentTitles`, which returns titles of shared documents only ([spec 017](docs/specs/017-diagnose-traces-admin-and-metrics.md)); the document title leads every embedding input (`EmbeddingInput`), and each version records `EmbeddingInputVersion`: a version embedded the old way stays searchable, counts as `outdatedEmbeddings` in admin health and is reprocessed by the harness `seed` ([spec 021](docs/specs/021-context-budget-title-embeddings-nightly-evaluation.md)); raise `EmbeddingInput.Version` whenever the input changes; hybrid search and reranking behind switches ([spec 019](docs/specs/019-hybrid-search-and-reranking.md)): a keyword pool (`passages.search_vector`, generated from `search_config`, which database triggers keep equal to the document language: `english`, `documents.french_unaccent` or `simple`) fused with the vector pool by reciprocal rank fusion (`HybridRanking`), then optionally a cross-encoder through `IPassageReranker` (`PassageReranking`: one call, its own timeout, any failure means `rerank: skipped` and cosine gating); every passage carries `PassageStages`, the trace and the admin inspector show them, and `GET /api/documents/search` takes `mode`, `rerank` and `queryLanguage` overrides; first-release polish ([spec 020](docs/specs/020-first-release-polish.md)): after indexing, `SuggestionGenerator` asks the chat model (`IChatClient`, structured output) for 3 questions and keeps those retrieval answers confidently as the owner, on the version (`SuggestedQuestions`, `SuggestionsStatus` pending/ready/failed/skipped/edited); it never fails the run, is off in demo mode (`Documents:Suggestions:Enabled`, turned off by the API) and records a `suggestions` stage event after `completed`; the owner edits them with `PUT …/{id}/suggestions` (`If-Match`), edited ones survive reprocess; `GET /api/documents/suggestions` and `/summary` feed the home page; every passage stores `TextHash` (`PassageTextHash` in the kernel, also written out in the migrations' SQL) and `IDocumentSources` gives each cited passage a source state (current, newer-unchanged, newer-changed, archived, unavailable) | `documents` |
+| `Documents` | Upload/ingest documents, text extraction, chunking, embedding generation, vector storage, similarity search. Exists: data model, `POST /api/documents`, `GET /api/documents/{id}` ([spec 006](docs/specs/006-documents-data-model-and-import.md)); background ingestion (`IngestionWorker`, chunker, batched embedding), `POST /api/documents/{id}/ingestion/retry` and the `GET …/ingestion/events` SSE stream ([spec 007](docs/specs/007-background-ingestion-and-progress.md)); listing with filters and paging, facets, versions and passages of a version (`GET /api/documents`, `/facets`, `/{id}/versions`, `/{id}/versions/{n}/passages`) and the Documents UI ([spec 008](docs/specs/008-browse-corpus-and-documents-ui.md)); edit metadata (`PUT`, `If-Match`/`ETag`), new version, reprocess, archive and restore, with the matching detail-page actions ([spec 009](docs/specs/009-document-lifecycle.md)); PDF import up to 100 MiB (streamed upload into a content-addressed file store, text-layer extraction with page numbers on passages, scanned PDFs rejected at upload), `GET …/versions/{n}/file` for the original and `progress` events inside long ingestion stages ([spec 010](docs/specs/010-pdf-import.md)); retrieval through `IDocumentSearch` (vector search over active, indexed versions with language and tag filters, evidence thresholds, near-duplicate reduction, cached query embeddings) and the maintainers' diagnostic `GET /api/documents/search` ([spec 011](docs/specs/011-retrieval.md)); private and shared documents: `Visibility` on `Document`, owner = `CreatedBy`, one `ReadableBy(viewerSub)` rule applied on every read path, so a document the caller cannot read is a 404 and `IDocumentSearch`/`IDocumentCatalog` take a required `viewerSub` ([spec 016](docs/specs/016-private-and-shared-documents.md)); maintainers' read side under `/api/admin/documents` (corpus health, currently failed versions, a metadata-only list of private documents that deliberately bypasses `ReadableBy`, configuration) and `IDocumentTitles`, which returns titles of shared documents only ([spec 017](docs/specs/017-diagnose-traces-admin-and-metrics.md)); the document title leads every embedding input (`EmbeddingInput`), and each version records `EmbeddingInputVersion`: a version embedded the old way stays searchable, counts as `outdatedEmbeddings` in admin health and is reprocessed by the harness `seed` ([spec 021](docs/specs/021-context-budget-title-embeddings-nightly-evaluation.md)); raise `EmbeddingInput.Version` whenever the input changes; hybrid search and reranking behind switches ([spec 019](docs/specs/019-hybrid-search-and-reranking.md)): a keyword pool (`passages.search_vector`, generated from `search_config`, which database triggers keep equal to the document language: `english`, `documents.french_unaccent` or `simple`; the question's lexemes are OR-ed, and a match holding at least `Hybrid:MinKeywordCoverage` of them passes the cosine floor) fused with the vector pool by reciprocal rank fusion (`HybridRanking`), then optionally a cross-encoder through `IPassageReranker` (`PassageReranking`: one call, its own timeout, any failure means `rerank: skipped` and cosine gating); every passage carries `PassageStages`, the trace and the admin inspector show them, and `GET /api/documents/search` takes `mode`, `rerank` and `queryLanguage` overrides; first-release polish ([spec 020](docs/specs/020-first-release-polish.md)): after indexing, `SuggestionGenerator` asks the chat model (`IChatClient`, structured output) for 3 questions and keeps those retrieval answers confidently as the owner, on the version (`SuggestedQuestions`, `SuggestionsStatus` pending/ready/failed/skipped/edited); it never fails the run, is off in demo mode (`Documents:Suggestions:Enabled`, turned off by the API) and records a `suggestions` stage event after `completed`; the owner edits them with `PUT …/{id}/suggestions` (`If-Match`), edited ones survive reprocess; `GET /api/documents/suggestions` and `/summary` feed the home page; every passage stores `TextHash` (`PassageTextHash` in the kernel, also written out in the migrations' SQL) and `IDocumentSources` gives each cited passage a source state (current, newer-unchanged, newer-changed, archived, unavailable) | `documents` |
 | `Chat` | Conversations and messages, prompt building, retrieval orchestration, LLM answer generation with citations. Exists: owned conversations (`POST/GET /api/chat/conversations`, `GET …/{id}`), the SSE answer stream `POST …/{id}/messages` (refusal without a model call on `NoEvidence`, one prompt builder, `[n]` citations validated in the stream, sentinel veto, complete/partial status, citation snapshots), `POST …/messages/{id}/retry`, the stale-answer sweep and the write side of request traces ([spec 012](docs/specs/012-chat-conversations-and-grounded-answers.md)); follow-up questions (a bounded history window feeds a `GetResponseAsync<RewriteResult>` rewrite into a standalone retrieval query, with fallback to the original question), ratings and issue reports on finished answers, and hard deletion of a conversation with its traces ([spec 014](docs/specs/014-follow-ups-feedback-and-deletion.md)); a per-conversation scope of documents and tags (`PUT …/scope`, narrowing retrieval through `SearchFilters.DocumentIds`) and the stored activity of each answer (`GET …/messages/{id}/activity`) ([spec 015](docs/specs/015-chat-scope-activity-and-layout.md)); maintainers' read side under `/api/admin/chat` (requests and the request inspector's data, summary with SQL percentiles, feedback list, configuration) and the `RagAssistant.Chat` meter, recorded from the finished trace ([spec 017](docs/specs/017-diagnose-traces-admin-and-metrics.md)); a context budget (`PromptBudget`, `Chat:Generation:ContextWindowTokens`, sent to Ollama as `num_ctx` by the API's `OllamaRequestOptions`) drops the lowest-ranked passages before `retrieval.completed` and records the trim and the prompt estimate in the trace ([spec 021](docs/specs/021-context-budget-title-embeddings-nightly-evaluation.md)); conversations can be renamed (`PATCH …/{id}`, `TitleEdited`; an empty title reverts to the first question), searched (`q`, `unaccent` + `ILIKE` over titles and questions) and exported as Markdown (`GET …/{id}/export`, the source lines of `SourceList` match the client's "Copy answer with sources" in `SourceListText`), and citations carry a `sourceState` computed from `CitationSnapshot.TextHash` when the conversation is read ([spec 020](docs/specs/020-first-release-polish.md)); load the `rag-pipeline` skill before changing it | `chat` |
 | `Evaluation` | Harness reports uploaded for maintainers ([spec 020](docs/specs/020-first-release-polish.md), roadmap 5.3): `POST /api/admin/evaluation/runs` (report JSON ≤ 10 MiB, schema 1, `sample` scope only, idempotent by run id, an optional `judge` property replaces the judged metrics), list, detail, compare and delete under `/api/admin/evaluation`, Maintainer only. It reads the report as JSON (`Common/Json.cs`) and binds none of the harness's types; no contract with other modules | `evaluation` |
 

docs/specs/019-hybrid-search-and-reranking.md

@@ -23,7 +23,7 @@ Success: the harness compares vector, hybrid, and hybrid + rerank on the same ru
   - `search_vector` (`tsvector`), a stored generated column `to_tsvector(search_config, text)` with a GIN index.
   The `french_unaccent` configuration (French stemmer plus `unaccent`) and the `unaccent` extension are created by the Documents migration. The generated column backfills existing passages.
 - **The passage's config follows its document's language.** When a document's language changes (edit, detection, spec 018), its passages' `search_config` is updated in the same transaction.
-- **Keyword query in the question's written language.** The config follows the detected question language from spec 018 (with its conversation fallback), never the requested answer language. `websearch_to_tsquery(config, query)`, ranked with `ts_rank_cd`. `IDocumentSearch` receives the language through `SearchFilters.QueryLanguage` (nullable → `simple`). French questions therefore get keyword matches from French passages only. Vector search still covers every language.
+- **Keyword query in the question's written language.** The config follows the detected question language from spec 018 (with its conversation fallback), never the requested answer language. The question's distinct lexemes (`to_tsvector(config, query)`) are OR-ed into one `tsquery`, ranked with `ts_rank_cd`. *Revised after the first variants run:* the original `websearch_to_tsquery` ANDs every word, so a natural-language question matched almost no passage and hybrid equalled vector. Each match also records its **coverage**: the share of the question's distinct lexemes the passage contains. `IDocumentSearch` receives the language through `SearchFilters.QueryLanguage` (nullable → `simple`). French questions therefore get keyword matches from French passages only. Vector search still covers every language.
 - **Reciprocal rank fusion.** The vector pool and the keyword pool (same pool size, same filters and `ReadableBy`) are merged with `score = Σ 1 / (k + rank)`, where `k = Documents:Retrieval:Hybrid:RrfK` (default 60).
 - **Switch:** `Documents:Retrieval:Hybrid:Enabled`.
 
@@ -37,7 +37,7 @@ Success: the harness compares vector, hybrid, and hybrid + rerank on the same ru
   3. near-duplicate reduction and `topK` apply after reranking.
 - **Evidence gate:**
   - **Reranking on:** `MinScore` and `ConfidentScore` apply to the rerank score. Their settings are `Documents:Retrieval:Reranking:MinScore` and `…:ConfidentScore`, calibrated by the sweep.
-  - **Reranking off or skipped:** cosine is the gate, as today. A passage found only by keywords is admitted when its cosine is ≥ `MinScore`.
+  - **Reranking off or skipped:** cosine is the gate, as today, except for strong keyword matches: a passage whose keyword coverage is ≥ `Documents:Retrieval:Hybrid:MinKeywordCoverage` (default 0.5) passes the cosine floor, whichever pool found it. `ConfidentScore` still applies to its cosine, so evidence found by keywords alone is low-confidence. *Revised:* the floor used to drop every keyword-only hit below `MinScore`, which are the exact-term matches hybrid search exists for. One common word ("office") is not evidence, hence the coverage condition.
 - **Fallback:** on timeout (`Documents:Retrieval:Reranking:Timeout`, default 2 s), an error or an unreachable TEI, retrieval continues without reranking and is gated by cosine. The trace, activity and metrics record `rerank: skipped (unavailable)`. A question never fails because of the reranker.
 - **Switch:** `Documents:Retrieval:Reranking:Enabled`. The AppHost adds the TEI resource only with the launch profile `https-rerank` (or the parameter `reranking=true`), which also sets the switch. The default clone-and-run is unchanged.
 
@@ -77,7 +77,7 @@ Success: the harness compares vector, hybrid, and hybrid + rerank on the same ru
 | `Passage` (`documents.passages`) | `search_config regconfig NOT NULL` (from the document language), `search_vector tsvector` generated stored, GIN index. |
 | Migration | `CREATE EXTENSION unaccent`; `CREATE TEXT SEARCH CONFIGURATION french_unaccent (COPY = french)` with `unaccent, french_stem` for word tokens; backfill `search_config` from each passage's document language. |
 | `SearchFilters` (contract) | `QueryLanguage` (two letters, nullable). |
-| `RetrievedChunk` (contract) | `Stages` record: `VectorRank`, `Cosine`, `KeywordRank`, `KeywordScore`, `FusedScore`, `RerankScore` (all nullable), `AdmittedBy`. `Score` stays the gating score (rerank score when reranked, else cosine). |
+| `RetrievedChunk` (contract) | `Stages` record: `VectorRank`, `Cosine`, `KeywordRank`, `KeywordScore`, `FusedScore`, `RerankScore`, `KeywordCoverage` (all nullable), `AdmittedBy`. `Score` stays the gating score (rerank score when reranked, else cosine). |
 | `SearchResult` (contract) | `Mode` (`vector` \| `hybrid`), `Rerank` (`applied` \| `off` \| `skipped`). |
 | `RequestTrace` / trace passages (`chat`) | Stage fields per passage; `RetrievalMode`, `RerankStatus`, `RerankMs`, `RerankModel`. |
 
@@ -104,7 +104,9 @@ There are no new policies. Keyword and fused pools apply the same `ReadableBy(vi
 |---|---|
 | Query with only stop-words or punctuation (empty tsquery) | Keyword pool is empty; hybrid equals vector. |
 | French question, only English documents | No keyword hits; vector results only; `admittedBy = vector`. |
-| Keyword-only hit with cosine < `MinScore`, rerank off | Not admitted. |
+| Keyword match with cosine < `MinScore`, rerank off, coverage ≥ `MinKeywordCoverage` | Admitted; the outcome is `LowConfidence` unless another kept passage reaches `ConfidentScore`. |
+| Keyword match with cosine < `MinScore`, rerank off, coverage below it | Not admitted (`minScore`). |
+| Question sharing only some words with a passage | Still a keyword match (OR), ranked lower by `ts_rank_cd`; its coverage tells how much it shares. |
 | Keyword-only hit, rerank on, rerank score ≥ rerank `MinScore` | Admitted. |
 | TEI timeout, 5xx or connection refused | Fallback to cosine gating, `rerank: skipped (unavailable)`, a `documents.rerank.skipped` counter, a warning log without query text. |
 | TEI returns a score count different from the texts sent | Treated as unavailable (fallback). |
@@ -151,6 +153,9 @@ There are no new policies. Keyword and fused pools apply the same `ReadableBy(vi
 - **AC-7** Given a query of only stop-words, then hybrid returns the vector result and no error.
 - **AC-8** Given filters `DocumentIds`, tags and language, then the keyword pool applies the same filters.
 - **AC-9** Given `k = 60` and known ranks, then the fused score equals `1/(60+rv) + 1/(60+rk)` (unit test).
+- **AC-23** Given a passage holding three of the four lexemes of "budget BX-204 for travel" (`budget`, `bx`, `-204`, `travel`), when hybrid search runs, then it is a keyword match with coverage 0.75.
+- **AC-24** Given a passage containing every term of "budget BX-204 approval" with cosine 0.3 < `MinScore`, then hybrid search keeps it with outcome `LowConfidence`, and vector search returns `NoEvidence`.
+- **AC-25** Given a passage sharing only "budget" with that query (coverage 0.25) and cosine below `MinScore`, then it is dropped by the floor and the outcome is `NoEvidence`.
 
 ### Reranking
 

src/Modules/Documents/RagAssistant.Modules.Documents/DocumentsModule.cs

@@ -87,7 +87,7 @@ public static IServiceCollection AddDocumentsModule(this IServiceCollection serv
             .Validate(
                 o => o.IsConsistent,
                 "Documents:Retrieval: DefaultTopK must not exceed MaxTopK, MaxCandidates must be at least MaxTopK, "
-                + "ConfidentScore must be at least MinScore, and both durations must be greater than zero; Hybrid:RrfK must be at least 1; "
+                + "ConfidentScore must be at least MinScore, and both durations must be greater than zero; Hybrid:RrfK must be at least 1 and Hybrid:MinKeywordCoverage between 0 and 1; "
                 + "Reranking:Candidates must be 1 to 100, its scores between 0 and 1 with ConfidentScore at least MinScore, and its "
                 + "Timeout greater than zero.")
             .ValidateOnStart();

src/Modules/Documents/RagAssistant.Modules.Documents/Features/Admin/AdminDocumentsModels.cs

@@ -70,4 +70,5 @@ internal sealed record RetrievalStagesConfiguration(
     int RerankCandidates,
     double RerankMinScore,
     double RerankConfidentScore,
-    double RerankTimeoutSeconds);
+    double RerankTimeoutSeconds,
+    double MinKeywordCoverage);

src/Modules/Documents/RagAssistant.Modules.Documents/Features/Admin/GetDocumentsConfiguration.cs

@@ -53,6 +53,7 @@ public ValueTask<Result<DocumentsConfiguration>> Handle(Query query, Cancellatio
                     retrieval.Value.Reranking.Candidates,
                     retrieval.Value.Reranking.MinScore,
                     retrieval.Value.Reranking.ConfidentScore,
-                    retrieval.Value.Reranking.Timeout.TotalSeconds)));
+                    retrieval.Value.Reranking.Timeout.TotalSeconds,
+                    retrieval.Value.Hybrid.MinKeywordCoverage)));
     }
 }