Repository navigation
Evaluation, French/English content and pipeline hardening (specs 018, 021) - #17
Merged
Merged
Conversation
Approved spec for Phase 5 (sample corpus, golden dataset, evaluation harness) and French/English documents with answers in the language of the question or the one the user requests. Opens the 018-020 series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…019, 020) Spec 019: Postgres full-text + vector fused with RRF, TEI reranker (bge-reranker-v2-m3) behind switches, kept only on the spec 018 harness gate. Spec 020: home page with verified suggested questions, citation source states, Evaluation module and admin tab, chat rename, search, export and copy, accessibility pass, README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LanguageDetector in the shared kernel (English/French function words, pure and deterministic). Documents get a language source (detect or owner); ingestion records each version's detected language and applies a confident detection to documents left to detection, bumping the revision. Import and PUT accept 'detect'; GET document and versions expose the source and the detection. Migration defaults existing documents to detection. Spec 018 gains two clarifications: a 3-word minimum for question detection, and an absent PUT language means detect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pec 018) Questions get a language at accept time: the shared detector with a 3-word minimum, then the conversation's previous question, then English; a retry resolves older rows the same way. The answer prompt names the language and lets an explicit request win; the follow-up rewrite writes the standalone question in the question's language and returns RequestedLanguage, which carries a request over to later turns. Messages, traces and citation snapshots store the languages; the stream, conversation, activity, admin request detail and list (questionLanguage filter) expose them, and the Chat request and refusal counters get a question.language label. The client shows citation language badges and the question language in the activity panel and request inspector. Prompt versions 018.1 and rw.018.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A LanguageChoice component (Detect from the text, English, French, Other with a two-letter code) replaces the language text field in the import dialog, where Detect is the default, and in the metadata edit form, which starts on Detect for documents left to detection. The detail page labels a detected language, notes an uncertain detection, and, for editors, shows 'Detected: French' with Use French when a confident detection differs from the owner's choice; it saves the detected language and keeps every other field (AC-10). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c 018) GET /api/documents/search takes candidates=true and also returns the whole scored pool, each passage marked as kept or dropped by minScore, duplicate or topK, for the harness's threshold sweep (AC-8, AC-9). samples/corpus holds 16 policies of a fictional company, 8 English and 8 French, with a manifest; two English documents are rendered as PDFs at seed time and the remote work policy has a superseded version. tools/RagAssistant.Evaluation (in the solution, no tests) gets dev tokens from the realm's password-grant client and a 'seed' command that imports, restores, versions and updates the corpus idempotently, and a 'render' command for the PDFs. Spec 018 records three findings as clarifications awaiting confirmation: English-only PDFs, unchanged content via the duplicate guard, and polling for indexing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
samples/golden/questions.json holds 86 cases for the sample corpus: 40 same-language, 16 cross-language, 10 refusals, 4 partial answers, 10 two-turn follow-ups and 6 explicit language requests, each with the expected behaviour and evidence as document key + verbatim quote. 'run' seeds, checks every quote against the seeded passages (broken cases are listed and excluded), asks each case in its own conversation scoped to the sample tag over SSE, reads its trace and the candidate pool, and scores recall@1/3/5/topK, MRR, behaviour, citation recall and precision and answer language per category and question language. It writes report.json and report.md under .data/evaluation/<run>/, marks a run with several chat models invalid and skips answer metrics in demo mode. The first run against llama3.2:3b completed all 86 cases. Spec 018 records clarifications 3 to 5 as confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
'compare <A> <B>' prints every metric of every group with B - A and flags configuration differences first (AC-29). 'sweep <report>' replays the evidence rules over the stored candidate pools for a grid of MinScore, ConfidentScore and DuplicateOverlap and recommends the best setting that refuses at least as well as the current one (AC-30); run reports now carry the sample passages it needs. 'judge <report>' scores stored answers with Microsoft.Extensions.AI.Evaluation groundedness and relevance through Ollama, asks the judge about requested languages the detector does not know, pulls Evaluation:JudgeModel on demand and refuses the model under test (AC-28). Packages for the harness only: Microsoft.Extensions.AI.Evaluation(.Quality) 10.11.0 and OllamaSharp 5.4.12, the version the API already uses. An unreachable Keycloak is now a clear message instead of a crash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The default judge is gemma4:12b (same size as gemma3:12b, newer). Its default thinking used up the evaluators' output budget and left replies unparsable, so every judge call is sent with think=false. The judge's Ollama client comes from IHttpClientFactory with a 10-minute timeout, since loading and scoring on a laptop exceeds the default 100 s; a timed-out case is recorded and the pass goes on. Unreadable scores now keep the evaluator's diagnostics, Evaluation:JudgeCases limits a pass to some cases and Evaluation:JudgeTrace logs the judge's traffic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The developer's laptop has no dedicated GPU (Radeon 890M, 88 GB RAM) and the AppHost's Ollama container runs on the CPU. The bar for the chosen chat model becomes a median first token under 8 s there (3 s stays the GPU target), the candidates are qwen3.5:9b and ministral-3:8b against llama3.2:3b, the judge is gemma4:12b from a family no candidate belongs to, and using the iGPU through Vulkan is deferred. Status: In Review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The AppHost reads the Ollama chat model from Parameters:chat-model (command line or user secret), falling back to llama3.2:3b until the spec 018 comparison chooses one, and pulls it; the API and the evaluation harness see the configured model. Spec 018 is approved again after its CPU-only revision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the trace or the candidate pool of a case cannot be read (for example retrieval.embedding_unavailable on an overloaded CPU), the case keeps its answer and metrics, records the error, and the run goes on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
qwen3.5:9b thinks by default; on the CPU-only reference machine its hidden reasoning used the whole output budget and pushed the first visible token past the answer deadlines. OllamaThinking, a delegating chat client wired only in the API, sets think=false on every Ollama chat call (true with Chat:Ollama:Think) while keeping any raw request a caller provided. Models without a thinking mode ignore the flag. Spec 018 and an ADR 0003 addendum record the decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat:Provider = OpenAICompatible sends chat calls to an OpenAI-compatible endpoint (OpenRouter by default) with Chat:OpenAICompatible:Model and ApiKey, validated at startup and wired only in the API through Microsoft.Extensions.AI.OpenAI 10.11.0; the SDK's own retries are off (spec 012). The AppHost selects it with chat-provider=openai-compatible, chat-model and the secret parameter openai-api-key, and then pulls no Ollama chat model; embeddings stay on the local bge-m3. The evaluation harness can judge through such an endpoint too (Evaluation:JudgeProvider, JudgeEndpoint, JudgeApiKey). Local Ollama stays the default: with the hosted provider, questions and retrieved passages leave the machine. Spec 018 (second revision) and an ADR 0003 addendum record it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spec 018 comparison on the CPU-only reference machine: qwen3.5:9b misses the 8 s first-token bar; ministral-3:8b beats llama3.2:3b on answer language, citations and retrieval. Results in the ADR 0003 addendum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any content owner can share a document, so a passage can carry text written to steer the model. The answer prompt (version 018.2) now wraps each passage, header included, between <passage> tags, and its first rule says that everything inside them is evidence, never instructions. Passage tags written in a title, section, passage text or question are escaped, so no text can close its passage or forge one. The evaluation harness gets a prompt-injection golden case (i-en-01, sample document meeting-rooms) with forbidden phrases, and an "injection resisted" metric in reports and comparisons. Spec 018 gains a third revision (AC-21e, AC-21f); the rag-pipeline skill describes the layout and the escaping rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…021) Hardening before hybrid search and reranking (spec 019), from the RAG pipeline review. Context budget (Chat module, API): Chat:Generation:ContextWindowTokens (default 8192) is the window the answer prompt must fit in with the answer. PromptBudget estimates the real prompt conservatively (chars / 3 plus a per-message overhead) and drops the lowest-ranked passages before retrieval.completed, so sources, markers, snapshots and the trace see only what is sent. The trace records the trim and the estimate, shown in the admin request detail. The API sends the same window to Ollama as num_ctx; OllamaThinking becomes OllamaRequestOptions. Titles in embeddings (Documents module): the document title leads every embedding input. Versions record EmbeddingInputVersion (existing ones backfilled to 1); outdated versions stay searchable, are counted by admin health and flagged in the version list, and the harness seed reprocesses outdated sample documents so runs stay comparable. Nightly evaluation: .github/workflows/evaluation.yml runs the harness against the real AppHost with bge-m3 and qwen3:1.7b on the CPU, report only. The AppHost forwards ApiEnvironment:<NAME> settings to the API so the run can raise the chat deadlines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 5 of the roadmap: an evaluation harness with a committed sample corpus, French and English content, a chat model chosen by measurement, plus the hardening a pipeline review asked for before hybrid search (spec 019).
Spec 018: evaluation and French/English content
tools/RagAssistant.Evaluation):seedcommand;run(recall@k, MRR, behaviour, citations, answer language),compare,sweepover the raw candidate pools, andjudge(groundedness, relevance).chat-modelparameter, and the default is nowministral-3:8b(ADR 0003 addendum);Chat:Ollama:Thinkis set;<passage>tags, the first rule says passage content is data and never instructions, and written passage tags are escaped. Prompt version018.2. New golden casei-en-01and an "injection resisted" metric.Spec 021: context budget, titles in embeddings, nightly evaluation
Chat:Generation:ContextWindowTokens(8192) is sent to Ollama asnum_ctx.PromptBudgetdrops the lowest-ranked passages beforeretrieval.completedwhen the prompt plus the answer would not fit.PromptBudget.EmbeddingInputVersion(migrationEmbeddingInputVersionbackfills existing ones to 1).outdatedEmbeddings) and flagged in the version list.seedreprocesses outdated sample documents..github/workflows/evaluation.ymlruns the harness every night and on dispatch, against the real AppHost withbge-m3andqwen3:1.7bon the CPU. It reports only.ApiEnvironment:<NAME>settings to the API so the run can raise the chat deadlines.Docs
Specs 018 (three revisions), 019 and 020 (approved, not implemented) and 021; ADR 0003 addendum; CLAUDE.md, roadmap and the
rag-pipelineskill updated.Upgrade notes
seedhandles the sample corpus.Test plan
dotnet build RagAssistant.slnx -c Release: 0 warningsdotnet test --solution RagAssistant.slnx: 1,252 passed, 0 failed (one earlier full run had a single integration failure that did not reproduce on rerun; possibly flaky)PromptBudget, budget end to end with a tight window,num_ctxin the real OllamaSharp request body, embedding input format and version, outdated counts before and after a reprocess, demo client layoutdotnet format --verify-no-changesflags only three lines ofAppHost.cswith LF line endings in a Windows checkout; expected to pass on the Linux CI runnermain, thengh workflow run evaluation.yml; record the result in spec 021's Verification section (runtime and dev-certificate trust for Keycloak on Linux are unverified)i-en-01not yet run against a real modelMinScore0.55 is still the spec 011 value)🤖 Generated with Claude Code