Trust Your RAG — retrieval, risk tiering, answer generation, judging, eval, and a chat UI (v1)

An implementation of the Insurance Policy Q&A tool described in Insurance_RAG_Trust_PRD.pdf: upload a policy document, ask it questions in a chat interface, get answers whose caution level matches the actual risk of being wrong, and every answer is scored live against the PRD's own rubric before you see it — plus eval/, the batch tooling used to check that claim across a whole labeled test set rather than one question at a time.

Retrieval and risk tiering (ragindex/, risk/) run fully offline. Answer generation, judging, eval, and the chat UI (answer/, judge/, eval/, server/) call a real LLM and need credentials for whichever provider you use — ANTHROPIC_API_KEY (or an ant auth login profile) for Claude, the default; GEMINI_API_KEY (or GOOGLE_API_KEY) plus the optional google-genai package for Gemini. Set these once via a .env file rather than exporting them every session:

cp .env.example .env
# then edit .env and fill in your key(s)

run.sh loads .env automatically (it's gitignored — never commit it).

What this does, end to end

The PRD describes a 6-step workflow (section 4). Here's what each step means in plain language, and exactly what in this repo implements it:

#	PRD step	What that means	Implemented as	Status
1	User uploads a policy document + background info	Turn a PDF/Markdown policy into something searchable, and record facts about the user (conditions, scheduled procedures, dependents)	./run.sh index <file> (ingest → chunk → TF-IDF index) + a background JSON file passed via --background	✅ Done
2	User asks a question in natural language	Just a question string	Any --q "..." / positional question argument on every CLI	✅ Done
3	Risk classifier assigns Low/Medium/High	Decide how cautious the answer needs to be, based on the question, the retrieved passages, and the user's background — not just the question's wording	risk/ package, ./run.sh classify	✅ Done
4	Answer generated per tier-specific logic	Low: direct answer + citation, or an honest "not found." Medium/High: clarifying questions, a terminal state if the gap can't close, and any general knowledge clearly walled off from document facts	answer/ package, ./run.sh answer	✅ Done
5	LLM judge screens the answer before delivery	A safety check that runs on every real answer before a user ever sees it, adding disclaimers for money/high-stakes cases	judge/ runs automatically after every generation, in the CLI and the chat UI (gate_answer, default on) — every real answer is scored against the PRD's own rubric (Grounded, Complete, Correctly tiered, Correctly routed, Honest on terminal state), and judge/gate.py now acts on the verdict: a Grounded/Honest-on-terminal-state failure silently regenerates (up to 3 attempts) and falls back to a "contact support" message if it never clears; a Complete-only failure still delivers the answer with a caveat note appended	✅ Done (screening + gating; still just the LLM's own rubric check, no independent hallucination detector)
6	Answer routed to the user, or escalated with consent	Either just show the answer, or offer a human callback and act on the user's yes/no	The chat UI (./run.sh serve) or any CLI. "Yes" appends a ticket record to escalations.jsonl — there's no real CRM behind it, and a decline gets an explicit statement, never silence	✅ Done

Beyond the PRD's 6 steps, this repo also has eval/ — a batch runner over the PRD's own 31-question Appendix test set, so tier accuracy and rubric pass rates can be checked in one command instead of by hand (which is how every one of the above pieces was actually validated while building this).

Project layout

ragindex/     Ingest a policy doc, chunk it, build/query a TF-IDF index. Offline.
risk/         Classify a question's risk tier (Low/Medium/High) with reasoning. Offline.
answer/       Generate a tier-appropriate answer via a pluggable LLM (Claude/Gemini).
judge/        Score a generated answer against the PRD's 5-dimension rubric.
eval/         Batch-run the PRD's 31-question test set and report results.
server/       FastAPI backend for the chat UI -- a thin HTTP wrapper around answer/.
web/          Static chat frontend (HTML/CSS/JS, no build step, no framework).
debugging.py  Shared --debug plumbing (bool parsing, env-var fallback).
data/         Sample policy doc + sample user-background JSON for smoke testing.
tests/        Unit tests for every package above — all offline, no API calls.
run.sh        Single entry point for every command below (handles venv + .env).

Each package's internals are documented in "How it works" further down — this table is just for finding your way around.

Usage

run.sh handles the venv setup automatically the first time you use it — no manual pip install needed.

1. Drop your policy document (.md, .txt, or .pdf) into data/.

2. Build the index (do this once, and again any time the document changes):

   ./run.sh index data/<filename>

Fastest way to try it: the chat UI. Once the index is built and a key is in .env:

./run.sh serve

Open http://127.0.0.1:8000. Ask a question, fill in the "Your background" panel on the left to see the same question land on a different risk tier, and watch clarifying questions / callback offers / citations render live. See "Chat UI" in How it works below for exactly what it's doing and its current limitations. Everything from here on is the CLI equivalent of the same pipeline, useful for scripting or when you want the raw output.

3. Ask it questions:

   ./run.sh ask "What's the maximum payout for a cancer diagnosis?"

   Optionally pass a number to change how many chunks come back (default 5): ./run.sh ask "your question" 3

Each result shows a similarity score, the source document, a heading/section label (used for citations), and a text snippet.

index/ holds whatever was built last — if ./run.sh index ... fails (e.g. wrong filename), a following ask will silently reuse the old index. If results look stale, check that the index step actually succeeded.

4. Classify a question's risk tier (Low/Medium/High), optionally with a user background JSON file:

   ./run.sh classify "Is this a network hospital?"
   ./run.sh classify "Is this a network hospital?" data/sample_background.json

   See data/sample_background.json for the expected shape (existing conditions, scheduled procedures with dates, dependents, claims in dispute, and a free-form provided_facts map).

5. Generate an actual answer (classifies the tier, then calls Claude to produce a tier-appropriate response). Requires ANTHROPIC_API_KEY (or an ant auth login profile) to be set — this is the one step that calls a real LLM and has a cost per call:

   ./run.sh answer "What's the maximum payout for a heart attack?"
   ./run.sh answer "Is this a network hospital?" data/sample_background.json

   For a Medium/High question, if the model says it needs more information, either add the answer to your background JSON's provided_facts and re-run, or re-run with --cannot-clarify to force the terminal-state branch (PRD: "states explicitly what would need to be true... does not guess to close the gap"). Once the model offers a human callback, re-run with --consent yes (creates a ticket, logged to escalations.jsonl) or --consent no (prints citations + an explicit "can't fully resolve this" statement — never silence):

   ./run.sh answer "If I go to a non-network hospital, will I still get close to the full benefit?" \
     data/sample_background.json --consent yes

   To use Gemini instead of Claude, add --provider gemini (run.sh auto-installs the optional google-genai package the first time you use it) and set GEMINI_API_KEY in .env (or GOOGLE_API_KEY):

   ./run.sh answer "What's the maximum payout for a heart attack?" --provider gemini

   --model MODEL_ID overrides either provider's default model.

   Add --debug yes (or 1) to see exactly what went into and came back from the LLM call — the full system prompt, the full user content (including every retrieved excerpt, not just the one that got cited), the JSON schema, and the raw JSON response — printed to stderr before the CLI's tier-specific rendering trims it down:

   ./run.sh answer "Am I covered against cancer?" --provider gemini --debug yes

   Accepts yes/no, 1/0, true/false (case-insensitive). Falls back to the DEBUG env var (settable in .env) if --debug isn't passed, so you can leave it on for a session without adding the flag every time. Off by default either way.

6. Score a generated answer against the PRD's own scoring rubric — same flags as answer (--background, --cannot-clarify, --provider, --model, --debug; no --consent, since scoring doesn't need it):

   ./run.sh judge "Am I covered against cancer?" --provider gemini

   Prints the generated answer, then a [PASS]/[FAIL] verdict + specific reasoning for each of the five rubric dimensions — never a single collapsed score. Useful for exactly the kind of before/after comparison this project has been doing by hand throughout — now automated.

7. Run the PRD's full ~31-question Appendix test set through the pipeline and get a report. This makes 2 LLM calls per question (generate + judge) — ~62 calls for the full set — so start small:

   # Cheap first: 1 question, 2 calls
   ./run.sh eval --provider gemini --bucket "Low - Happy Path" --limit 1
   
   # One PRD bucket at a time
   ./run.sh eval --provider gemini --bucket "Medium"
   
   # The full set (default provider is claude; override with --provider)
   ./run.sh eval

   Valid --bucket values: "Low - Happy Path", "Low - Not Found", "Medium", "High" (see eval/testset.py). Prints a compact progress line per question as it runs:

   [LOW    = LOW   ] 0/5 rubric dim(s) failed  --  What's the maximum payout for a cancer diagnosis?
   

   = means the tier matched exactly what the PRD expects; > means it came back stricter (acceptable per the PRD's own rubric); < means looser (the actual failure direction). Then a summary table grouped by PRD bucket — tier-match rates and a pass-count per rubric dimension — and the full per-question detail (complete rendered answer + every judge verdict's reasoning) gets written to eval_report.json (--out to change the path). One question erroring (rate limit, network blip) doesn't kill the run — it's recorded with its error and excluded from the summary table, and the rest continue.

   Same --provider, --model, --background, --debug flags as answer/judge.

8. Run the unit test suite:

   ./run.sh test

Running the underlying CLIs directly (optional)

./.venv/bin/python -m ragindex.cli build --input data/<filename> --out index
./.venv/bin/python -m ragindex.cli query --index index --q "your question" --k 5 --full
./.venv/bin/python -m risk.cli --index index --q "your question" --background data/sample_background.json
./.venv/bin/python -m answer.cli --index index --q "your question" --background data/sample_background.json --consent yes
./.venv/bin/python -m judge.cli --index index --q "your question" --background data/sample_background.json
./.venv/bin/python -m eval.cli --index index --bucket "Low - Happy Path" --limit 1
./.venv/bin/python -m uvicorn server.app:app --reload
./.venv/bin/python -m pytest -q

--full on query prints the whole chunk instead of a 400-character snippet.

How it works

• ragindex/loader.py — splits markdown on #/##/... headings (each section becomes one citable unit), or extracts PDF text page-by-page (labeled Page N, with a best-effort heading hint if the page's first line looks like a section header).
• ragindex/chunker.py — splits each section into ~220-word chunks with 40-word overlap. Chunks never cross a section boundary, so every chunk has one clean citation label.
• ragindex/indexer.py — TF-IDF (scikit-learn, word + bigram features) over chunk text, cosine similarity at query time. No embedding model downloads, runs fully offline.

Risk classifier (risk/)

Per PRD section 5: tier is a function of the question, the retrieved passages, and the user's background — not the question's phrasing alone. Each PRD anchor maps to one module, so extending or swapping any signal never touches the others:

• risk/models.py — shared dataclasses (RiskTier, UserBackground, ScheduledProcedure, evidence types, TierResult). No dependency on ragindex or sklearn, so the rest of the package is unit-testable without a real index.
• risk/dollar_exposure.py — the "scheduled procedures, major diagnoses, claims in dispute" anchor. Reads only UserBackground; a scheduled procedure within 30 days (or undated) counts as exposure regardless of what the question asks, per the PRD's own example ("is this a network hospital" is high-risk if surgery is scheduled next week, independent of phrasing).
• risk/ambiguity.py — the "retrieval conflict" and "multi-section reconciliation" anchors. Computed from retrieval scores only: a chunk counts as a genuine competing candidate if its score is at least half the top match's (not merely nonzero — see calibration notes below).
• risk/fact_requirements.py — the "missing user fact" anchor. A flat, inspectable list of (question pattern -> required background field); extending coverage means adding a row, not touching the classifier.
• risk/retriever.py — a Retriever Protocol plus TfidfRetriever, which adapts ragindex.indexer.Index to it. Keeps risk/ decoupled from the retrieval implementation (tests inject a fake; a future embedding-based retriever is a drop-in swap).
• risk/classifier.py — resolve_tier (pure decision logic, unit-tested directly with fabricated evidence) and classify (orchestrator: runs retrieval, gathers all signals, calls resolve_tier). Any concrete anchor -> HIGH; otherwise a retrieval-confidence threshold decides Low vs. Medium, rounding up when it's close to the line — per the PRD's "round up, never down" misclassification default.

Tests: tests/test_signals.py (each signal in isolation) and tests/test_classifier.py (tier resolution + classify wiring, via a FakeRetriever — no real index needed).

Answer generation (answer/)

Per PRD section 6, split by tier into two independent generation paths (a Low-risk question never runs the Medium/High prompt and vice versa), and decoupled from any specific model provider:

• answer/generator.py — generate_low_risk_answer (direct answer + citation, not_found branch that refuses to answer from general knowledge) and generate_medium_high_answer (clarifying questions, terminal state, ambiguity notes, and a general_context field kept structurally separate from document_answer — the model is instructed to never blend the two into one sentence or claim general knowledge is verified against the user's policy). This module builds prompts and JSON schemas and never imports a model SDK — it calls whatever AnswerModelProvider it's given.
• answer/providers/ — the swap point:
  • base.py — AnswerModelProvider, a one-method Protocol (generate_structured(system, user_content, schema) -> dict). Any backend that can turn a prompt + JSON schema into a parsed dict can plug in here.
  • claude.py — ClaudeProvider, the default. Uses the official anthropic SDK (claude-opus-4-8) with a JSON-schema output_config.format, so the response parses deterministically.
  • gemini.py — GeminiProvider, using Google's google-genai SDK (response_mime_type="application/json" + response_schema). Gemini's structured-output schema is a stricter OpenAPI-style subset of JSON Schema than Claude's (no additionalProperties), so this provider strips the unsupported keys recursively before sending — the schema itself is still defined exactly once, in generator.py, for every provider. google-genai is an optional dependency (requirements-gemini.txt), only needed if you actually use --provider gemini.
  • Selected in answer/cli.py via --provider claude|gemini (default claude) and --model (overrides either provider's default model ID).
• answer/escalation.py — create_ticket appends a record to escalations.jsonl when the user consents to a human callback. No real CRM integration in v1 (same non-goal as the PRD's stance on live insurer data) — it's a local, inspectable stand-in for "ticket created."
• answer/render.py — pure, side-effect-free functions turning a LowRiskAnswer/MediumHighAnswer + tier info into the exact text a user sees. Deliberately excludes consent resolution (ticket creation / decline messaging) — that's a runtime CLI workflow driven by a flag the model never saw, not part of what the tool itself produced. Factored out so judge/ can score precisely what a user would see, not raw fields.
• answer/pipeline.py — run_pipeline: retrieval -> classify -> generate -> render, in one place. Used by both answer/cli.py and judge/cli.py so the retrieval/classifier/generator wiring isn't duplicated between them. Also holds build_provider (the --provider claude|gemini switch) and PrecomputedRetriever (shares one TF-IDF query between classification and generation).
• answer/cli.py — thin now: parses args, builds the provider (optionally wrapped in DebugLoggingProvider), calls run_pipeline, prints the result, and resolves --consent if a callback was offered.

Design decisions worth knowing:

• Clarifying questions are asked as a batch, not turn-by-turn. The CLI is single-shot (no session state), so instead of an iterative back-and-forth, the model asks up to 3 questions at once. Answer them by editing your background JSON's provided_facts and re-running, or force the terminal-state branch with --cannot-clarify if you can't supply more.
• Consent is a two-step CLI interaction, not a live prompt. The first answer call surfaces the callback offer; a second call with --consent yes/--consent no resolves it. There's no server or session to hold that state in between.
• The web-citation question is deliberately left open in the PRD itself ("Open Questions" #1: "is the web-citation feature in v1 at all, or deferred"). This build takes the side of including it, since the task asked for "external content clearly separated and labeled": the model may populate general_context from its own knowledge, but there's no live web-search tool wired in — it's the model's own training knowledge, always labeled as unverified against the policy. If you want it deferred entirely instead, that's a one-line prompt change in answer/generator.py's MEDIUM_HIGH_SYSTEM.

Tests: tests/test_answer_generator.py (prompt construction + response parsing, via a fake AnswerModelProvider — no model SDK involved at all), tests/test_providers.py (each provider's request-shaping logic, via fake Anthropic/Gemini clients), tests/test_render.py (every render branch), and tests/test_pipeline.py (retrieval -> classify -> generate wiring, against a real tiny Index built in a temp dir, with a schema-aware fake provider). No real API calls, no cost, in any of them.

Chat UI (server/, web/)

A chat interface on top of answer/pipeline.py — no new pipeline logic, just an HTTP + browser skin over what already existed. Deliberately split into two independently-deployable pieces from the start:

• server/app.py — a FastAPI backend, three routes:
  • GET /api/health — liveness check.
  • POST /api/chat — the main endpoint: {question, background?, provider?, model?, k?, cannot_clarify?, judge?, max_attempts?} in, and the same PipelineResult the CLIs print (tier, tier_reasoning, rendered, needs_clarification, clarifying_questions, terminal_state, offer_human_callback) out as JSON, plus judge_verdicts (one {dimension, criterion, passed, reasoning} per rubric dimension), judge_error, and consistency ({consistent, disagreements} from the self-consistency check, or null — always null for a Low-tier answer, since that check isn't computed for Low at all; see "Judge gate" below). judge defaults to true — every answer goes through the full judge/gate.py gate (rubric judge, plus self-consistency for Medium/High) before the response goes out, same as the CLI's --judge default: a blocking failure silently regenerates (up to max_attempts, default 3) and rendered becomes the support-fallback message if it never clears; a Complete-only failure still delivers the answer with a caveat note appended. A failed judge call (as opposed to a failing verdict) sets judge_error instead of failing the whole request — a good answer still reaches the user even if scoring it didn't work that time. Loads the Index once at startup from INDEX_DIR (default index/) and fails fast with a clear message — not a bare traceback — if you haven't run ./run.sh index yet.
  • POST /api/consent — resolves a previously-offered human callback ({question, consent: "yes"|"no"} → {message}). See the bug writeup below for why this is its own endpoint instead of a flag on /api/chat.
  • Mounts web/ as static files at /, so ./run.sh serve is one process, one port, for local hosting. CORS is wide open (allow_origins=["*"]) — fine for local dev, tighten it before this is reachable from anywhere else.
• web/ — index.html + style.css + app.js, no build step, no framework, no dependencies. app.js's API_BASE constant is the one line to change if the frontend and backend ever get deployed separately (e.g. this static folder on Vercel, the API elsewhere) — empty string means "same origin as this page," which is what makes local hosting a single command.
  • The sidebar's background form (existing conditions, a scheduled procedure + date, dependents, claims in dispute) is sent with every message, so you can watch the same question land on a different risk tier live — the PRD's own "is this a network hospital" example, directly interactive.
  • A "Run judge on every answer" checkbox (checked by default) sends judge in the request. When verdicts come back, a collapsed <details> panel under the answer shows a one-line pass/fail summary in its <summary> and, expanded, PASS/FAIL + reasoning for each rubric dimension plus one more row for the self-consistency check — collapsed by default so it doesn't crowd out the answer itself, but always there. If the answer you're looking at needed a silent retry or two, or got replaced with the support-fallback message, this panel is where you'd see why — the retry loop itself is still invisible, but its conclusion isn't hidden.
  • Clarifying questions and consent are single-shot backend concepts (see answer/'s design decisions above) wearing chat clothing: when needs_clarification comes back, the next message you type gets merged with the original question client-side before it's sent — it feels like a follow-up turn, but it's still one stateless /api/chat call. When offer_human_callback comes back, Yes/No buttons render inline and call /api/consent directly.

A real bug this build caught, worth knowing about — not just a formality: the first version resolved consent by re-running the entire generation pipeline a second time with consent as an extra field on /api/chat (mirroring how answer/cli.py already worked). Manual end-to-end testing against a real model surfaced the problem immediately: the second call is an independent LLM call, and nothing guarantees it lands on the same offer_human_callback=true state as the first. In testing, a real re-ask came back with needs_clarification instead — clicking "Yes, schedule a callback" silently did nothing. Fixed by giving consent resolution its own endpoint that never touches the LLM at all (ticket creation doesn't need one), and applied the identical fix to answer/cli.py's --consent handling, which had the exact same latent bug and had simply never been exercised twice in a row with a real model before. answer/pipeline.py's resolve_consent is now the one place both the CLI and the server call.

Known limitations specific to the chat UI:

• No real conversation memory server-side — every /api/chat call is still the single-shot pipeline underneath; the "merge with the original question" trick approximates continuity but isn't true multi-turn state.
• The background form supports one scheduled procedure at a time (the underlying UserBackground model supports a list; the form just doesn't expose more than one slot yet).
• No auth, no per-user session, no rate limiting — it's a local dev tool. Don't point allow_origins=["*"] at the public internet as-is.
• Verified via curl against the real running server (including a real LLM round trip and the background-driven tier change) and the TestClient-based test suite below — not verified in an actual browser window, since no browser-automation tooling (Playwright, chromium-cli) was available in the environment this was built in. If something looks visually off, that's the untested part.

Tests: tests/test_server.py, using FastAPI's TestClient and the same schema-aware fake-provider pattern as tests/test_pipeline.py — covers health, validation errors, a full chat response, clarifying questions, the consent endpoint (including that it works with no provider configured at all, proving it never touches the LLM), and that background data actually changes the tier through the real classifier.

Debug framework (debugging.py, answer/providers/debug.py)

The backbone for inspecting exactly what any LLM call actually sent and received, meant to be reused for every provider added later, not just the two here:

• debugging.py — provider-agnostic plumbing: parse_bool_flag (accepts yes/no, 1/0, true/false, case-insensitive — used as an argparse type=), is_debug_enabled (CLI value wins, else the DEBUG env var, else off), and debug_section (consistent, greppable formatting for debug output).

• answer/providers/debug.py — DebugLoggingProvider, a decorator that wraps any AnswerModelProvider and logs the full input (system prompt, user content, JSON schema) and full output (raw parsed JSON) to stderr around the call, then returns the result unchanged. It only depends on the AnswerModelProvider protocol, so it works on Claude, Gemini, or any future provider with zero changes to that provider's own code — answer/cli.py just conditionally wraps whichever provider it built:

  if is_debug_enabled(args.debug):
      provider = DebugLoggingProvider(provider, label=args.provider)

  Debug output goes to stderr, not stdout, so scripting against the answer's stdout output still works with --debug on.

Tests: tests/test_debugging.py (bool parsing + env precedence) and tests/test_debug_provider.py (the decorator passes results through unchanged, logs input even when the wrapped provider raises).

Judge (judge/)

Scores a generated answer against the PRD's own Appendix: Scoring rubrics — five independent dimensions (Grounded, Complete, Correctly tiered, Correctly routed, Honest on terminal state), each with a worked pass/fail example straight from the spec. Returns one verdict per dimension, never a single collapsed score — a fail on one dimension is not supposed to influence the verdict on another. Runs automatically at runtime now (PRD section 7's "an LLM judge screens every answer before delivery"), not just as a manual eval step — every real ./run.sh answer and every /api/chat call gets scored before you see the output, default on.

• judge/rubric.py — the rubric as data: each RubricDimension carries the PRD's exact example question, pass example, fail example, and why the fail example fails. This is the only file you'd edit to change what "pass" means for a dimension — judge.py just renders whatever's here into a prompt.
• judge/judge.py — two entry points:
  • judge_answer(question, tier, tier_reasoning, context_excerpts, rendered_answer, provider, generation_system_prompt=""): builds a system prompt from every dimension's worked example (as calibration, not as data to reuse in the verdict), a JSON schema with one object per dimension ({passed, reasoning}), and calls the same AnswerModelProvider protocol answer/generator.py uses. Since this build now runs the judge automatically, generation_system_prompt -- the exact LOW_RISK_SYSTEM/MEDIUM_HIGH_SYSTEM text the generator was instructed to follow -- is passed as an extra artefact, so the judge can check for a specific rule violation (e.g. general knowledge blended into a citation sentence when told never to), not just surface plausibility.
  • judge_pipeline_result(question, pipeline_result, provider) — the runtime entry point: takes the PipelineResult run_pipeline just produced, picks the right generation system prompt for that tier, and calls judge_answer with everything extracted. One place answer/cli.py, server/app.py, and eval/runner.py all call, instead of each reconstructing the same composition.
• judge/models.py — DimensionVerdict (dimension, passed, reasoning) and JudgeResult (the verdicts list is the primary output; failed / all_passed are conveniences, not a replacement for reading the per-dimension reasoning).
• judge/render.py — render_verdicts(result) -> str, the [PASS]/[FAIL] + reasoning text shown by answer/cli.py and judge/cli.py (the same fields go out as JSON from server/app.py for the chat UI, not through this function).
• judge/cli.py — the standalone manual version: runs run_pipeline, then judge_pipeline_result, printing verdicts. Useful for a one-off check without needing to re-run answer with --judge on.

Validated against real model calls, not just fakes — with a deliberately bad answer closely mirroring the PRD's own "Grounded" FAIL example (omitting the recurring-cancer exclusion present in the source excerpt), the judge correctly returned [FAIL] Grounded and [FAIL] Complete with reasoning nearly identical to the PRD's own stated rationale, while still correctly passing Tiering/Routing/Honesty, which genuinely weren't at fault. That's the important property for a judge: it discriminates, it doesn't rubber-stamp.

Known simplification: generation and judging use the same provider (and model) by default in every caller. A judge scoring its own generator's output is a real bias risk generally — using an independent/stronger model as judge is the more rigorous setup. Nothing in judge_answer's or judge_pipeline_result's signatures forces this; pass a different provider instance for judging if you want that separation (e.g. Gemini generates, Claude judges, or vice versa) — none of the current callers wire that up by default yet.

Tests: tests/test_rubric.py (the rubric data itself — five dimensions, unique keys, every dimension has a complete worked example) and tests/test_judge.py (schema/prompt construction, verdict parsing and ordering, the generation-system-prompt artefact, and judge_pipeline_result's tier-based prompt selection — via a fake provider, no real model calls).

Judge gate (judge/gate.py) — turning verdicts into a delivery decision

judge/judge.py only scores. judge/gate.py is the piece that decides what happens next — PRD section 7's actual runtime gate, not just a report. Not every rubric dimension is equally dangerous, so instead of a single priority ranking, the five dimensions are grouped into three tiers of consequence:

• Block (Grounded, Honest on terminal state, and an inconsistent self-consistency check — see below) — the answer asserts something not actually supported by the document, whether a confidently invented fact, a confidently invented not-found response, or a claim that doesn't reproduce across independent samples of the same context. gate_answer regenerates and re-judges, up to 3 attempts total. If it's still failing after that, the user never sees the fabricated answer — they get a fixed "please contact support" message (SUPPORT_FALLBACK_MESSAGE) instead.
• Caveat (Complete) — the core claim is real, just missing a caveat that could flip its practical meaning. Not dangerous enough to withhold: the real answer is still delivered, with a short note appended (CAVEAT_NOTE) telling the user to double-check for exclusions.
• Log (Correctly tiered, Correctly routed) — about whether the process picked the right path, not whether this answer is unsafe to show. Recorded in the verdicts (still visible in the judge panel/output) but never blocks or modifies delivery.

Entirely transparent to the user by design. Every retry re-runs the full pipeline — a fresh LLM call, on the chance it doesn't repeat the same fabrication — and re-scores the new answer, all inside gate_answer(). Callers (answer/cli.py, server/app.py) only ever see the final, already-gated result: one rendered answer, one set of verdicts. There's no "retrying..." message, no partial output, no visible attempt count. Each attempt's verdicts are kept on GatedResult.attempts for whoever's debugging or building on this next, but nothing about the loop reaches the user.

Fails open on infrastructure errors, not verdict failures. If the scoring call itself errors (model unavailable, network blip — not a verdict), that's not evidence of a hallucination, so gate_answer doesn't retry or block on it. It delivers the generated answer as-is and sets judge_error, same as the old ungated behavior — matching the PRD's own framing of the judge as "a compliance backstop, not the core trust mechanism."

Cost is tiered by risk, not flat — this changed after real testing showed it needed to. The first version of this gate ran the same 5-call bundle (1 generate + 2 consistency samples + 1 comparison + 1 judge) on every answer regardless of tier, and it was slow enough in practice (worst case 15 calls across 3 retries) that a browser request could sit on "Thinking…" for the better part of a minute with no progress indicator — indistinguishable from actually being stuck. Two changes fixed this without weakening the gate where it matters:

• Low-tier answers skip self-consistency sampling entirely. A Low-tier answer is a single, direct passage lookup — every real test of it came back consistent anyway, so paying for 2 extra samples + a comparison call was pure overhead. judge/gate.py's _score_attempt now branches on tier: Low calls the plain judge_pipeline_result (1 generate + 1 judge = 2 calls/attempt, the same cost as before self-consistency existed at all). Medium/High still get the full treatment, since that's where the extra scrutiny is actually worth it.
• Judge and consistency are merged into ONE call for Medium/High, not two separate ones. judge/judge.py's build_schema/build_system_prompt now take an include_consistency flag that splices a sixth self_consistency field onto the same rubric schema/prompt, and judge_answer_with_consistency / judge_pipeline_result_with_consistency score both from a single generate_structured call. Medium/High cost drops to 1 generate + 2 samples + 1 merged call = 4 calls/attempt (down from 5) — the self-consistency check now costs exactly 1 extra call over the plain judge, not 3.

The retry budget is configurable, not a hardcoded 3. gate_answer(..., max_attempts=MAX_ATTEMPTS) — CLI: --max-attempts N; API: max_attempts on the request, both defaulting to 3 (judge/gate.py's MAX_ATTEMPTS). This mostly matters for Medium/High, since that's the tier where the retry loop's cost is highest — tune it down for cheaper, faster iteration, or up if you want the gate to try harder before falling back to the support message.

A dead field this refactor caught and removed: merging judge and consistency into one call means a failure there can't be attributed to "the judge failed" vs. "consistency failed" separately anymore — it's one call, one failure, always reported as judge_error. The old, separate consistency_error field (on GatedResult and the API's ChatResponse) could never actually be set anymore once the merge landed, so it was removed from judge/gate.py, server/app.py, and web/app.js rather than left as dead code nobody would ever see populated.

Wiring: --judge (CLI, default on) and judge (API request field, default on) go through gate_answer instead of a bare judge_pipeline_result call — turning judging off skips gating entirely too, for the cheapest possible iteration loop. judge/cli.py and eval/runner.py deliberately still call judge_pipeline_result directly, ungated and unmerged — the standalone judge CLI is for inspecting one raw verdict, and eval is measuring the system's natural, un-retried failure rate for calibration, not simulating what a live user would actually see.

Known gap: the retry-attempt history on GatedResult.attempts is kept for whoever wants to dig deeper into hallucination patterns later, but there's still no independent, non-LLM hallucination detector (e.g. an entailment model cross-checking claims token-by-token against retrieved text) — both signals blocking delivery today (the rubric judge and the self-consistency check below) are themselves LLM calls, and for Medium/High now literally the same call.

Tests: tests/test_gate.py (dimension partitioning by consequence tier; Low-tier tests against a real built index, asserting the cheaper 2-call cost; Medium/High tests with run_pipeline stubbed out to exercise the merged-call path — retry-then-succeed, retry-exhaustion-to-fallback, caveat-only delivery, log-only dimensions never gating, fail-open on a scoring-call error, inconsistent-samples triggering a retry even when the rubric passes, and a custom, lower max_attempts cutting the loop short) plus tests/test_server.py cases exercising the same paths through the real /api/chat endpoint, including one that forces Medium/High tier (self-consistency isn't computed for Low, so this test can't rely on whatever tier the sample doc happens to classify a plain question as).

Self-consistency check (judge/consistency.py)

A second, independent hallucination signal for Medium/High-tier answers, folded into the same gate attempt (and, since the cost pass above, the same LLM call as the rubric judge) rather than stacked as a separate retry loop. The idea (closest published version: "SelfCheckGPT"): sample the generation step a few times over the exact same, deterministic retrieval and risk tier, then assess whether the samples agree on every material fact. A claim genuinely grounded in the retrieved excerpt should come out almost identical every time; a fabricated detail tends to drift, since it's not anchored to anything in the context — the model is improvising, not reading it off the page.

Why this is worth having alongside the rubric judge, not redundant with it: the judge and generator share the same model/provider by default (a bias risk already called out above — a judge scoring its own generator's output). Self-consistency doesn't depend on the judge believing anything; it's a check on the generator's own stability across independent samples. A confident fabrication that happens to fool the judge would still likely come out differently on a second or third sampling — so this catches a failure mode the rubric judge structurally can't.

sample_variants(question, background, pipeline_result, provider, n=3) is the reusable piece: the already-generated answer counts as sample 1, so only n - 1 new generations happen. judge/gate.py calls this directly and hands the samples to the merged judge+consistency call (see above); check_consistency (still here, still independently tested) wraps the same sampling with its own standalone comparison call, for anyone who wants a consistency verdict without the rubric attached.

Validated live, not just against fakes: on the old, unmerged design, a real run correctly reported "Consistent across samples" on a clean question, and a separate run correctly caught a genuine Correctly routed FAIL from the rubric judge while leaving the answer undisturbed (that dimension is LOG-tier, not BLOCK-tier) — and, hitting Gemini's free-tier rate limit mid-test, the consistency check degraded gracefully instead of crashing the request. After the merge, a live Low-tier run confirmed the new "not computed" path (judge/cli.py now prints "Not computed for Low-tier answers" instead of a verdict) and measurably dropped wall-clock time for that request; a live Medium/High run to confirm the merged call end-to-end is still pending — the free-tier daily quota ran out testing the rest of this — but the merged path is thoroughly covered by tests/test_gate.py and tests/test_judge.py.

Tests: tests/test_consistency.py (the given answer is reused as sample 1 rather than wasting a call regenerating it, sample count respects n, the comparison call receives the question and every sample, and consistent/inconsistent payloads round-trip correctly) plus tests/test_judge.py for the merged path (build_schema/ build_system_prompt with include_consistency=True, and judge_answer_with_consistency/judge_pipeline_result_with_consistency making exactly one call and returning both results) — all via fake providers, no real model calls.

Eval (eval/)

Batch-runs the PRD's own Appendix: Evaluation Test Set Design — all 31 questions across its four buckets (9 Low - Happy Path, 4 Low - Not Found, 6 Medium, 12 High) — through the real pipeline, and reports tier accuracy and rubric pass rates per bucket. This is the manual before/after comparison this project did by hand throughout its build (retrieval calibration, tier calibration, the judge's own validation against a deliberately bad answer) — now a single repeatable command instead of a one-off script each time.

• eval/testset.py — TEST_SET: the PRD's 31 questions transcribed verbatim as EvalQuestion(question, bucket, expected_tier, reference). reference is the PRD's stated expected answer, kept for human reference only — the judge never sees it and doesn't score against it; it scores the answer against the rubric, not against a canned "correct" answer. Add coverage by adding a row here.
• eval/runner.py — run_eval(...): for each question, calls answer.pipeline.run_pipeline then judge.judge.judge_answer, exactly the same functions answer/cli.py and judge/cli.py use. One question's failure (rate limit, network blip) doesn't lose the whole batch — it's caught, recorded on the EvalRecord as error, and the run continues. EvalRecord.tier_exact_match / .tier_at_least_as_strict implement the PRD's own "correctly tiered" standard (RiskTier is an IntEnum ordered LOW < MEDIUM < HIGH, so "at least as strict" is a plain >= comparison) — None on both when the question errored, never a fabricated result.
• eval/report.py — progress_line (one compact line per question as the run streams by, with a =/>/< tier-match marker), summary_table (grouped by PRD bucket, plus an OVERALL row: tier-match rates and a pass-count per rubric dimension — errored questions are counted and excluded, never silently dropped), and to_json (full detail: every rendered answer and every judge verdict's reasoning, for the report file).
• eval/cli.py — wires it together with --bucket/--limit filtering (so a quick, cheap run doesn't require burning ~62 calls on the full set), --out for the JSON report path, and the same --provider/--model/--debug flags as answer/judge.

Validated with a real (not just faked) end-to-end run — ./run.sh eval --provider gemini --bucket "Low - Happy Path" --limit 1 correctly classified LOW, matched the PRD's expected tier exactly, and passed all 5 rubric dimensions, with the full reasoning landing in the JSON report exactly as expected.

A real bug this build caught, worth knowing about: the first version of to_json wrote r.actual_tier.name if r.actual_tier else None — which silently turned every genuinely-correct LOW result into null, because RiskTier.LOW is an IntEnum with value 0, and 0 is falsy in Python. A round-trip test (assert entry["actual_tier"] == "LOW") caught it immediately. Fixed to if r.actual_tier is not None. Worth remembering if you extend this code: never truthiness-check a RiskTier value, always compare to None explicitly.

Tests: tests/test_eval_testset.py (31 questions, correct bucket distribution, expected tiers match bucket naming), tests/test_eval_runner.py (one record per question, error isolation via a fake provider that raises for a specific question, tier_exact_match/tier_at_least_as_strict correctness), and tests/test_eval_report.py (progress-line markers, summary table structure, JSON round-tripping — including a regression test for the RiskTier.LOW truthiness bug above).

Included sample document

data/sample_policy.md is a fabricated policy used only to smoke-test the pipeline — it's built to line up with the facts referenced in the PRD's own evaluation test set (Appendix), e.g. cancer benefit $75,000 in Section 5A.2, heart attack $50,000 in Section 5A.1, 48-hour pre-authorization in Appendix C, cash payments not accepted in Appendix B, etc. Replace it with your real policy document — drop it in data/ and rebuild the index.

A quick way to sanity-check retrieval quality: ask a question whose answer is in the document (e.g. "What's the maximum payout for a heart attack?") and one whose answer isn't (e.g. "Is this a term insurance?"). In this sample doc the not-in-document question actually still scores ~0.11 — its highest-scoring chunk turns out to be the doc's own disclaimer preamble, which happens to contain the word "insurance." That's a corpus artifact of the smoke-test doc, not a fluke you should expect to disappear on a real policy — see the calibration notes below for why "low score = not found" isn't as clean a signal as it sounds.

Calibration notes (read before trusting the risk classifier)

The classifier was validated against the PRD's own labeled eval set (Appendix: Evaluation Test Set Design) run against the bundled sample policy — not just built and assumed to work. Two honest findings came out of that:

1. The concrete anchors are reliable. Dollar exposure, retrieval conflict, multi-section reasoning, and missing-fact detection are all structural (background fields, relative retrieval scores, regex matches) rather than a single fuzzy number, so they behave predictably. None of the PRD's "Low Risk – Happy Path" questions came back looser than Low — some came back Medium or High instead, which the PRD's own scoring rubric calls acceptable ("erred toward the stricter side").
2. Retrieval score is a weak proxy for "directly stated vs. needs interpretation." That's the test step 3/4 in resolve_tier uses to decide Low vs. Medium. Measured top-1 scores for the PRD's Low-happy and Medium-interpretation buckets on the sample doc: medians ~0.22 vs. ~0.23 — they overlap almost completely. Retrieval score tells you a passage is topically relevant, not whether it states the answer outright or requires inference — that's a reasoning judgment, not a lexical-overlap one. No threshold tuning fixes this; it's a ceiling on what TF-IDF alone can tell you. This is exactly why the PRD puts a separate grounding/completeness checker after answer generation (section 6) rather than relying on the classifier for that call — the judge/ package's Grounded/Complete dimensions are that checker, run as a separate step, not folded back into the classifier. Treat this classifier's Low-vs-Medium output as provisional, and don't wire anything downstream that assumes it's precise.

If you re-run the eval against your real policy document and see the same pattern, that's expected, not a regression — the fix is an LLM-based groundedness check downstream, not a better retrieval threshold.

Known limitations (expected for this v1 slice)

• TF-IDF is lexical, not semantic — it won't match paraphrases well (e.g. "heart attack" vs. "myocardial infarction" only works here because the sample doc names both). Fine for v1 per the PRD; an embedding-based upgrade is a natural v2 step if recall on paraphrased questions is poor.
• PDF heading detection is a heuristic, not real structure extraction — section citations for PDFs default to page numbers.
• The missing-fact registry (risk/fact_requirements.py) is a small, hand-written keyword list — expected to miss real gaps until grown from eval failures, not a general-purpose fact-tracking system.
• The runtime judge now gates, not just screens — judge/gate.py's gate_answer runs automatically in answer/cli.py (--judge, default on) and server/app.py (judge request field, default on), at a cost tiered by risk: Low-tier answers get the plain rubric judge only (2 calls/attempt); Medium/High get the rubric judge merged with a 3-sample self-consistency check into one call (4 calls/attempt). A Grounded/Honest on terminal state failure, or (Medium/High only) the samples disagreeing on a material fact, triggers a silent regeneration (up to --max-attempts/max_attempts, default 3) before falling back to a "contact support" message; a Complete-only failure still delivers the answer with a caveat note appended. See "Judge gate" and "Self-consistency check" above for the full policy and the rationale for the tiered cost model. What's still missing: both signals blocking delivery today are themselves LLM calls (the rubric judge and, for Medium/High, the same merged call for consistency) — there's no independent, non-LLM check (e.g. an entailment model cross-checking claims against retrieved text) backing them up, and the judge/generator still share the same model/provider by default, which is a real bias risk for a judge grading its own work (self-consistency mitigates this specific risk somewhat, since it doesn't depend on the judge agreeing with the generator, but it's still the same underlying model doing the sampling).
• Clarifying questions and consent are still single-shot pipeline concepts under the hood (see "Design decisions" above) — the chat UI's merge trick and consent buttons make it feel conversational, but there's no real server-side session state. See "Chat UI" above for the full list of what that UI does and doesn't do yet.
• GeminiProvider's default model ID (answer/providers/gemini.py) is a best-effort guess, not verified against a live, current Gemini model catalog — unlike the Claude provider's model ID, which is authoritative. Override it with --model or GEMINI_MODEL if it 404s.