[patelankeet2]

Closes #393 (US-60).

What it does

A held-out question set for ai/query.py, scored every time it runs, mirroring docs/evaluation.md Part 2's own four-outcome shape from the one hand measurement that has existed since 10 August (answered-correct, refused-correctly, refused-but-should-answer, and the dangerous one, answered-wrong).

The one design difference from #385's held-out set, and why

#385 (tools/explanation_quality.py) had to freeze a human rating of model-written prose, because the model's text varies run to run and judging whether it reads well needs a person. ai/query.py calls no model in either direction (CLAUDE.md 7c) -- it is pure, deterministic classification and resolution over a snapshot. There is nothing here that can go stale the way #385's captured text can, so:

• No frozen output, no staleness-hash check.
• The held-out set freezes the input (a question plus a FakeSession configuration, the exact harness tests/test_query_translation.py already uses) and the expected outcome.
• The tool actually executes answer_question() against every entry, every single run -- in a unit test, in CI, by hand, two years from now. Stronger evidence than replaying a snapshot of a past run, not weaker.

The held-out set

20 entries (tests/fixtures/query_quality/holdout.json), 10 answered-correct / 10 refused-correctly, built from scenarios already individually proven in tests/test_query_translation.py rather than invented fresh -- plus Q05, named explicitly as a regression pin for the #70 bug (a network destination resolved to a real host, not the network address, which the historical measurement found and fixed the same day). The exact 20 questions measured on 10 August were never preserved verbatim in docs/evaluation.md, so this is a rebuilt, equivalent set rather than a literal replay -- said plainly in the doc update.

A real gap I found in my own first version

While writing the classification logic, caught a case where an entry with no answer_contains/answer_starts_with to check against would silently fall through to answered_correct if the code answered when it should have refused -- nothing to text-match against means the check just... passed. Fixed before it ever shipped: should_ground is now compared explicitly against the entry's own outcome_expected, so "answered when a refusal was expected" is always answered_wrong, regardless of what the answer text happens to say. Pinned with its own test (test_answering_when_a_refusal_was_expected_is_answered_wrong_regardless_of_text) and mutation-tested by reverting to the original logic -- failed for exactly the right reason, reverted.

Verified

python -m tools.query_quality          20 entries, 10/10/0/0, 0 mismatches
python tools/query_quality.py          same -- both documented invocation forms

python -m pytest tests/test_query_quality.py tests/test_query_translation.py tests/test_tools_invocation.py -v
  60 passed, 10 skipped

python -m pytest tests/ -q
  1901 passed, 13 skipped   (main: 1885 passed -- +16 accounted for by the new
                             test file plus tools/test_tools_invocation.py's
                             auto-discovery of the new tool)

Mutation-tested twice: flipped the committed fixture's own expectation on one entry (caught by both the threshold test and the mismatch test, reverted), and reverted the classification-logic fix described above (caught by its own dedicated test, reverted).

Acceptance criteria

• A held-out question set for ai/query.py, scored with the same method as A held-out, repeatable score for the explanation layer (#346) #385 (the pattern -- A held-out, repeatable score for the explanation layer (#346) #385 itself isn't merged yet; see note below)
• The score runs in CI, which has neither Batfish nor Ollama -- every FakeSession here needs neither
• "Answered and wrong" is counted separately from "refused", exactly as docs/evaluation.md Part 2 does

One thing worth knowing before merge

#393 says "depends on #385: its method is reused, so it merges first." #385 isn't merged -- its last CHANGES_REQUESTED review is stale under M-2 (flagged separately, on #385 itself). This PR only reuses #385's pattern (the held-out-set-plus-documented-threshold shape), not its code or data, so it does not need #385 to land first -- #394 (US-61) is the one that actually needs #385's real numbers. Flagging this so nobody assumes the dependency note above means this PR is blocked when it isn't.