[Faultline-lab]

Summary

Add a reproducible τ-bench retail integration under evals/tau2_retail/:

• pinned to tau2-bench 7aad81c37a3a78b4ed3e4866769f851400c64415 (2026-10-28);
• a runner that instruments the live tau2 environment and writes OTLP/JSON GenAI traces (one invoke_agent root, one execute_tool child per call);
• a two-task smoke test (retail 0, 16) and a ten-task slice of tau2's published retail_50 subset;
• faultgate precheck / check output with examples/policy_retail.json;
• a comparison script and a manual review of the disagreements.

Results

Smoke: both tasks scored tau2 reward 1.0, faultgate 0/2 failed. No disagreement.

Expanded slice 6,8,11,13,14,16,17,18,19,21:

task	tau2	faultgate	notes
6	1.0	pass
8	1.0	pass
11	1.0	pass
13	1.0	pass
14	1.0	pass
16	1.0	FAIL	unrecovered_tool_error false positive; recovered via the alternate identity tool
17	1.0	pass
18	0.0	pass	user simulator appended ###STOP### before the agent could act
19	1.0	pass
21	1.0	pass

Two disagreements were reviewed by hand and are written up in
evals/tau2_retail/FINDINGS.md. tau2 passed 9/10 (mean reward 0.9); faultgate passed 9/10; agreement 8/10,
with one disagreement in each direction. The task 16 flag is useful but
mislabeled; the task 18 failure is outside faultgate's process checks.

faultgate precheck --sample 0 read 10/10 traces and 64 tool steps with 0
errors / 0 warnings. faultgate check read the same 10 traces, fired once
(task 16), and exited 1.

User-facing report

check --outcomes ... --markdown ... produces
artifacts/retail_50_first10/report-with-outcomes.md: tau2's 9/10 task
outcomes and faultgate's process checks in one table, with failing trace IDs
and a reproduce command. This is the hand-off artifact; the longer
Aster-&-Oak-style PDF is a narrative layer built from the same JSON/evidence.

Trace capture

The runner patches Environment.get_response only for the live environment.
tau2's evaluator replays the trajectory on a fresh environment; recording that
replay duplicates every write and makes max_calls_per_arg / invalidated_by
fire on calls the agent never made. This was the only integration bug found.

Setup and cost

Setup, credentials, commands, and the pinned-commit guard are in
evals/tau2_retail/README.md. cost.json records calls and prompt/completion
tokens. The smoke used 35 calls / 189,276 prompt / 8,251 completion tokens
over 198 s. The ten-task expanded run used 157 calls / 768,057 prompt / 34,102
completion tokens over 738 s. The gateway returned no USD price, so no dollar
estimate is fabricated.

Verification

• python -m py_compile evals/tau2_retail/run.py evals/tau2_retail/compare.py
• faultgate precheck ... --policy examples/policy_retail.json (exit 0)
• faultgate check ... --policy examples/policy_retail.json (exit 1 on task 16)
• python evals/tau2_retail/compare.py ... (agreement table)
• Traces in artifacts/ are the actual OTLP exports from the runs.

Relationship to #20

#20 is an open two-task recipe at an older tau2 commit (ebf990b). This PR
pins current main (7aad81c), records live spans rather than converting the
CLI's results file, adds the ten-task slice, and reports the two disagreements
#20's sample does not cover. I did not modify #20.