Add pinned tau2 retail smoke and trace comparison

.gitignore

@@ -10,3 +10,4 @@ build/
 dist/
 .demo/
 .aes/runs/
+evals/tau2_retail/artifacts/**/task-*.simulation.json

evals/tau2_retail/FINDINGS.md

@@ -0,0 +1,118 @@
+# tau2-bench retail: faultgate vs tau2
+
+Two runs at tau2-bench `7aad81c37a3a78b4ed3e4866769f851400c64415`, model
+`openai/qwen3.8-flash` for the agent, user simulator, and NL-assertion judge.
+The exact commands are in `README.md`.
+
+## Smoke test: 2 tasks
+
+Tasks `0` and `16`, chosen before the run: task `0` exercises a two-item
+exchange and task `16` exercises two cancellations, one return, and a computed
+refund total.
+
+| task | tau2 reward | termination | tool steps | faultgate | agreement |
+|---|---|---|---|---|---|
+| 0 | 1.0 | user_stop | 8 | pass | yes (both clean) |
+| 16 | 1.0 | user_stop | 8 | pass | yes (both clean) |
+
+`faultgate precheck`: 2 of 2 traces, 16 tool steps read, 0 errors, 0 warnings.
+`faultgate check --policy examples/policy_retail.json`: 0 of 2 traces failed,
+every check `ok 1.00`, exit 0.
+
+Manual review of the two traces:
+
+- **Task 0.** The agent found `yusuf_rossi_9620`, read the order and both
+  products, then made one `exchange_delivered_order_items` call covering both
+  items. The order was `delivered`, the call carried the matching `order_id`,
+  and the exchange used the original payment method. tau2's DB check and the
+  empty NL-assertion set both scored 1.0. There is no faultgate finding to
+  disagree with.
+- **Task 16.** The agent read all three orders, calculated
+  `3131.10 + 4777.75 + 367.38 = 8276.23`, cancelled the two pending orders with
+  the allowed reason `no longer needed`, and returned only the watch from the
+  delivered order. Each write was preceded by the matching `get_order_details`
+  call with the same `order_id`; no order was written twice. tau2's DB check
+  and the $8,276.23 NL assertion both scored 1.0.
+
+No disagreements in the smoke test.
+
+### Trace-capture pitfall found during the smoke test
+
+The first version of the runner instrumented `Environment.get_response` for
+the whole `run_simulation` call. tau2 evaluates a run by replaying its
+trajectory on a fresh environment, so the trace contained a second copy of
+every write. faultgate correctly flagged those duplicate spans as
+`max_calls_per_arg` violations and as writes that invalidated their own
+`get_order_details` prerequisite. The fix records only the live environment
+(`run.py`, `state["env"]`), and the policy findings disappeared without
+changing the checker. The stale two-task artifact was discarded.
+
+This is a reminder for any tau2 integration: capture the simulation, not the
+evaluation replay.
+
+## Expanded run
+
+The first ten tasks of tau2's published `retail_50` stratified subset:
+`6,8,11,13,14,16,17,18,19,21`. The machine-readable join is
+`artifacts/retail_50_first10/comparison.json`; the CLI table is
+`comparison.txt`.
+
+| task | tau2 reward | faultgate | finding |
+|---|---|---|---|
+| 6 | 1.0 | pass | - |
+| 8 | 1.0 | pass | - |
+| 11 | 1.0 | pass | - |
+| 13 | 1.0 | pass | - |
+| 14 | 1.0 | pass | - |
+| 16 | 1.0 | **FAIL** | `unrecovered_tool_error` (reviewed below) |
+| 17 | 1.0 | pass | - |
+| 18 | **0.0** | pass | tau2 DB check failed (reviewed below) |
+| 19 | 1.0 | pass | - |
+| 21 | 1.0 | pass | - |
+
+tau2 passed 9 of 10 (mean reward 0.9); faultgate passed 9 of 10; the two
+systems agreed on 8 of 10. There is one disagreement in each direction.
+`report-with-outcomes.md` is the faultgate-generated user view with both
+signals in one table; `comparison.txt` is the external join used for review.
+
+### Disagreement A: task 16, tau2 pass / faultgate FAIL
+
+Trace `974062dc7ad2621aad5ccf62fb2fdbb5`.
+
+Step 1 called `find_user_id_by_email` with
+`sofia.doe_496@example.com` before the user had given an email. The tool
+returned `Error: User not found`. The agent then asked for identity details,
+received Fatima Johnson's name and ZIP, and succeeded with
+`find_user_id_by_name_zip`. It went on to read all three orders, cancel the two
+pending ones, and return only the watch. tau2 scored the final DB state and the
+$8,276.23 assertion at 1.0.
+
+Faultgate's `unrecovered_tool_error` fires because a tool errored and no later
+call **to the same tool** succeeded. That is not what happened here: the agent
+recovered through the other identity tool. In the retail domain the two
+lookups are alternative ways to establish the same principal, so the verdict
+is a false positive for "unrecovered".
+
+The flag is still useful: the fabricated email is a genuine bad step that
+tau2's task reward ignored. But the reason is wrong. A future rule could treat
+a configured group of equivalent tools as one recovery unit, or check whether
+the intent (identify the user) succeeded rather than the same function name.
+
+### Disagreement B: task 18, tau2 fail / faultgate pass
+
+Trace `e058f9be20d72c97aa15a745f024edee`.
+
+The agent found the office chair, quoted the return, and the user changed their
+mind to an exchange. The agent described the exchange and asked for
+confirmation. The user replied "Yeah, go ahead. I'll take the same one." with
+`###STOP###` appended; tau2 ended the simulation on that user stop before the
+agent got another turn. No `exchange_delivered_order_items` call was made, so
+the database did not change and tau2 returned 0.0. The NL assertion still
+scored 1.0.
+
+Faultgate passes because there is no loop, abstention, policy violation,
+repeated call, tool error, or overflow. The failure is a turn-taking artifact
+of the half-duplex user simulator, not a process fault in faultgate's list. A
+single sample cannot tell us whether the agent would have acted on a real
+follow-up turn. This is the expected scope limit: a clean faultgate verdict is
+not a task-success claim.

evals/tau2_retail/README.md

@@ -0,0 +1,165 @@
+# tau2-bench retail smoke test
+
+Run a small slice of the public [tau2-bench](https://github.com/sierra-research/tau2-bench)
+retail benchmark, export the actual agent/tool spans as OTLP/JSON, and gate them
+with faultgate's retail policy. This is an integration smoke test, not a
+benchmark score.
+
+## Pinned versions
+
+| Component | Version |
+|---|---|
+| tau2-bench | [`7aad81c37a3a78b4ed3e4866769f851400c64415`](https://github.com/sierra-research/tau2-bench/commit/7aad81c37a3a78b4ed3e4866769f851400c64415) (2026-10-28) |
+| faultgate | 0.4.1 (this checkout) |
+| Python | 3.12.15 |
+| uv | 0.12.24 |
+| opentelemetry-sdk | 1.45.2 |
+
+The runner records `tau2_commit` in `results.json` and refuses to run against a
+different checkout unless `--allow-other-commit` is passed; a run against a
+different commit is not comparable.
+
+## Setup
+
+```bash
+git clone https://github.com/sierra-research/tau2-bench.git
+cd tau2-bench
+git checkout 7aad81c37a3a78b4ed3e4866769f851400c64415
+uv sync
+# The runner instruments tau2 directly; tau2 does not depend on the SDK.
+uv pip install --python .venv/bin/python "opentelemetry-sdk>=1.27"
+```
+
+On Windows the interpreter is `.venv\Scripts\python.exe`.
+
+Install faultgate in its own environment from the faultgate checkout:
+
+```bash
+python -m venv .venv
+. .venv/bin/activate        # Windows: .venv\Scripts\activate
+pip install -e .
+```
+
+### Credentials
+
+The runner reads the standard environment variables, then falls back to the
+tau2 checkout's `.env` for any missing `LLM_*` / `OPENAI_*` value:
+
+```bash
+export LLM_API_BASE=https://your-openai-compatible-endpoint/v1
+export LLM_API_KEY=...
+export LLM_MODEL=openai/qwen3.8-flash
+# Only for gateways that route by session header:
+export LLM_EXTRA_HEADERS='{"x-opencode-session":"..."}'
+```
+
+The run in `artifacts/` used an OpenAI-compatible OpenCode inference gateway
+and `qwen3.8-flash` for the agent, user simulator, and NL-assertion judge. The
+gateway needs `x-opencode-session`; a normal OpenAI or LiteLLM endpoint does
+not. No key or endpoint credential is written to the artifacts.
+
+## Run
+
+From the faultgate checkout, with the pinned tau2 checkout beside it:
+
+```bash
+python evals/tau2_retail/run.py \
+  --tau2-dir ../tau2-bench \
+  --out-dir evals/tau2_retail/artifacts/smoke \
+  --task-ids 0,16 \
+  --model openai/qwen3.8-flash
+```
+
+The smoke pair is retail task `0` (two-item exchange) and `16` (cancel two
+orders, return a watch, report the total). The expanded slice is the first ten
+tasks of tau2's published `retail_50` stratified subset:
+
+```bash
+python evals/tau2_retail/run.py \
+  --tau2-dir ../tau2-bench \
+  --out-dir evals/tau2_retail/artifacts/retail_50_first10 \
+  --task-ids 6,8,11,13,14,16,17,18,19,21 \
+  --model openai/qwen3.8-flash
+```
+
+Outputs:
+
+| File | Contents |
+|---|---|
+| `traces.json` | OTLP/JSON, one `invoke_agent` root per task and one `execute_tool` child per live tool call |
+| `results.json` | tau2 commit, model, per-task reward/termination/answer and `reward_info` |
+| `cost.json` | call count, prompt/completion tokens, models |
+| `faultgate-report.json` | faultgate per-check verdicts and reasons |
+| `precheck.txt`, `check.txt` | exact CLI output and exit codes |
+
+The exporter deliberately does **not** record tau2's evaluation replay. tau2
+replays each trajectory on a fresh environment to check the final database
+state; recording that replay would duplicate every write and make faultgate's
+`max_calls_per_arg` and `invalidated_by` rules fire on calls the agent never
+made.
+
+## Gate
+
+```bash
+faultgate precheck evals/tau2_retail/artifacts/smoke/traces.json \
+  --policy examples/policy_retail.json | tee evals/tau2_retail/artifacts/smoke/precheck.txt
+faultgate check evals/tau2_retail/artifacts/smoke/traces.json \
+  --policy examples/policy_retail.json \
+  --json evals/tau2_retail/artifacts/smoke/faultgate-report.json \
+  | tee evals/tau2_retail/artifacts/smoke/check.txt
+```
+
+`check` exits 1 when a check fires and 2 on bad input. Capture `$?` if you
+need the exit code in CI.
+
+### Put τ-bench's outcome in the same report
+
+Faultgate can take your evaluator's per-scenario pass/fail with `--outcomes`
+and show it beside its own checks:
+
+```bash
+faultgate check evals/tau2_retail/artifacts/retail_50_first10/traces.json \
+  --policy examples/policy_retail.json \
+  --outcomes evals/tau2_retail/artifacts/retail_50_first10/outcomes.json \
+  --markdown evals/tau2_retail/artifacts/retail_50_first10/report-with-outcomes.md
+```
+
+`artifacts/retail_50_first10/outcomes.json` is derived from the run's
+`results.json` (`retail-<task id>` → `reward == 1.0`). The generated Markdown
+report is the PR/user-facing view: 9/10 outcomes true, 2/10 traces failed
+(one process check and one task outcome), with the failing trace IDs and a
+reproduce command.
+
+## Cost
+
+`cost.json` records every model call and the prompt/completion token totals for
+the run. The gateway used here did not return a per-call USD price, so the
+files contain no dollar estimate; multiply the token totals by the rates of the
+provider you use. The two-task smoke made 35 calls and used 189,276 prompt /
+8,251 completion tokens over 198 seconds. The ten-task expanded run made 157
+calls and used 768,057 prompt / 34,102 completion tokens over 738 seconds.
+
+No model-based faultgate judge was enabled. tau2's NL-assertion judge is a
+model call and is included in the token totals.
+
+## How to read the comparison
+
+tau2 and faultgate answer different questions:
+
+- **tau2 reward** is task success: the final database state and, where present,
+  the natural-language assertion.
+- **faultgate** is a process gate: loops, abstention, policy violations,
+  repeated calls, unrecovered tool errors, context overflow. It does not score
+  correctness unless `--golden` is supplied.
+
+So the useful cells are the disagreements:
+
+| tau2 | faultgate | reading |
+|---|---|---|
+| pass | pass | clean run; no known process fault |
+| fail | fail | both saw a problem, possibly different ones |
+| **pass** | **fail** | faultgate false positive, or a real process fault that did not change the task outcome |
+| **fail** | **pass** | task failure outside faultgate's checks (e.g. stopped before acting, wrong business outcome) |
+
+The manual review in `artifacts/*/FINDINGS.md` reads the flagged traces against
+tau2's messages before calling either side wrong.

evals/tau2_retail/artifacts/retail_50_first10/check.txt

@@ -0,0 +1,18 @@
+note: wrong_direction skipped - it needs a judge (--judge laya | litellm:<model>); see README for each judge's measured \u03ba
+note: wrong_answer skipped - it needs --golden golden.json (expected answers)
+note: truncation skipped - it needs --golden golden.json (expected answers)
+trace         search_loop       abstention        policy            repeated_call     unrecovered_tool_error  context_overflow
+------------------------------------------------------------------------------------------------------------------------------
+7592bae12a21  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+75d4fb0fe0a8  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+8a8111108d85  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+43211beff721  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+67efaceb17c0  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+974062dc7ad2  ok   1.00         ok   1.00         ok   1.00         ok   1.00         FAIL 1.00         ok   1.00       
+f8cf6fd29a62  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+e058f9be20d7  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+4cc6470c1816  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+e3f1c31efa7f  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00       
+1/10 traces failed - search_loop=0, abstention=0, policy=0, repeated_call=0, unrecovered_tool_error=1, context_overflow=0 - judge none - 0.0s
+report written to faultgate\evals\tau2_retail\artifacts\retail_50_first10\faultgate-report.json
+EXIT=1

evals/tau2_retail/artifacts/retail_50_first10/comparison.json

@@ -0,0 +1,77 @@
+[
+  {
+    "task_id": "6",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "8",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "11",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "13",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "14",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "16",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": false,
+    "fired": [
+      {
+        "check": "unrecovered_tool_error",
+        "reason": "Tool 'find_user_id_by_email' failed at step 1 (Response starts with 'Error:') and no subsequent call to 'find_user_id_by_email' succeeded."
+      }
+    ]
+  },
+  {
+    "task_id": "17",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "18",
+    "tau2_reward": 0.0,
+    "tau2_pass": false,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "19",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  },
+  {
+    "task_id": "21",
+    "tau2_reward": 1.0,
+    "tau2_pass": true,
+    "faultgate_pass": true,
+    "fired": []
+  }
+]