diff --git a/evals/dynamic/README.md b/evals/dynamic/README.md index 0edf36c..672ec86 100644 --- a/evals/dynamic/README.md +++ b/evals/dynamic/README.md @@ -98,7 +98,8 @@ the spec-analyzer agent open the plugin's spec model. `session-docs` has Claude session context. The control, `without-rule`, runs without the rule, where the same read is denied. Agents that ask for `opus` run on Haiku (`ANTHROPIC_DEFAULT_OPUS_MODEL`). Three sessions cost about $0.60; run it after any change to how the plugin's skills, agents, or hooks name the reference docs. -Report: [`reports/2026-10-05-plugin-docs.md`](reports/2026-10-05-plugin-docs.md). +Report: [`reports/2026-10-05-plugin-docs.md`](reports/2026-10-05-plugin-docs.md). Both suites on +Haiku 5.5: [`reports/2026-10-09-haiku-5-5.md`](reports/2026-10-09-haiku-5-5.md). The **adopt** suite builds its own project (`fixtures/adopt/project.sh`: a new repository with no commits) and appends each run's end state (`inspect.sh`) to the transcript. `{{FRAMEWORK}}` in a diff --git a/evals/dynamic/reports/2026-10-09-haiku-5-5.md b/evals/dynamic/reports/2026-10-09-haiku-5-5.md new file mode 100644 index 0000000..1934f84 --- /dev/null +++ b/evals/dynamic/reports/2026-10-09-haiku-5-5.md @@ -0,0 +1,57 @@ +# Session evals — the plugin suites on Haiku 5.5 — 2026-10-09 + +The `plugin-hooks` and `plugin-docs` suites first ran on Haiku 4.5 (the 2026-10-04 and 2026-10-05 +reports). This run repeats them on Haiku 5.5, passed as `--models claude-haiku-5-5`, with real +headless Claude Code sessions (v2.1.286) and the `adf` plugin loaded from a checkout of +`fix/prompt-audit`. Agents that ask for `opus` now run on `claude-haiku-5-5` too: the runner's +stand-in was still Haiku 4.5. Every transcript names `claude-haiku-5-5`, the agents' included. + +| Run | Sessions | Checks | +|---|---|---| +| `plugin-hooks` | 7 | 8 passed, 0 failed | +| `plugin-docs` | 3 | 7 passed, 0 failed | +| `agent-spec-model` without the reference-docs note | 1 | 2 passed, 0 failed | + +## Results + +Haiku 5.5 matched Haiku 4.5 on every check. In `plugin-docs`, `agent-spec-model`'s agent read the +plugin's `SPEC-MODEL.md` and the project's files without asking. That session then stopped at its +$0.75 cap before the final reply (see the CLI note below); its checks had already passed. + +## Without the note + +Decision 0019 added one line to the plugin's skills and agents — the reference docs they name are the +plugin's copies, outside the project — after `spec-analyzer`, on Haiku 4.5, read `docs/SPEC-MODEL.md` +in the project although its instructions gave the plugin's full path. To see whether Haiku 5.5 still +needs it, `agent-spec-model` ran once from a copy of the checkout with the line removed from +`plugins/adf/agents/spec-analyzer.md`, and a $2 cap. + +The agent read the plugin's `SPEC-MODEL.md` at the full path and never looked for one in the project, +and the session ended normally. + +| Note | Model | The agent read the plugin's spec model | +|---|---|---| +| None | Haiku 4.5 (0019) | No — `docs/SPEC-MODEL.md` in the project | +| Present | Haiku 4.5 (0019) | Yes | +| Present | Haiku 5.5 | Yes | +| None | Haiku 5.5 | Yes | + +The note stays. One run doesn't show the failure can't happen, the line states a fact about where the +docs are, and in projects the agent runs on Opus. What changes is its evidence: the failure that +motivated it was Haiku 4.5's, and Haiku 5.5 didn't repeat it. + +## The CLI doesn't know Haiku 5.5 yet + +Claude Code v2.1.286 logs `unrecognized_model` for `claude-haiku-5-5` and the API serves it, but the +CLI prices it at a rate other than Haiku 5.5's: these runs report about $4.26 API-equivalent in all, +more than the Haiku 4.5 runs, for a model priced lower. The session budget cap is checked against the +same estimate, which is why `agent-spec-model` stopped at $0.75. Update Claude Code before reading the +costs or tightening the caps. The suites' default `haiku` alias resolves per CLI version; pass +`--models claude-haiku-5-5` to be sure which Haiku runs. + +## What Haiku doesn't stand in for + +These suites test plumbing — that hooks fire and their messages arrive, and that a read rule and a +path reach the plugin's files — where the cheapest model is enough. They don't test judgment, and +they don't test how Opus 5.5 or Sonnet 5.5 return text written just before a tool call, which can come +back as a progress-update thinking block. The `triage` and `debug` suites stay on `sonnet` and `opus`. diff --git a/evals/dynamic/run-session-evals.sh b/evals/dynamic/run-session-evals.sh index aa4069b..1a40bd1 100755 --- a/evals/dynamic/run-session-evals.sh +++ b/evals/dynamic/run-session-evals.sh @@ -257,7 +257,7 @@ PY [ -n "$DOCS_RULE" ] || dirs+=("$FWC") fi if [ -n "$DOCS_RULE" ]; then # reads in the project need no rule; the plugin's folder, the committed one - reads=() run_env=(ANTHROPIC_DEFAULT_OPUS_MODEL=claude-haiku-4-5-20251001) + reads=() run_env=(ANTHROPIC_DEFAULT_OPUS_MODEL=claude-haiku-5-5) grep -q '' "$input" || reads=("Read(/$FWC/plugins/adf/**)") fi [ -d "$SOURCE" ] && [ "$SOURCE" != "$FWC" -o ${#dirs[@]} -eq 0 ] && dirs+=("$SOURCE")