Record every eval run as a schema-versioned, replayable run record

[richardmhope]

What & why

Part of #74.

Every evals/run_evals.py run now writes a run record, run-record.json. It isn't tied to one AI provider and has a versioned schema (evals/run-record.schema.json); keys are sorted and line endings are LF. It records:

• the skilldeck version, the git commit, whether the checkout had uncommitted changes, and a digest of the runner script itself;

• for each fixture, a digest of its files:

  • git-tracked files plus untracked files git doesn't ignore;
  • junk such as __pycache__ and .DS_Store excluded;
  • executable bits included.

  The review repo is built from exactly these files;

• each skill's version, canonical digest and rendered digest. The rendered digest is the install stamp's hash=, which is also skilldeck catalog's rendered_sha256 for that adapter; a test checks all three agree;

• the harness: its exact command template, version and model;

• one entry per planned run, with its outcome (passed, failed, agent_failed, timed_out, error or not_run), timing, exit code, number of findings and the scorer's reasons.

Raw reports and stderr stay in the work directory, and the record points to them by relative path. --include-reports embeds them in the record instead.

The record is always written once runs start. Failures to build the repo or score a report, including SkillError, are recorded against that run. An interrupt (exit 130) or a runner bug still writes the record, keeping the runs already done.

New options:

• --harness claude|codex|custom: claude -p and codex exec presets, each paired with its adapter. --model is passed through {model}; a model value starting with - is refused. The version is probed only from the preset's own executable, so a wrapper command records null.
• Agents run with stdin closed. Without this, codex exec blocks reading non-TTY stdin.
• --dry-run validates fixtures and prints the plan without running an agent or the version probe. It runs only read-only git ls-files.
• --max-runs (default 50) refuses an oversized plan before anything starts. Runs stay sequential.
• --replay RECORD re-runs a record's configuration:
  • it refuses if any fixture, skill, rendered skill or prompt has changed;
  • it runs only a built-in preset's own command, unless --trust-record-command is given. A record is data, and replaying a custom command executes it.

Tests (tests/test_eval_runs.py, 76 tests) use stand-in agents only. A new autouse guard in tests/conftest.py fails any test that sends a bare claude or codex to subprocess.run.

Review. An independent review raised findings, all fixed here:

• stdin inheritance;
• lost records after unexpected errors;
• replay executing arbitrary commands;
• the wording of rendered_sha256;
• version probes crediting wrapper programs;
• recorded prompts not compared on replay;
• the fixture digest including junk files and ignoring the exec bit;
• nits: schema_version: true, SHA-256 commit IDs, absolute paths in messages.

Still open for #74, because these need paid runs: variance reports from repeated real runs, a real check of the codex preset, and token/cost capture (for example by parsing Claude's --output-format json).

Tracking: #94

Type of change

• Bug fix
• New feature
• New or updated skill
• Docs only
• Refactor / internal

Checklist

• Ran uv run --extra dev ruff check . && uv run --extra dev ruff format --check . && uv run --extra dev mypy && uv run --extra dev pytest (895 passed, also with -W error::EncodingWarning and --resolution lowest-direct; plus mypy --strict evals/run_evals.py)
• Added or updated tests
• Updated docs where relevant (evals/README.md)
• Added a CHANGELOG.md entry under ## [Unreleased]
• For a skill change: bumped that skill's version in meta.yaml (no skill content changed)

🤖 Generated with Claude Code

https://claude.ai/code/session_01HtiCGzpikMrkDYBkfQG5CX

---

Generated by Claude Code