Skip to content

evals: reproducible scratch fixtures for pilot skill trials - #187

Merged
wdm0006 merged 1 commit into
mainfrom
autopilot/issue-183
Nov 15, 2026
Merged

wdm0006 merged 1 commit into
mainfrom
autopilot/issue-183

Conversation

@wdm0006

@wdm0006 wdm0006 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Closes #183

Summary

  • evals/fixtures/<id>/{fixture.json,files/} for one positive prompt per pilot skill: setup-clean-parent, security-inert-examples, ci-gate-mismatch. Scenario JSON and schema version 1 are untouched; fixtures reference scenario file + exact prompt + exact rubric text.
  • evals/fixtures/materialize.py: stdlib-only, FIXTURE_ID DEST. Validates id, source hashes and destination before the first write; copies only, never executes/installs; prints id and tree_sha256 for recording with a trial.
  • CI fixture: local ruff check . exits 0 (ruff.toml excludes tests/), workflow ruff check --isolated ... exits 1 (F401), config alignment makes local fail identically, removing the import makes both exit 0. Pinned ruff==0.9.10.
  • evals/README.md: fixture table, manual trial workflow, and the distinction between fixture validation and observed agent success (no rates claimed).

Verification

  • uv run --no-project --python 3.12 --with 'pytest>=9' python -m pytest tests: 76 passed. python -m unittest discover -s tests: OK.
  • New tests/test_eval_fixtures.py runs the real materializer and the real pinned-ruff exit statuses (skipped without uv). Stale prompt/rubric mappings and changed/extra fixture files each fail their check; refusal tests assert an external sentinel tree is unchanged.
  • No agent or paid calls.

Screenshots

n/a — no user-visible change

@wdm0006
wdm0006 merged commit 702645e into main Nov 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[autopilot] Add reproducible scratch fixtures for pilot skill trials

1 participant