Skip to content

evals: local trial worksheets with separate activation and procedure reports - #188

Merged
wdm0006 merged 1 commit into
mainfrom
autopilot/issue-176
Nov 15, 2026
Merged

wdm0006 merged 1 commit into
mainfrom
autopilot/issue-176

Conversation

@wdm0006

@wdm0006 wdm0006 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Closes #176

Summary

Adds evals/trial.py (stdlib only, no network, no agent launch) with init (unscored worksheet to a new file, never overwrites, refuses evals/ so it stays out of the scenario glob) and report (validate, then text/JSON). The worksheet snapshots scenarios (file and content hashes) and records trial id/date, catalog commit, client, model and reviewer. Reports read only the snapshot, so historical worksheets stay readable after scenarios change (--check-current notes drift). No scenario schema change, no new dependency, no transcripts committed.

Semantics: activation passes on expected-skill presence (positive) or expected skill present and pilot skill absent (negative); extra activated skills are preserved. Unobserved/pending are never passes and are excluded from the denominator with counts shown. Procedure is reported separately (scored rate, rate over all applicable with unscored as not passed, N/A visibly excluded; N/A needs a reason). Zero denominators print undefined.

Verification

  • uv run --no-project --python 3.12 --with 'pytest>=9' python -m pytest tests: 93 passed; python -m unittest discover -s tests: OK. No network/model access.
  • tests/test_eval_trials.py runs the real helper on a mixed fixture and asserts exact per-case outcomes and separate counts (extra skill, pilot wrongly activating on a negative, N/A, incomplete, all-unscored), plus diagnostics for unknown/duplicate cases, bad statuses, missing provenance, mismatched snapshots.
  • Mutations (verified live, bytecode disabled): counting unscored as pass fails 4 tests; letting procedure results alter activation counts fails 3.
  • CLI run end to end on the three current scenarios (18 cases, 48 rubric items); second init to same path refused.

Screenshots: n/a — no user-visible change.

@wdm0006
wdm0006 merged commit ab8f96c into main Nov 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[autopilot] Record local skill evaluation trials and report separate outcomes

1 participant