Repository navigation
evals: reproducible scratch fixtures for pilot skill trials - #187
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #183
Summary
evals/fixtures/<id>/{fixture.json,files/}for one positive prompt per pilot skill:setup-clean-parent,security-inert-examples,ci-gate-mismatch. Scenario JSON and schema version 1 are untouched; fixtures reference scenario file + exact prompt + exact rubric text.evals/fixtures/materialize.py: stdlib-only,FIXTURE_ID DEST. Validates id, source hashes and destination before the first write; copies only, never executes/installs; prints id andtree_sha256for recording with a trial.ruff check .exits 0 (ruff.toml excludes tests/), workflowruff check --isolated ...exits 1 (F401), config alignment makes local fail identically, removing the import makes both exit 0. Pinnedruff==0.9.10.evals/README.md: fixture table, manual trial workflow, and the distinction between fixture validation and observed agent success (no rates claimed).Verification
uv run --no-project --python 3.12 --with 'pytest>=9' python -m pytest tests: 76 passed.python -m unittest discover -s tests: OK.tests/test_eval_fixtures.pyruns the real materializer and the real pinned-ruff exit statuses (skipped withoutuv). Stale prompt/rubric mappings and changed/extra fixture files each fail their check; refusal tests assert an external sentinel tree is unchanged.Screenshots
n/a — no user-visible change