Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,32 @@ Bump `schema_version` for any change that alters how a file must be read.
2. For positive prompts, run the agent in a scratch project and check the `rubric` items against the resulting files and command transcript. This scores the procedure.

The two scores should be reported separately and versioned with the model and catalog commit. Live-model runs are opt-in and must stay out of default CI.

## Scratch fixtures for procedure trials

`evals/fixtures/<fixture-id>/` holds a small, versioned set of starting projects so reviewers run a procedure trial on the same inputs. Each has `fixture.json` (metadata) and `files/` (the starting tree). Scenario files above are unchanged (schema version 1); a fixture refers to its scenario by file name, the exact prompt text and the exact rubric item text.

| Fixture | Skill | Prompt (exact) |
|---|---|---|
| `setup-clean-parent` | `setting-up-python-libraries` | Create a new Python library called acme-utils with pytest, ruff and CI ready to go. |
| `security-inert-examples` | `auditing-python-security` | Audit my Python library for security vulnerabilities before we release. |
| `ci-gate-mismatch` | `reproducing-ci-locally` | The lint job fails in CI but passes on my machine; help me reproduce it locally. |

All other prompts remain activation-only; they have no procedure fixture yet. The insecure code in `security-inert-examples` is inert: never called, no payloads, no credentials. Do not import or run it.

### Manual trial workflow

1. Materialize into a NEW directory outside this checkout (existing destinations and unknown or unsafe fixture ids are refused before anything is written; nothing is executed or installed):

```bash
uv run --no-project --python 3.12 python evals/fixtures/materialize.py ci-gate-mismatch /tmp/trial-ci-1
```

It prints the fixture id and `tree_sha256`.
2. Follow the fixture's `setup` list, give the agent the `prompt` verbatim with the skill installed, and let it work in that directory.
3. Compare the result with the fixture's `expected_observations` and score the listed `rubric_items` from the resulting files and transcript. For `setup-clean-parent`, assess the generated child directory against those scaffold rubric items.
4. Record with the trial: fixture id, `tree_sha256`, the catalog commit, the agent client and model version, and which rubric items were observed, not observed, or not assessable. Worksheets and reports (proposed separately) can adopt these fields unchanged.

### Fixture validation is not agent success

`tests/test_eval_fixtures.py` checks deterministic facts only: each prompt and rubric text still exists in its scenario, starting-file hashes are unchanged, the materializer is repeatable and refuses unsafe requests, and the `ci-gate-mismatch` reproduction behaves as documented (local command exits 0, the workflow command exits 1, aligned configuration plus the code fix exit 0, using pinned `ruff==0.9.10` through `uvx`; skipped when `uv` is absent). It never runs an agent. A passing suite says the inputs are intact, not that any skill works; no success rate is measured or claimed here. Changing a fixture file means regenerating its hashes in `fixture.json` and re-reading the affected trials.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: ci
on: [pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- run: uvx --from ruff==0.9.10 ruff check --isolated --select E4,E7,E9,F .
3 changes: 3 additions & 0 deletions evals/fixtures/ci-gate-mismatch/files/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.PHONY: lint
lint:
uvx --from ruff==0.9.10 ruff check .
9 changes: 9 additions & 0 deletions evals/fixtures/ci-gate-mismatch/files/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[project]
name = "sample-pkg"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = []

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
4 changes: 4 additions & 0 deletions evals/fixtures/ci-gate-mismatch/files/ruff.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
extend-exclude = ["tests"]

[lint]
select = ["E4", "E7", "E9", "F"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
def add(a: int, b: int) -> int:
return a + b
7 changes: 7 additions & 0 deletions evals/fixtures/ci-gate-mismatch/files/tests/test_add.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
import os

from sample_pkg import add


def test_add():
assert add(1, 2) == 3
80 changes: 80 additions & 0 deletions evals/fixtures/ci-gate-mismatch/fixture.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
{
"fixture_version": 1,
"id": "ci-gate-mismatch",
"skill": "reproducing-ci-locally",
"scenario": "reproducing-ci-locally.json",
"scenario_schema_version": 1,
"prompt": "The lint job fails in CI but passes on my machine; help me reproduce it locally.",
"rubric_items": [
"The commands run locally are taken from the workflow file (working directory, flags, env, interpreter version), not assumed from a Makefile",
"The tool version CI resolves is identified and the same version is used locally",
"The final report states the exit status of each reproduced step and whether the run is green",
"A red result is not explained away without a reproduction or a cited log line"
],
"setup": [
"Materialize into a new directory and run the agent with it as the project root. Network access to fetch ruff 0.9.10 through uvx is needed.",
"Local command (what the Makefile runs): uvx --from ruff==0.9.10 ruff check . -> exit 0.",
"Workflow command (the run: line in .github/workflows/ci.yml): uvx --from ruff==0.9.10 ruff check --isolated --select E4,E7,E9,F . -> exit 1 (F401 in tests/test_add.py).",
"Cause: ruff.toml excludes tests/ locally, but the workflow runs --isolated and so lints them."
],
"expected_observations": [
"Local passes and the workflow command fails with exit 1 on the unused import in tests/test_add.py.",
"Config alignment (remove the extend-exclude line from ruff.toml) makes the local command fail the same way, exit 1.",
"After also removing the unused import, both commands exit 0.",
"A correct run identifies the pinned ruff version and the --isolated flag from the workflow file rather than the Makefile."
],
"files": {
".github/workflows/ci.yml": "8a9ceb0228ccd42d8429fed7d5abe4f376bc16b28f1d77ef2af02a88d2c55c51",
"Makefile": "a4a3a8ebfdeae1d8a823bd2a9d5e3e1e6f19e884027b28d6f7fc6aadc1551819",
"pyproject.toml": "2e0dd90ae65d2564442d9da7c21619e9e7b41225205efd4167c83977a77b2d82",
"ruff.toml": "8e95761db53ad1afa902fd7d23c63e0e5204b1a56e0efd3d2c942487eeaa4f91",
"src/sample_pkg/__init__.py": "b414e3e8a1cc84d091471727544c353f60db59fc2bea10ee562ece77410915be",
"tests/test_add.py": "a4d9f900d266670b34d84e1453ecba46d1f29ad16dfc036a8ef6aa675df94e76"
},
"tree_sha256": "af714c7e6469f90c776d7349fbcf5531e287d2390fd36c6d0ecde5bac8e255fb",
"reproduction": {
"tool": "ruff==0.9.10",
"local": {
"argv": [
"uvx",
"--from",
"ruff==0.9.10",
"ruff",
"check",
"."
],
"source": "Makefile",
"expect_exit": 0
},
"workflow": {
"argv": [
"uvx",
"--from",
"ruff==0.9.10",
"ruff",
"check",
"--isolated",
"--select",
"E4,E7,E9,F",
"."
],
"source": ".github/workflows/ci.yml",
"expect_exit": 1
},
"align": [
{
"file": "ruff.toml",
"old": "extend-exclude = [\"tests\"]\n\n",
"new": "",
"then_local_exit": 1
},
{
"file": "tests/test_add.py",
"old": "import os\n\n",
"new": "",
"then_local_exit": 0,
"then_workflow_exit": 0
}
]
}
}
128 changes: 128 additions & 0 deletions evals/fixtures/materialize.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#!/usr/bin/env python3
"""Copy a versioned evaluation fixture into a new directory.

Only copies files; nothing from the fixture is executed or installed.
Usage: materialize.py FIXTURE_ID DESTINATION
"""
import argparse
import hashlib
import json
import re
import shutil
import sys
from pathlib import Path

FIXTURES = Path(__file__).resolve().parent
EVALS = FIXTURES.parent
ID_PATTERN = re.compile(r"^[a-z0-9]+(-[a-z0-9]+)*$")
MANIFEST_VERSION = 1


class FixtureError(Exception):
pass


def sha256_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()


def hash_tree(files_dir: Path) -> dict[str, str]:
"""Map relative POSIX path -> sha256 for every file; reject symlinks."""
hashes = {}
for path in sorted(files_dir.rglob("*")):
if path.is_symlink():
raise FixtureError(f"symlink in fixture: {path.relative_to(files_dir)}")
if path.is_file():
hashes[path.relative_to(files_dir).as_posix()] = sha256_bytes(path.read_bytes())
return hashes


def tree_digest(hashes: dict[str, str]) -> str:
text = "".join(f"{name}\0{digest}\n" for name, digest in sorted(hashes.items()))
return sha256_bytes(text.encode())


def load_manifest(fixture_id: str, root: Path = FIXTURES) -> dict:
if not isinstance(fixture_id, str) or not ID_PATTERN.fullmatch(fixture_id):
raise FixtureError(f"unsafe fixture id: {fixture_id!r}")
manifest_path = root / fixture_id / "fixture.json"
if not manifest_path.is_file():
raise FixtureError(f"unknown fixture: {fixture_id}")
manifest = json.loads(manifest_path.read_text())
if manifest.get("fixture_version") != MANIFEST_VERSION or manifest.get("id") != fixture_id:
raise FixtureError(f"bad manifest header for {fixture_id}")
return manifest


def input_errors(manifest: dict, root: Path = FIXTURES) -> list[str]:
"""Compare the starting files on disk with the recorded hashes."""
files_dir = root / manifest["id"] / "files"
try:
actual = hash_tree(files_dir)
except FixtureError as exc:
return [str(exc)]
recorded = manifest["files"]
errors = [f"unexpected file: {n}" for n in sorted(set(actual) - set(recorded))]
errors += [f"missing file: {n}" for n in sorted(set(recorded) - set(actual))]
errors += [f"hash changed: {n}" for n in sorted(set(actual) & set(recorded)) if actual[n] != recorded[n]]
if not errors and tree_digest(actual) != manifest["tree_sha256"]:
errors.append("tree_sha256 does not match files")
return errors


def prompt_errors(manifest: dict, evals_dir: Path = EVALS) -> list[str]:
"""Check the exact prompt and rubric text still exist in the scenario file."""
scenario_path = evals_dir / manifest["scenario"]
if not scenario_path.is_file():
return [f"scenario file missing: {manifest['scenario']}"]
scenario = json.loads(scenario_path.read_text())
errors = []
if scenario.get("schema_version") != manifest["scenario_schema_version"]:
errors.append("scenario schema_version differs from the fixture's")
if scenario.get("skill") != manifest["skill"]:
errors.append("scenario skill differs from the fixture's")
if manifest["prompt"] not in scenario.get("positive_prompts", []):
errors.append(f"prompt not found in positive_prompts: {manifest['prompt']!r}")
rubric = scenario.get("rubric", [])
errors += [f"rubric item not found: {item!r}" for item in manifest["rubric_items"] if item not in rubric]
return errors


def materialize(fixture_id: str, destination: Path, root: Path = FIXTURES) -> dict:
"""Validate everything first, then create destination and copy files."""
manifest = load_manifest(fixture_id, root)
errors = input_errors(manifest, root)
if errors:
raise FixtureError("fixture inputs invalid: " + "; ".join(errors))
destination = Path(destination)
if destination.exists() or destination.is_symlink():
raise FixtureError(f"destination already exists: {destination}")
if not destination.parent.is_dir():
raise FixtureError(f"destination parent is not a directory: {destination.parent}")
source = root / fixture_id / "files"
destination.mkdir()
for name in manifest["files"]:
target = destination / name
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(source / name, target)
return manifest


def main(argv=None) -> int:
parser = argparse.ArgumentParser(description="Copy an evaluation fixture into a new directory.")
parser.add_argument("fixture_id")
parser.add_argument("destination", type=Path)
args = parser.parse_args(argv)
try:
manifest = materialize(args.fixture_id, args.destination)
except FixtureError as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
print(f"fixture: {manifest['id']}")
print(f"tree_sha256: {manifest['tree_sha256']}")
print(f"destination: {args.destination}")
return 0


if __name__ == "__main__":
sys.exit(main())
5 changes: 5 additions & 0 deletions evals/fixtures/security-inert-examples/files/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# sample-service (evaluation fixture)

Anonymous project for auditing practice. `src/sample_service/` contains
deliberately unsafe patterns that are INERT: nothing here is called, no payload
is included, and there are no credentials. Do not import or execute it.
9 changes: 9 additions & 0 deletions evals/fixtures/security-inert-examples/files/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[project]
name = "sample-service"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = []

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"""INERT EVALUATION EXAMPLE: shell command built from a parameter."""
import subprocess


def archive(directory: str):
# Unsafe on purpose: shell=True with an interpolated argument.
return subprocess.run(f"tar -czf backup.tgz {directory}", shell=True)
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"""INERT EVALUATION EXAMPLE: deserializing bytes from an unspecified source."""
import pickle


def load_state(blob: bytes):
# Unsafe on purpose: pickle can run code when loading untrusted bytes.
return pickle.loads(blob)
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
"""INERT EVALUATION EXAMPLE: SQL built by string formatting."""
import sqlite3


def find_user(conn: sqlite3.Connection, name: str):
# Unsafe on purpose: caller-supplied text is interpolated into SQL.
return conn.execute(f"SELECT * FROM users WHERE name = '{name}'").fetchall()


def find_user_safely(conn: sqlite3.Connection, name: str):
return conn.execute("SELECT * FROM users WHERE name = ?", (name,)).fetchall()
35 changes: 35 additions & 0 deletions evals/fixtures/security-inert-examples/fixture.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"fixture_version": 1,
"id": "security-inert-examples",
"skill": "auditing-python-security",
"scenario": "auditing-python-security.json",
"scenario_schema_version": 1,
"prompt": "Audit my Python library for security vulnerabilities before we release.",
"rubric_items": [
"Runs skills/python/security-audit/scripts/security_scan.py (or the individual scanners) against the target project path",
"Reports which scanners ran and which did not; a scanner that was unavailable is reported as not run, never as a clean result",
"Findings are listed with severity and file location, with blocking findings (high severity, vulnerable dependencies, detected secrets) called out separately",
"Each reported injection or deserialization issue includes a concrete remediation"
],
"setup": [
"Materialize into a new directory; run the agent with that directory as the project root.",
"Scanners (bandit, pip-audit, semgrep, detect-secrets) may or may not be installed; record which were available. Do not install them on the agent's behalf if you want to observe the unavailable-scanner behaviour.",
"The fixture has no dependencies, so pip-audit has nothing to find."
],
"expected_observations": [
"Source-level findings a reviewer should expect: SQL built by f-string in queries.find_user (find_user_safely is the correct contrast); shell=True with an interpolated argument in commands.archive; pickle.loads of caller-supplied bytes in loaders.load_state.",
"Each finding should come with a concrete remediation (parameterised query, argument list without a shell, a data-only format).",
"No secrets exist in the fixture; a report claiming a detected secret is a false positive to note.",
"Scanners differ in which of the three they flag and at what severity; do not require identical findings across tools. A scanner that did not run must be reported as not run.",
"No live vulnerability database is needed to check the expected source findings."
],
"files": {
"README.md": "caaa84058996dd7c9b60706e83ed9c9097ac69fde3a867bff563a27905d639ce",
"pyproject.toml": "8d4c2993bac0c21b665a7ccdd4aadfc5df80195f658eb20d856bc3797009b6e2",
"src/sample_service/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"src/sample_service/commands.py": "9b860ef030e16ffdab6b0defa59431d6a8463ca681903e4dfe5bf3533c27e6c5",
"src/sample_service/loaders.py": "b27f6599d0e82301c5f5818187f9e1839b4f450d31fb15316a1da630dc67fbdf",
"src/sample_service/queries.py": "603255f7a247c7c984e72a68c95649b7f7e957699e47a34306b74be1eb1dbfd4"
},
"tree_sha256": "ed64dfe7d2c10da2b8b277cf613c4bef5955832cf985509a7ee7307853b2dc3b"
}
5 changes: 5 additions & 0 deletions evals/fixtures/setup-clean-parent/files/PARENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Scratch parent directory

This directory is intentionally empty of Python projects. Create the new
library as a child directory here; do not run anything inside the skills
repository checkout.
Loading
Loading