[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

A platform checks a result against criteria an author wrote, such as a smoke score: {min: 0} or a board column on pass@1. A misspelled name shows up only as a key missing from report.json, after the run. This PR makes each task's report metric names known before the run, in the dry-run JSON.

• tasks.<task>.metrics is {"names": [...], "patterns": [...]}, or null when unknown. names are the numeric keys the report may write. A pattern such as score_{category} stands for keys built from the data; a run whose data lacks that group does not write the key. It sits next to feat(cli): sieval run --dry-run reports each task's stage plan #220's stages and needs_inference, and is also present next to a refused task's error.
• Declared, with a test as the check. Deriving the names by running report() on stub records would need a stub per task's feedback_result shape, and it would miss the data-dependent and n > 1 keys. So instead every task unit test wraps the task's report() and fails if the report writes a numeric key its declaration lacks.
• Built-in declarations live in sieval/meta/report_metrics.py, keyed by the class that defines report. A task module's source is part of its stage identities, so a class attribute there would change every stage identity and rerun every resumed directory from postprocess. sieval.meta is outside the grader walk.
• Plugin tasks declare on the class: report_metrics = report_metrics("score", patterns=("score_{subset}",)), from sieval.core.tasks.report_metrics. A subclass inherits a declaration together with the report it describes. A subclass with its own report is unknown until it declares. So is a report defined in another module than its class: a patch, or a shared function bound in the class body, which look the same at runtime (wrap it in a def). A generative task also gets the runner's n_truncated and n_scored_rollouts.
• A malformed declaration fails the dry-run. A report_metrics that is not a ReportMetrics or None fails a new report_metrics check, also when a nearer report shadows it, rather than reading as null, which a check takes as "unknown, do not refuse". Only a class that does not resolve is advisory (null); the imports check reports it. ReportMetrics | "score" raises instead of adding one name per character.
• Text output: a task whose metrics are unknown gets , report metrics unknown on its line.

Identity impact. None. For all 84 registered tasks, stage_code() and legacy_grader_keys() are byte-identical before and after this change, checked on this branch's base. A test also asserts that no built-in task class carries the attribute and that sieval.meta is outside GRADER_PACKAGES.

_task_json. One parameter and one key are added next to #220's additions. No other change in that function.

Related Issues

Closes #223. Follows #217 and #220.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass (pdm run pytest)

pytest tests/unit tests/integration tests/acceptance -m "not stress and not benchmark" -n auto --cov --cov-fail-under=95, installed with every CI group, passes at 97.11% coverage on macOS, rebased on #224. The one consistent failure is test_exec_quotebench_anchor.py, which also fails on main on macOS. In one loaded run test_signals.py::test_sigterm_runs_shielded_cleanup_and_exits_143 timed out once; it passes alone three times out of three. tests/performance -m "not stress" passes. After the review fixes, on Linux: tests/unit/tasks with test_lazy_launch.py (1969 passed) and tests/unit/cli tests/unit/meta tests/unit/core/tasks (2233 passed, 1 skipped). ty check, sync_package_stubs.py --check, check_preflight.py and pre-commit run -a are clean.

• tests/unit/tasks/conftest.py: every built-in task's report() is checked against its declaration in every test that runs it. Removing acc_norm from HellaSwag's entry fails its test, naming the key.
• tests/unit/meta/test_report_metrics.py: every registered task is declared, every entry names a class that defines report, every task but T-Eval declares score, generative tasks get the runner's keys, and the table stays outside every stage identity.
• tests/unit/core/tasks/test_report_metrics.py: pattern matching (whole name, non-empty placeholder, literal @), the rejection of a pattern with no literal part, inheritance, a report below the declaration, a patched report, None, table versus attribute precedence, a union with a bare str, and a malformed declaration below a nearer report. 100% line and branch coverage of sieval/core/tasks/report_metrics.py.
• tests/integration/test_lazy_launch.py runs the real CLI. A declared plugin task reports its names on a fresh run, a resume and a refused resume. With a patched report its metrics are null.
• tests/unit/cli/test_run.py: _run_dry_run, through sieval.cli.validation.task_report_metrics_json, with a built-in task, one with patterns, an undeclared subclass, an unresolvable class and an entry with no class. The last three are null and fail nothing. A malformed plugin declaration is null beside a failed report_metrics check naming the task, and the other tasks keep their metrics.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it