feat(cli): sieval run --dry-run reports each task's report metric names

CHANGELOG.md

@@ -29,7 +29,8 @@ Planned as 0.12.0.
 - `sieval.core.runners.task_resume_plan` / `plan_resume`: the per-stage resume plan, without running anything.
 - `TaskRunner.plan_run` / `MultiTaskRunner.plan_run` (`RunPlan`): the runner's per-sample plan of a run, `to_infer` included, read-only; the one `sieval run` launches by.
 - **`sieval run --dry-run` reports what the run would do** (closes #214). `-o json` adds `gpu`, `models` (each served model's plan as `infer_plans.yaml` would record it, each process its launch starts, and `needs_inference`, whether this run starts it), `tasks` (each task's `start`, `resume_plan` with the counts `runs[].resume` will record, `null` for a fresh start, `new`, `to_infer` and `to_grade`) and `needs_inference`, whether any engine starts. The plans come from the call `sieval run` launches by. A resume that would be refused, or planning that raises, fails the new `resume_plan` check; when the dry-run stops before planning, `tasks` and `needs_inference` are `null`. It launches no engine, writes no result directory, reads `--resume` directories in place, and runs on a node without GPUs given `SIEVAL_TARGET_GPU`. Planning loads the datasets and starts each vendored service a task declares, as the run does, so a dry-run that passes its checks costs a session setup. The existing checks are unchanged.
-- `sieval run --dry-run` reports each task's `stages` (per-stage `same`/`differs`/`unknown`, and `rerun_from`, the stage a resume reruns from) and its own `needs_inference`; `RunPlan` gains `stages` (closes #219).
+- `sieval run --dry-run` reports each task's `stages` (per-stage `same`/`differs`/`unknown`, and `rerun_from`, the stage a resume reruns from), its own `needs_inference`, and its `metrics` (below); `RunPlan` gains `stages` (closes #219).
+- **A task's report metric names are known before it runs** (closes #223): `tasks.<task>.metrics` in the dry-run is `{names, patterns}`, or `null` when unknown, and a malformed declaration fails a new `report_metrics` check. Built-in tasks are declared in `sieval/meta/report_metrics.py`, outside every stage identity; a plugin task declares `report_metrics = report_metrics(...)` (`sieval.core.tasks.report_metrics`) on its class. Every task test checks its report against the declaration.
 
 ### Removed
 

docs/guide/configuration.md

@@ -422,7 +422,8 @@ in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-dec
       "new": 1,
       "to_infer": 1,
       "to_grade": 1,
-      "needs_inference": true
+      "needs_inference": true,
+      "metrics": {"names": ["correct", "score", "total"], "patterns": ["score_{category}"]}
     }
   },
   "needs_inference": true
@@ -448,6 +449,10 @@ in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-dec
   `legacy_grader` is `kept` or `regrade` for a schema-1 directory;
   `legacy_preprocess` is `true` for a schema-1 or schema-2 directory, compared
   by rendering its recorded samples once. `stages` is `null` for a fresh start.
+- `tasks.<task>.metrics` is the numeric keys the task's `report.json` may
+  write: `names` as spelled, `patterns` such as `score_{category}` for keys
+  built from the data. `null` when unknown. A malformed declaration fails the
+  `report_metrics` check. See [Report metric names](metrics.md#report-metric-names).
 - `needs_inference` is whether the run starts any engine. A task that infers
   through an `api_base` model counts in its `to_infer` but starts none.
 

docs/guide/metrics.md

@@ -663,6 +663,39 @@ unit), `timeouts` (the code family), `exact_match` / `flexible_exact_match`
 `score_<category>` (MMLU, MMLU-Pro), and UGMathBench's `eacc` / `aacc` / `cacc` /
 `delta`. `score_key` always names which of them the headline came from.
 
+## Report metric names
+
+`sieval run --dry-run -o json` reports the keys each task's report may write, as
+`tasks.<task>.metrics` ([What a run would do](configuration.md#what-a-run-would-do)),
+so a criterion naming a metric the task never reports can be refused before the run.
+
+```json
+{"names": ["fails", "n_problems", "score"], "patterns": ["score_{category}"]}
+```
+
+`names` are written as spelled; a `{placeholder}` in a pattern stands for one or
+more characters taken from the data. Either may be absent from a run whose data
+lacks it, or that cannot compute it. Only numeric keys are named; a generative
+task also gets the runner's `n_truncated` and `n_scored_rollouts`.
+
+Built-in tasks are declared in `sieval/meta/report_metrics.py`, not on the class,
+because a task module's source is in its stage identities. A plugin task declares
+on its class:
+
+```python
+from sieval.core.tasks.report_metrics import report_metrics
+
+
+class MyTask(Task[...]):
+    report_metrics = report_metrics("score", "accuracy", patterns=("score_{subset}",))
+```
+
+A subclass inherits the declaration with its parent's `report`. Metrics are
+`null` (unknown, so a check should not refuse) when a subclass defines its own
+`report` without declaring, when `report` comes from another module, or when
+`report_metrics = None`. A malformed declaration fails the dry-run's
+`report_metrics` check.
+
 ## Reading a report across versions
 
 Adding a key is non-destructive: downstream consumers read `score` and

sieval/cli/output.py

@@ -366,13 +366,14 @@ def _render_dry_run_plan(data: dict) -> None:
         )
         rerun_from = (entry.get("stages") or {}).get("rerun_from")
         log_user(
-            "task {}: {}{}{}, {} to infer, {} to grade",
+            "task {}: {}{}{}, {} to infer, {} to grade{}",
             name,
             entry["start"],
             f" ({outcomes})" if outcomes else "",
             f", from {rerun_from}" if rerun_from else "",
             entry["to_infer"],
             entry["to_grade"],
+            "" if entry.get("metrics") else ", report metrics unknown",
         )
     log_user("needs inference: {}", "yes" if data.get("needs_inference") else "no")
 

sieval/cli/run.py

@@ -399,16 +399,22 @@ def _plan_summary(plans: Mapping[str, RunPlan | Exception]) -> str:
     return f"{len(ok)} task(s): {to_infer} sample(s) to infer, {to_grade} to grade"
 
 
-def _task_json(model: str | None, plan: RunPlan | Exception) -> dict[str, object]:
+def _task_json(
+    model: str | None,
+    plan: RunPlan | Exception,
+    metrics: dict[str, list[str]] | None = None,
+) -> dict[str, object]:
     """A task's entry in the dry-run report.
 
     ``resume_plan`` is the plan's ``outcomes``, the counts ``runs[].resume``
     records, and ``stages`` its stage comparison; both ``None`` for a fresh
     start, which has no recorded samples. ``needs_inference`` is
     :func:`_infers`, the test :func:`_models_to_launch` launches by.
+    ``metrics`` is :func:`~sieval.cli.validation.task_report_metrics_json`,
+    kept beside a refusal.
     """
     if isinstance(plan, Exception):
-        return {"model": model, "error": cli_error_message(plan)}
+        return {"model": model, "error": cli_error_message(plan), "metrics": metrics}
     return {
         "model": model,
         "start": plan["start"],
@@ -420,6 +426,7 @@ def _task_json(model: str | None, plan: RunPlan | Exception) -> dict[str, object
         "to_infer": plan["to_infer"],
         "to_grade": plan["to_grade"],
         "needs_inference": _infers(plan),
+        "metrics": metrics,
     }
 
 
@@ -440,7 +447,7 @@ async def _run_dry_run(
     starts each vendored service a task declares, as the run does.
     """
 
-    from sieval.cli.validation import run_dry_run
+    from sieval.cli.validation import run_dry_run, task_report_metrics_json
 
     def checks(
         infer_plans: dict[str, dict[str, Any]] | None = None,
@@ -526,9 +533,26 @@ def unplanned(result: dict[str, object]) -> dict[str, object]:
         result["n_errors"] = cast(int, result["n_errors"]) + 1
         return unplanned(result)
     refused = {t: p for t, p in plans.items() if isinstance(p, Exception)}
+    metrics: dict[str, dict[str, list[str]] | None] = {}
+    malformed: dict[str, TypeError] = {}
+    for task in plans:
+        try:
+            metrics[task] = task_report_metrics_json(loaded, task)
+        except TypeError as exc:
+            metrics[task], malformed[task] = None, exc
     result["tasks"] = {
-        task: _task_json(task_models.get(task), plan) for task, plan in plans.items()
+        task: _task_json(task_models.get(task), plan, metrics[task])
+        for task, plan in plans.items()
     }
+    if malformed:
+        checks_list.append(
+            {
+                "name": "report_metrics",
+                "ok": False,
+                "detail": "; ".join(f"{t}: {e}" for t, e in sorted(malformed.items())),
+            }
+        )
+        result["n_errors"] = cast(int, result["n_errors"]) + 1
     if refused:
         checks_list.append(
             {

sieval/cli/validation.py

@@ -704,6 +704,32 @@ def validate_eval_config_imports(cfg: dict) -> ValidationResult:
     return result
 
 
+def task_report_metrics_json(
+    cfg: Mapping[str, Any], task: str
+) -> dict[str, list[str]] | None:
+    """*task*'s report metric names and patterns, ``None`` where unknown.
+
+    See :mod:`sieval.core.tasks.report_metrics`. A class that does not resolve
+    is unknown; the imports check reports it.
+
+    Raises:
+        TypeError: if the class's ``report_metrics`` is malformed.
+    """
+    from sieval.meta.report_metrics import task_report_metrics
+
+    tasks = cfg.get("tasks")
+    task_cfg = tasks.get(task) if isinstance(tasks, Mapping) else None
+    spec = task_cfg.get("class") if isinstance(task_cfg, Mapping) else None
+    if not isinstance(spec, str):
+        return None
+    try:
+        task_cls = resolve_task_class(spec)
+    except Exception:
+        return None
+    declared = task_report_metrics(task_cls)
+    return declared.to_json() if declared is not None else None
+
+
 # ---------------------------------------------------------------------------
 # Dry-run orchestration
 # ---------------------------------------------------------------------------

sieval/core/tasks/report_metrics.py

@@ -0,0 +1,144 @@
+"""The metric names a task's ``report.json`` may write, known before it runs.
+
+``names`` are written as spelled; ``patterns`` such as ``score_{category}`` stand
+for keys built from the data. Either may be absent from one run; a name neither
+declares is one the task never writes. Only numeric keys are metrics.
+
+AI-Generated Code - Claude Opus 5.5 (Anthropic)
+"""
+
+import re
+from collections.abc import Iterable, Mapping
+from dataclasses import dataclass
+from functools import cache
+from typing import Literal
+
+#: Keys the runner adds to a generative task's report.
+RUNNER_GEN_METRICS = frozenset({"n_truncated", "n_scored_rollouts"})
+
+#: What :func:`~sieval.core.tasks.metrics.sampling_report` can write; the vote
+#: keys only with votes.
+SAMPLING_METRICS = frozenset(
+    {"pass@1", "avg@n", "pass@k", "pass^k", "n", "k", "n_short", "n_problems"}
+)
+SAMPLING_VOTE_METRICS = frozenset({"maj@k", "self_consistency"})
+
+#: What :func:`~sieval.core.tasks.metrics.health_metrics` writes.
+HEALTH_METRICS = frozenset({"n_unextracted"})
+
+_PLACEHOLDER = re.compile(r"\{[A-Za-z_][A-Za-z0-9_]*\}")
+
+
+@cache
+def _pattern_regex(pattern: str) -> re.Pattern[str]:
+    """Each placeholder matches one or more characters; the rest is literal."""
+    parts = _PLACEHOLDER.split(pattern)
+    # Without a literal part a pattern matches every name and checks nothing.
+    if len(parts) < 2 or not any(parts):
+        raise ValueError(
+            f"report metric pattern {pattern!r} needs a literal part and at least "
+            "one {placeholder}, such as 'score_{category}'"
+        )
+    return re.compile(".+".join(re.escape(part) for part in parts))
+
+
+@dataclass(frozen=True)
+class ReportMetrics:
+    """The numeric keys one task's report may write."""
+
+    names: frozenset[str]
+    patterns: tuple[str, ...] = ()
+
+    def __post_init__(self) -> None:
+        object.__setattr__(self, "names", frozenset(self.names))
+        object.__setattr__(self, "patterns", tuple(dict.fromkeys(self.patterns)))
+        if bad := sorted(n for n in self.names if not isinstance(n, str) or not n):
+            raise ValueError(f"report metric names are non-empty strings: {bad!r}")
+        for pattern in self.patterns:
+            _pattern_regex(pattern)
+
+    def match(self, name: str) -> Literal["name", "pattern"] | None:
+        """``name`` when declared, ``pattern`` when only a pattern matches."""
+        if name in self.names:
+            return "name"
+        if any(_pattern_regex(p).fullmatch(name) for p in self.patterns):
+            return "pattern"
+        return None
+
+    def __or__(self, other: "ReportMetrics | Iterable[str]") -> "ReportMetrics":
+        if isinstance(other, ReportMetrics):
+            return ReportMetrics(
+                self.names | other.names, (*self.patterns, *other.patterns)
+            )
+        if isinstance(other, str):
+            # Iterating it would add one name per character.
+            raise TypeError(f"ReportMetrics | str: wrap the name, ({other!r},)")
+        return ReportMetrics(self.names | frozenset(other), self.patterns)
+
+    def to_json(self) -> dict[str, list[str]]:
+        """``{"names": [...], "patterns": [...]}``, names sorted."""
+        return {"names": sorted(self.names), "patterns": list(self.patterns)}
+
+
+def report_metrics(*names: str, patterns: Iterable[str] = ()) -> ReportMetrics:
+    """:class:`ReportMetrics` from names written out."""
+    return ReportMetrics(frozenset(names), tuple(patterns))
+
+
+def _patched(owner: type) -> bool:
+    """Whether *owner*'s ``report`` is defined in another module than *owner*.
+
+    That is how a patch (``setattr(Task, "report", report)``) looks, and also a
+    shared function bound in the class body: at runtime the two are the same.
+    """
+    report = vars(owner)["report"]
+    function = getattr(report, "__func__", report)
+    return getattr(function, "__module__", owner.__module__) != owner.__module__
+
+
+def class_path(cls: type) -> str:
+    """``module.QualName``: how a table names a class."""
+    return f"{cls.__module__}.{cls.__qualname__}"
+
+
+def declared_report_metrics(
+    task_cls: type, table: Mapping[str, ReportMetrics] | None = None
+) -> ReportMetrics | None:
+    """The declaration describing *task_cls*'s report, or ``None`` when unknown.
+
+    The nearest class in the MRO with a ``report_metrics`` attribute, or a
+    *table* entry under its :func:`class_path`, answers. Unknown when a nearer
+    class defines ``report`` (its keys are its own), when ``report`` comes from
+    another module (a patch, or a shared function bound in the class body: wrap
+    it in a ``def``), or when the attribute is ``None``. A generative task also
+    gets :data:`RUNNER_GEN_METRICS`.
+
+    Raises:
+        TypeError: if a ``report_metrics`` attribute is not a
+            :class:`ReportMetrics` or ``None``.
+    """
+    table = table or {}
+    mro = task_cls.__mro__
+    for klass in mro:
+        found = vars(klass).get("report_metrics")
+        if found is not None and not isinstance(found, ReportMetrics):
+            raise TypeError(
+                f"{class_path(klass)}.report_metrics is a ReportMetrics or None, "
+                f"not {type(found).__name__}"
+            )
+    report_at = next((i for i, k in enumerate(mro) if "report" in vars(k)), None)
+    for depth, klass in enumerate(mro):
+        if "report_metrics" in vars(klass):
+            found = vars(klass)["report_metrics"]
+        elif class_path(klass) in table:
+            found = table[class_path(klass)]
+        else:
+            continue
+        if found is None or report_at is None or report_at < depth:
+            return None
+        if _patched(mro[report_at]):
+            return None
+        if "gen" in getattr(task_cls, "tags", ()):
+            found = found | RUNNER_GEN_METRICS
+        return found
+    return None