diff --git a/CHANGELOG.md b/CHANGELOG.md index 3fc44e17..53b91786 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,8 @@ Planned as 0.12.0. - `sieval.core.runners.task_resume_plan` / `plan_resume`: the per-stage resume plan, without running anything. - `TaskRunner.plan_run` / `MultiTaskRunner.plan_run` (`RunPlan`): the runner's per-sample plan of a run, `to_infer` included, read-only; the one `sieval run` launches by. - **`sieval run --dry-run` reports what the run would do** (closes #214). `-o json` adds `gpu`, `models` (each served model's plan as `infer_plans.yaml` would record it, each process its launch starts, and `needs_inference`, whether this run starts it), `tasks` (each task's `start`, `resume_plan` with the counts `runs[].resume` will record, `null` for a fresh start, `new`, `to_infer` and `to_grade`) and `needs_inference`, whether any engine starts. The plans come from the call `sieval run` launches by. A resume that would be refused, or planning that raises, fails the new `resume_plan` check; when the dry-run stops before planning, `tasks` and `needs_inference` are `null`. It launches no engine, writes no result directory, reads `--resume` directories in place, and runs on a node without GPUs given `SIEVAL_TARGET_GPU`. Planning loads the datasets and starts each vendored service a task declares, as the run does, so a dry-run that passes its checks costs a session setup. The existing checks are unchanged. -- `sieval run --dry-run` reports each task's `stages` (per-stage `same`/`differs`/`unknown`, and `rerun_from`, the stage a resume reruns from) and its own `needs_inference`; `RunPlan` gains `stages` (closes #219). +- `sieval run --dry-run` reports each task's `stages` (per-stage `same`/`differs`/`unknown`, and `rerun_from`, the stage a resume reruns from), its own `needs_inference`, and its `metrics` (below); `RunPlan` gains `stages` (closes #219). +- **A task's report metric names are known before it runs** (closes #223): `tasks..metrics` in the dry-run is `{names, patterns}`, or `null` when unknown, and a malformed declaration fails a new `report_metrics` check. Built-in tasks are declared in `sieval/meta/report_metrics.py`, outside every stage identity; a plugin task declares `report_metrics = report_metrics(...)` (`sieval.core.tasks.report_metrics`) on its class. Every task test checks its report against the declaration. ### Removed diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index c0697e59..602574d4 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -422,7 +422,8 @@ in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-dec "new": 1, "to_infer": 1, "to_grade": 1, - "needs_inference": true + "needs_inference": true, + "metrics": {"names": ["correct", "score", "total"], "patterns": ["score_{category}"]} } }, "needs_inference": true @@ -448,6 +449,10 @@ in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-dec `legacy_grader` is `kept` or `regrade` for a schema-1 directory; `legacy_preprocess` is `true` for a schema-1 or schema-2 directory, compared by rendering its recorded samples once. `stages` is `null` for a fresh start. +- `tasks..metrics` is the numeric keys the task's `report.json` may + write: `names` as spelled, `patterns` such as `score_{category}` for keys + built from the data. `null` when unknown. A malformed declaration fails the + `report_metrics` check. See [Report metric names](metrics.md#report-metric-names). - `needs_inference` is whether the run starts any engine. A task that infers through an `api_base` model counts in its `to_infer` but starts none. diff --git a/docs/guide/metrics.md b/docs/guide/metrics.md index 9bfea18d..95135279 100644 --- a/docs/guide/metrics.md +++ b/docs/guide/metrics.md @@ -663,6 +663,39 @@ unit), `timeouts` (the code family), `exact_match` / `flexible_exact_match` `score_` (MMLU, MMLU-Pro), and UGMathBench's `eacc` / `aacc` / `cacc` / `delta`. `score_key` always names which of them the headline came from. +## Report metric names + +`sieval run --dry-run -o json` reports the keys each task's report may write, as +`tasks..metrics` ([What a run would do](configuration.md#what-a-run-would-do)), +so a criterion naming a metric the task never reports can be refused before the run. + +```json +{"names": ["fails", "n_problems", "score"], "patterns": ["score_{category}"]} +``` + +`names` are written as spelled; a `{placeholder}` in a pattern stands for one or +more characters taken from the data. Either may be absent from a run whose data +lacks it, or that cannot compute it. Only numeric keys are named; a generative +task also gets the runner's `n_truncated` and `n_scored_rollouts`. + +Built-in tasks are declared in `sieval/meta/report_metrics.py`, not on the class, +because a task module's source is in its stage identities. A plugin task declares +on its class: + +```python +from sieval.core.tasks.report_metrics import report_metrics + + +class MyTask(Task[...]): + report_metrics = report_metrics("score", "accuracy", patterns=("score_{subset}",)) +``` + +A subclass inherits the declaration with its parent's `report`. Metrics are +`null` (unknown, so a check should not refuse) when a subclass defines its own +`report` without declaring, when `report` comes from another module, or when +`report_metrics = None`. A malformed declaration fails the dry-run's +`report_metrics` check. + ## Reading a report across versions Adding a key is non-destructive: downstream consumers read `score` and diff --git a/sieval/cli/output.py b/sieval/cli/output.py index 8b9a50ca..e96542e2 100644 --- a/sieval/cli/output.py +++ b/sieval/cli/output.py @@ -366,13 +366,14 @@ def _render_dry_run_plan(data: dict) -> None: ) rerun_from = (entry.get("stages") or {}).get("rerun_from") log_user( - "task {}: {}{}{}, {} to infer, {} to grade", + "task {}: {}{}{}, {} to infer, {} to grade{}", name, entry["start"], f" ({outcomes})" if outcomes else "", f", from {rerun_from}" if rerun_from else "", entry["to_infer"], entry["to_grade"], + "" if entry.get("metrics") else ", report metrics unknown", ) log_user("needs inference: {}", "yes" if data.get("needs_inference") else "no") diff --git a/sieval/cli/run.py b/sieval/cli/run.py index 5765edb6..58dfa309 100644 --- a/sieval/cli/run.py +++ b/sieval/cli/run.py @@ -399,16 +399,22 @@ def _plan_summary(plans: Mapping[str, RunPlan | Exception]) -> str: return f"{len(ok)} task(s): {to_infer} sample(s) to infer, {to_grade} to grade" -def _task_json(model: str | None, plan: RunPlan | Exception) -> dict[str, object]: +def _task_json( + model: str | None, + plan: RunPlan | Exception, + metrics: dict[str, list[str]] | None = None, +) -> dict[str, object]: """A task's entry in the dry-run report. ``resume_plan`` is the plan's ``outcomes``, the counts ``runs[].resume`` records, and ``stages`` its stage comparison; both ``None`` for a fresh start, which has no recorded samples. ``needs_inference`` is :func:`_infers`, the test :func:`_models_to_launch` launches by. + ``metrics`` is :func:`~sieval.cli.validation.task_report_metrics_json`, + kept beside a refusal. """ if isinstance(plan, Exception): - return {"model": model, "error": cli_error_message(plan)} + return {"model": model, "error": cli_error_message(plan), "metrics": metrics} return { "model": model, "start": plan["start"], @@ -420,6 +426,7 @@ def _task_json(model: str | None, plan: RunPlan | Exception) -> dict[str, object "to_infer": plan["to_infer"], "to_grade": plan["to_grade"], "needs_inference": _infers(plan), + "metrics": metrics, } @@ -440,7 +447,7 @@ async def _run_dry_run( starts each vendored service a task declares, as the run does. """ - from sieval.cli.validation import run_dry_run + from sieval.cli.validation import run_dry_run, task_report_metrics_json def checks( infer_plans: dict[str, dict[str, Any]] | None = None, @@ -526,9 +533,26 @@ def unplanned(result: dict[str, object]) -> dict[str, object]: result["n_errors"] = cast(int, result["n_errors"]) + 1 return unplanned(result) refused = {t: p for t, p in plans.items() if isinstance(p, Exception)} + metrics: dict[str, dict[str, list[str]] | None] = {} + malformed: dict[str, TypeError] = {} + for task in plans: + try: + metrics[task] = task_report_metrics_json(loaded, task) + except TypeError as exc: + metrics[task], malformed[task] = None, exc result["tasks"] = { - task: _task_json(task_models.get(task), plan) for task, plan in plans.items() + task: _task_json(task_models.get(task), plan, metrics[task]) + for task, plan in plans.items() } + if malformed: + checks_list.append( + { + "name": "report_metrics", + "ok": False, + "detail": "; ".join(f"{t}: {e}" for t, e in sorted(malformed.items())), + } + ) + result["n_errors"] = cast(int, result["n_errors"]) + 1 if refused: checks_list.append( { diff --git a/sieval/cli/validation.py b/sieval/cli/validation.py index dfefac41..01efd380 100644 --- a/sieval/cli/validation.py +++ b/sieval/cli/validation.py @@ -704,6 +704,32 @@ def validate_eval_config_imports(cfg: dict) -> ValidationResult: return result +def task_report_metrics_json( + cfg: Mapping[str, Any], task: str +) -> dict[str, list[str]] | None: + """*task*'s report metric names and patterns, ``None`` where unknown. + + See :mod:`sieval.core.tasks.report_metrics`. A class that does not resolve + is unknown; the imports check reports it. + + Raises: + TypeError: if the class's ``report_metrics`` is malformed. + """ + from sieval.meta.report_metrics import task_report_metrics + + tasks = cfg.get("tasks") + task_cfg = tasks.get(task) if isinstance(tasks, Mapping) else None + spec = task_cfg.get("class") if isinstance(task_cfg, Mapping) else None + if not isinstance(spec, str): + return None + try: + task_cls = resolve_task_class(spec) + except Exception: + return None + declared = task_report_metrics(task_cls) + return declared.to_json() if declared is not None else None + + # --------------------------------------------------------------------------- # Dry-run orchestration # --------------------------------------------------------------------------- diff --git a/sieval/core/tasks/report_metrics.py b/sieval/core/tasks/report_metrics.py new file mode 100644 index 00000000..4cde957a --- /dev/null +++ b/sieval/core/tasks/report_metrics.py @@ -0,0 +1,144 @@ +"""The metric names a task's ``report.json`` may write, known before it runs. + +``names`` are written as spelled; ``patterns`` such as ``score_{category}`` stand +for keys built from the data. Either may be absent from one run; a name neither +declares is one the task never writes. Only numeric keys are metrics. + +AI-Generated Code - Claude Opus 5.5 (Anthropic) +""" + +import re +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from functools import cache +from typing import Literal + +#: Keys the runner adds to a generative task's report. +RUNNER_GEN_METRICS = frozenset({"n_truncated", "n_scored_rollouts"}) + +#: What :func:`~sieval.core.tasks.metrics.sampling_report` can write; the vote +#: keys only with votes. +SAMPLING_METRICS = frozenset( + {"pass@1", "avg@n", "pass@k", "pass^k", "n", "k", "n_short", "n_problems"} +) +SAMPLING_VOTE_METRICS = frozenset({"maj@k", "self_consistency"}) + +#: What :func:`~sieval.core.tasks.metrics.health_metrics` writes. +HEALTH_METRICS = frozenset({"n_unextracted"}) + +_PLACEHOLDER = re.compile(r"\{[A-Za-z_][A-Za-z0-9_]*\}") + + +@cache +def _pattern_regex(pattern: str) -> re.Pattern[str]: + """Each placeholder matches one or more characters; the rest is literal.""" + parts = _PLACEHOLDER.split(pattern) + # Without a literal part a pattern matches every name and checks nothing. + if len(parts) < 2 or not any(parts): + raise ValueError( + f"report metric pattern {pattern!r} needs a literal part and at least " + "one {placeholder}, such as 'score_{category}'" + ) + return re.compile(".+".join(re.escape(part) for part in parts)) + + +@dataclass(frozen=True) +class ReportMetrics: + """The numeric keys one task's report may write.""" + + names: frozenset[str] + patterns: tuple[str, ...] = () + + def __post_init__(self) -> None: + object.__setattr__(self, "names", frozenset(self.names)) + object.__setattr__(self, "patterns", tuple(dict.fromkeys(self.patterns))) + if bad := sorted(n for n in self.names if not isinstance(n, str) or not n): + raise ValueError(f"report metric names are non-empty strings: {bad!r}") + for pattern in self.patterns: + _pattern_regex(pattern) + + def match(self, name: str) -> Literal["name", "pattern"] | None: + """``name`` when declared, ``pattern`` when only a pattern matches.""" + if name in self.names: + return "name" + if any(_pattern_regex(p).fullmatch(name) for p in self.patterns): + return "pattern" + return None + + def __or__(self, other: "ReportMetrics | Iterable[str]") -> "ReportMetrics": + if isinstance(other, ReportMetrics): + return ReportMetrics( + self.names | other.names, (*self.patterns, *other.patterns) + ) + if isinstance(other, str): + # Iterating it would add one name per character. + raise TypeError(f"ReportMetrics | str: wrap the name, ({other!r},)") + return ReportMetrics(self.names | frozenset(other), self.patterns) + + def to_json(self) -> dict[str, list[str]]: + """``{"names": [...], "patterns": [...]}``, names sorted.""" + return {"names": sorted(self.names), "patterns": list(self.patterns)} + + +def report_metrics(*names: str, patterns: Iterable[str] = ()) -> ReportMetrics: + """:class:`ReportMetrics` from names written out.""" + return ReportMetrics(frozenset(names), tuple(patterns)) + + +def _patched(owner: type) -> bool: + """Whether *owner*'s ``report`` is defined in another module than *owner*. + + That is how a patch (``setattr(Task, "report", report)``) looks, and also a + shared function bound in the class body: at runtime the two are the same. + """ + report = vars(owner)["report"] + function = getattr(report, "__func__", report) + return getattr(function, "__module__", owner.__module__) != owner.__module__ + + +def class_path(cls: type) -> str: + """``module.QualName``: how a table names a class.""" + return f"{cls.__module__}.{cls.__qualname__}" + + +def declared_report_metrics( + task_cls: type, table: Mapping[str, ReportMetrics] | None = None +) -> ReportMetrics | None: + """The declaration describing *task_cls*'s report, or ``None`` when unknown. + + The nearest class in the MRO with a ``report_metrics`` attribute, or a + *table* entry under its :func:`class_path`, answers. Unknown when a nearer + class defines ``report`` (its keys are its own), when ``report`` comes from + another module (a patch, or a shared function bound in the class body: wrap + it in a ``def``), or when the attribute is ``None``. A generative task also + gets :data:`RUNNER_GEN_METRICS`. + + Raises: + TypeError: if a ``report_metrics`` attribute is not a + :class:`ReportMetrics` or ``None``. + """ + table = table or {} + mro = task_cls.__mro__ + for klass in mro: + found = vars(klass).get("report_metrics") + if found is not None and not isinstance(found, ReportMetrics): + raise TypeError( + f"{class_path(klass)}.report_metrics is a ReportMetrics or None, " + f"not {type(found).__name__}" + ) + report_at = next((i for i, k in enumerate(mro) if "report" in vars(k)), None) + for depth, klass in enumerate(mro): + if "report_metrics" in vars(klass): + found = vars(klass)["report_metrics"] + elif class_path(klass) in table: + found = table[class_path(klass)] + else: + continue + if found is None or report_at is None or report_at < depth: + return None + if _patched(mro[report_at]): + return None + if "gen" in getattr(task_cls, "tags", ()): + found = found | RUNNER_GEN_METRICS + return found + return None diff --git a/sieval/meta/report_metrics.py b/sieval/meta/report_metrics.py new file mode 100644 index 00000000..6c07c399 --- /dev/null +++ b/sieval/meta/report_metrics.py @@ -0,0 +1,512 @@ +"""The report metrics of each built-in task, by the class that defines ``report``. + +A table rather than an attribute on each class: a task module's source is in its +stage identities, so an attribute there would rerun every resumed directory from +``postprocess``. ``sieval.meta`` is outside the grader walk. Plugin tasks +declare on the class. Each entry is checked against the reports that +``tests/unit/tasks`` runs (``tests/unit/tasks/conftest.py``). + +AI-Generated Code - Claude Opus 5.5 (Anthropic) +""" + +from types import MappingProxyType + +from sieval.core.tasks.report_metrics import ( + HEALTH_METRICS, + SAMPLING_METRICS, + SAMPLING_VOTE_METRICS, + ReportMetrics, + declared_report_metrics, + report_metrics, +) + +_SAMPLED = SAMPLING_METRICS | HEALTH_METRICS +_VOTED = _SAMPLED | SAMPLING_VOTE_METRICS + +#: The math competitions: a sampling block with votes, headline `pass@1`. +_COMPETITION = report_metrics("score", "fails") | _VOTED +#: The code tasks: a sampling block without votes, and timeouts. +_CODE = report_metrics("score", "fails", "timeouts") | _SAMPLED +#: One accuracy over problems, with its population and extraction health. +_ACCURACY = report_metrics("score", "accuracy", "fails", "n_problems") | HEALTH_METRICS +#: A judged three-way grade: correct, incorrect, and what was graded. +_JUDGED = ( + report_metrics( + "score", "accuracy", "correct", "incorrect", "fails", "n_graded", "n_problems" + ) + | HEALTH_METRICS +) +#: ARC's shared report. +_ARC = report_metrics("score", "acc", "fails", "n_problems") +#: MMLU-style macro: the headline and one score per category. +_CATEGORY = report_metrics("score", "fails", patterns=("score_{category}",)) +#: C-Eval and CMMLU: the subject macro, and one rate per category that has a +#: subject in the run, named by the category. +_SUBJECT_MACRO = report_metrics("score", "overall", "fails", "n_subjects") +_CEVAL_CATEGORIES = ("stem", "social_science", "humanities", "other") +_CMMLU_CATEGORIES = (*_CEVAL_CATEGORIES, "china_specific") +#: IFEval-style strict and loose rates. +_INSTRUCTION = report_metrics( + "score", + "fails", + "n_problems", + patterns=( + "{grade}_prompt_level_accuracy", + "{grade}_instruction_level_accuracy", + ), +) +#: BFCL's two families share a report and differ in their categories. +_BFCL = report_metrics("score", "fails", "ast_summary", "n_problems", "n_ast") + +_P = "sieval.tasks" + +#: Each built-in class defining ``report``, by import path. +BUILTIN_REPORT_METRICS: MappingProxyType[str, ReportMetrics] = MappingProxyType( + { + f"{_P}._math_perturb_base.MathPerturbZeroShotGenTask": report_metrics( + "score", + "fails", + patterns=( + "score_seed_{split}", + "n_problems_seed_{split}", + "score_type_{type}", + ), + ) + | _SAMPLED, + f"{_P}._nl2sh_alfa_base.NL2SHAlfaSharedZeroShotGenTask": report_metrics( + "score", + "accuracy", + "fails", + "mean_reward", + "n_gold_timeouts", + "n_model_nonzero_exit", + "n_model_timeouts", + "n_p3_command_match", + "n_p3_output_match", + "n_p3_empty_output", + "n_p3_embed", + "n_problems", + ) + | HEALTH_METRICS, + f"{_P}._quotebench_base.QuoteBenchTask": report_metrics( + "score", + "pass_rate_pct", + "fails", + "n_problems", + "n_pass", + "n_environment_invalid", + "n_timeout", + "n_shell_syntax", + "n_tool_error", + "n_runtime_error", + "n_silent_wrong", + "n_unknown_class", + patterns=("pass_rate_pct_tier{tier}", "pass_rate_pct_{scenario}"), + ) + | HEALTH_METRICS, + f"{_P}.aa_lcr_0shot_gen.AALCRZeroShotGenTask": _JUDGED, + f"{_P}.advanced_if_0shot_gen.AdvancedIFZeroShotGenTask": report_metrics( + "score", + "overall_pass_rate", + "macro_pass_rate", + "micro_pass_rate", + "fails", + "n_graded", + "n_grader_unparsed", + "n_problems", + "n_rubric_checks", + patterns=( + "{benchmark}_pass_rate", + "{benchmark}_micro_pass_rate", + "{benchmark}_macro_pass_rate", + "{benchmark}_n_grader_unparsed", + "{benchmark}_n_graded", + ), + ) + | HEALTH_METRICS, + f"{_P}.ag_livecodebench_x_0shot_gen.AgLiveCodeBenchXZeroShotGenTask": ( + report_metrics("score", "fails", "n_run_errors") | _SAMPLED + ), + f"{_P}.agieval_0shot_gen.AGIEvalZeroShotGenTask": report_metrics( + "score", + "fails", + "n_subsets", + "macro_en_mcq", + "macro_zh_mcq", + "macro_math", + patterns=("score_{subset}",), + ), + f"{_P}.aime_2024_0shot_gen.AIME2024ZeroShotGenTask": _COMPETITION, + f"{_P}.aime_2025_0shot_gen.AIME2025ZeroShotGenTask": _COMPETITION, + f"{_P}.aime_2026_0shot_gen.AIME2026ZeroShotGenTask": _COMPETITION, + f"{_P}.apex_2025_0shot_gen.Apex2025ZeroShotGenTask": _COMPETITION, + f"{_P}.apex_shortlist_2025_0shot_gen.ApexShortlist2025ZeroShotGenTask": ( + _COMPETITION + ), + f"{_P}.arc.arc_challenge_kshot_clp.ARCChallengeFewShotClpTask": _ARC, + f"{_P}.arc.arc_challenge_kshot_ppl.ARCChallengeFewShotPplTask": _ARC, + f"{_P}.arc.arc_easy_kshot_clp.ARCEasyFewShotClpTask": _ARC, + f"{_P}.arc.arc_easy_kshot_ppl.ARCEasyFewShotPplTask": _ARC, + f"{_P}.bfcl_v3._base.BfclV3NonLiveTask": _BFCL + | report_metrics( + "non_live_overall_acc", + "simple_ast", + "simple", + "multiple", + "parallel", + "parallel_multiple", + "java", + "javascript", + "irrelevance", + "n_simple", + "n_multiple", + "n_parallel", + "n_parallel_multiple", + "n_java", + "n_javascript", + "n_irrelevance", + ), + f"{_P}.bfcl_v3._base.BfclV3LiveTask": _BFCL + | report_metrics( + "live_overall_acc", + "live_simple", + "live_multiple", + "live_parallel", + "live_parallel_multiple", + "live_irrelevance", + "live_relevance", + "n_live_simple", + "n_live_multiple", + "n_live_parallel", + "n_live_parallel_multiple", + "n_live_irrelevance", + "n_live_relevance", + ), + f"{_P}.browsecomp_0shot_gen.BrowseCompZeroShotGenTask": _JUDGED, + f"{_P}.brumo_2025_0shot_gen.BRUMO2025ZeroShotGenTask": _COMPETITION, + f"{_P}.c_eval_kshot_clp.CEvalFewShotCLPTask": _SUBJECT_MACRO + | _CEVAL_CATEGORIES, + f"{_P}.cmimc_2025_0shot_gen.CMIMC2025ZeroShotGenTask": _COMPETITION, + f"{_P}.cmmlu_kshot_clp.CMMLUFewShotClpTask": _SUBJECT_MACRO | _CMMLU_CATEGORIES, + f"{_P}.complex_constraints_0shot_gen.ComplexConstraintsZeroShotGenTask": ( + report_metrics( + "score", + "task_pass_rate", + "criterion_pass_rate_macro", + "criterion_pass_rate_micro", + "fails", + "n_criteria_graded", + "n_graded", + "n_grader_unparsed", + "n_problems", + ) + | HEALTH_METRICS + ), + f"{_P}.drop_kshot_gen.DROPFewShotGenTask": report_metrics( + "score", "em", "f1", "fails", "n_problems" + ), + f"{_P}.gpqa_diamond_0shot_gen.GPQADiamondZeroShotGenTask": report_metrics( + "score", "fails", "n_problems" + ) + | HEALTH_METRICS, + f"{_P}.gsm1k_0shot_gen.GSM1KZeroShotGenTask": _ACCURACY, + f"{_P}.gsm1k_kshot_base_gen.GSM1KFewShotBaseGenTask": report_metrics( + "score", "strict_exact_match", "flexible_exact_match", "fails", "n_problems" + ) + | HEALTH_METRICS, + f"{_P}.gsm8k_0shot_gen.GSM8KZeroShotGenTask": _ACCURACY | _VOTED, + f"{_P}.gsm8k_kshot_base_gen.GSM8KFewShotBaseGenTask": report_metrics( + "score", "exact_match", "flexible_exact_match", "fails" + ) + | _VOTED, + f"{_P}.gsm_plus_0shot_gen.GSMPlusZeroShotGenTask": _ACCURACY + | report_metrics( + "score_wo_critical_thinking", + "n_problems_wo_critical_thinking", + patterns=("score_{perturbation}",), + ), + f"{_P}.hellaswag_kshot_ppl.HellaSwagFewShotPPLTask": report_metrics( + "score", "acc", "acc_norm", "fails", "n_problems" + ), + f"{_P}.hendrycks_math_kshot_base_gen.HendrycksMathFewShotBaseGenTask": ( + _ACCURACY | _VOTED + ), + f"{_P}.hle_0shot_gen.HLEZeroShotGenTask": report_metrics( + "score", + "accuracy", + "calibration_error", + "confidence_interval", + "fails", + "n", + "n_graded", + "n_grader_unparsed", + "n_problems", + ) + | HEALTH_METRICS, + f"{_P}.hmmt_feb_2025_0shot_gen.HMMTFeb2025ZeroShotGenTask": _COMPETITION, + f"{_P}.hmmt_feb_2026_0shot_gen.HMMTFeb2026ZeroShotGenTask": _COMPETITION, + f"{_P}.hmmt_nov_2025_0shot_gen.HMMTNov2025ZeroShotGenTask": _COMPETITION, + f"{_P}.human_eval_0shot_base_gen.HumanEvalZeroShotBaseGenTask": _CODE, + f"{_P}.human_eval_0shot_gen.HumanEvalZeroShotGenTask": _CODE, + f"{_P}.ifbench_0shot_gen.IFBenchZeroShotGenTask": _INSTRUCTION, + f"{_P}.ifeval_0shot_gen.IFEvalZeroShotGenTask": _INSTRUCTION + | report_metrics(patterns=("{grade}_accuracy",)), + f"{_P}.iheval_0shot_gen.IHEvalZeroShotGenTask": report_metrics( + "score", + "fails", + "reference_cells_degraded", + patterns=( + "score_{setting}", + "score_{subtask}_{setting}", + "cell_{subtask}_{setting}_{variant}", + "diff_{setting}", + "abs_diff_{setting}", + ), + ), + f"{_P}.imo_answer_bench_0shot_gen.IMOAnswerBenchZeroShotGenTask": ( + _COMPETITION + ), + f"{_P}.inverse_ifeval_0shot_gen.InverseIFEvalZeroShotGenTask": report_metrics( + "score", + "fails", + "n_graded", + "n_grader_unparsed", + patterns=("score_{group}", "n_{group}"), + ) + | _SAMPLED, + f"{_P}.livecodebench_code_generation_0shot_gen." + "LiveCodeBenchCodeGenerationZeroShotGenTask": _CODE, + f"{_P}.livecodebench_code_generation_kshot_base_gen." + "LiveCodeBenchCodeGenerationFewShotBaseGenTask": _CODE, + f"{_P}.liveoibench_0shot_gen.LiveOIBenchZeroShotGenTask": report_metrics( + "score", + "relative_score", + "pass_rate", + "tests_passed_pct", + "human_percentile", + "medal_rate", + "gold_rate", + "fails", + "n", + "k", + "n_short", + "n_problems", + "n_scored", + "n_contests", + "n_contests_ranked", + "n_contests_medalled", + ) + | HEALTH_METRICS, + f"{_P}.math_500_0shot_gen.MATH500ZeroShotGenTask": _COMPETITION, + f"{_P}.mbpp_kshot_base_gen.MBPPFewShotBaseGenTask": _CODE, + f"{_P}.mmlu_0shot_gen.MMLUZeroShotGenTask": _CATEGORY + | report_metrics("n_problems") + | HEALTH_METRICS, + f"{_P}.mmlu_kshot_clp.MMLUFewShotCLPTask": _CATEGORY + | report_metrics("n_problems"), + f"{_P}.mmlu_pro_0shot_gen.MMLUProZeroShotGenTask": _CATEGORY + | report_metrics("n_problems") + | HEALTH_METRICS, + f"{_P}.mmmlu_kshot_clp.MMMLUKShotClpTask": report_metrics( + "score", + "score_mmmlu", + "fails", + "n_problems", + patterns=( + "score_locale_{locale}", + "score_locale_{locale}_category_{category}", + "score_locale_{locale}_subject_{subject}", + ), + ), + f"{_P}.multi_if_0shot_gen.MultiIFZeroShotGenTask": report_metrics( + "score", + "fails", + "all_turns_all_languages_overall", + patterns=( + "turn_{turn}_prompts_number", + "turn_{turn}_{grade}_prompt_level_accuracy", + "turn_{turn}_{grade}_instruction_level_accuracy", + "turn_{turn}_{language}_overall", + ), + ), + f"{_P}.multipl_e._base.MultiPLETask": report_metrics( + "score", + "fails", + "timeouts", + "n_build_errors", + "n_execution_errors", + "n_languages", + "pass@1_macro", + patterns=("pass@1_{language}", "n_problems_{language}"), + ) + | _SAMPLED, + f"{_P}.openbookqa_kshot_gen.OpenBookQAFewShotGenTask": _ACCURACY, + f"{_P}.platinum_bench._base.PlatinumMathGenTask": _ACCURACY + | report_metrics("errors") + | _VOTED, + f"{_P}.ruler_0shot_gen.RulerZeroShotGenTask": report_metrics( + "score", + "fails", + patterns=("score_{length}", "score_{subtask}_{length}"), + ), + f"{_P}.scicode_0shot_gen.SciCodeZeroShotGenTask": report_metrics( + "score", + "main_problem_accuracy", + "sub_problem_accuracy", + "correct_problems", + "correct_steps", + "total_problems", + "total_steps", + "unevaluated_steps", + "empty_extractions", + "import_errors", + "memory_errors", + "timeouts", + "fails", + "n_problems", + ), + f"{_P}.scitarc_0shot_gen.SciTaRCZeroShotGenTask": report_metrics( + "score", + "accuracy", + "exact_match", + "partial", + "fails", + "n", + "n_graded", + "n_grader_unparsed", + "n_problems", + ) + | HEALTH_METRICS, + f"{_P}.simpleqa_verified_0shot_gen.SimpleQAVerifiedZeroShotGenTask": ( + report_metrics( + "score", + "f1", + "accuracy_given_attempted", + "correct", + "incorrect", + "not_attempted", + "fails", + "n_graded", + "n_problems", + ) + | HEALTH_METRICS + ), + f"{_P}.smt_2025_0shot_gen.SMT2025ZeroShotGenTask": _COMPETITION, + f"{_P}.spider.spider_0shot_gen.SpiderZeroShotGenTask": report_metrics( + "score", + "execution_accuracy", + "exact_match", + "test_suite_accuracy", + "fails", + "n", + "n_execution_errors", + "n_parser_rejected", + "n_problems", + patterns=("test_suite_accuracy_{level}", "n_{level}"), + ) + | HEALTH_METRICS, + f"{_P}.spider2_lite_0shot_gen.Spider2LiteZeroShotGenTask": report_metrics( + "score", + "execution_accuracy", + "fails", + "n", + "n_execution_errors", + "n_missing_credentials", + patterns=( + "execution_accuracy_{backend}", + "n_{backend}", + "n_missing_credentials_{backend}", + ), + ) + | HEALTH_METRICS, + f"{_P}.sysbench_0shot_gen.SysBenchZeroShotGenTask": report_metrics( + "score", + "csr", + "isr", + "ssr", + "csr_macro", + "csr_graded", + "ungradeable_rate", + "fails", + "n_criteria_graded", + "n_criteria_unaligned", + "n_criteria_untyped", + "n_grader_unparsed", + "n_grader_unparsed_turns", + "n_sessions", + "n_turns", + "n_turns_unaligned", + patterns=( + "csr_{alignment}", + "isr_{alignment}", + "csr_{alignment}_n_criteria", + "isr_{alignment}_n_turns", + "csr_type_{type}", + "csr_type_{type}_n_criteria", + "turn_{turn}_csr", + "turn_{turn}_isr", + "turn_{turn}_n_turns", + "turn_{turn}_n_criteria", + "turn_{turn}_isr_cumulative", + "turn_{turn}_isr_cumulative_n_sessions", + ), + ) + | HEALTH_METRICS, + f"{_P}.t_eval_before_calling_0shot_gen.TEvalBeforeCallingZeroShotGenTask": ( + report_metrics( + "thought", + "name", + "args_precision", + "args_recall", + "args_f1_score", + "parse_rate", + "args_precision_parsed", + "args_recall_parsed", + "args_f1_score_parsed", + "fails", + "n_graded", + "n_parsed", + ) + ), + f"{_P}.theoremqa_kshot_base_gen.TheoremQAKShotBaseGenTask": _ACCURACY + | report_metrics("empty") + | _VOTED, + f"{_P}.ugmathbench_0shot_gen_fixed.UGMathBenchZeroShotGenFixedTask": ( + report_metrics( + "score", + "aacc", + "eacc", + "cacc", + "delta", + "relative_delta", + "fails", + "incomplete_problems", + "n_problems", + "n_versions", + "n_versions_judged", + "unattributed_fails", + "unattributed_finals", + patterns=("eacc_{subject}",), + ) + | _VOTED + | HEALTH_METRICS + ), + f"{_P}.wikisql_0shot_gen.WikiSQLZeroShotGenTask": report_metrics( + "score", + "ex_accuracy", + "lf_accuracy", + "fails", + "n", + "n_execution_errors", + ) + | HEALTH_METRICS, + } +) + + +def task_report_metrics(task_cls: type) -> ReportMetrics | None: + """*task_cls*'s report metrics: its own declaration, else this table's. + + ``None`` when neither describes the class that owns its ``report``. + """ + return declared_report_metrics(task_cls, BUILTIN_REPORT_METRICS) diff --git a/tests/integration/test_lazy_launch.py b/tests/integration/test_lazy_launch.py index 8f3b0bfa..20045458 100644 --- a/tests/integration/test_lazy_launch.py +++ b/tests/integration/test_lazy_launch.py @@ -27,6 +27,7 @@ from sieval.core.models import connection_factory from sieval.core.models.request_log import RecordingOpenAIClient from sieval.core.tasks import Task +from sieval.core.tasks.report_metrics import report_metrics from sieval.infer.backends.translator import BackendCommand from sieval.infer.config import InferHandle from sieval.infer.deployer import command_endpoint @@ -40,6 +41,8 @@ class EchoTask(Task): """Asks each question once; right when the reply names the question.""" model_type = "chat" + # Declared as a plugin task declares it; the dry-run reports it. + report_metrics = report_metrics("correct", "total") def __init__(self, dataset, model, name=None, strict=False): super().__init__(dataset=dataset, model=model, name=name) @@ -427,6 +430,7 @@ def test_a_resume_reports_its_plan_and_reads_dirs_read_only( "to_infer": 1, "to_grade": 1, "needs_inference": True, + "metrics": {"names": ["correct", "total"], "patterns": []}, } stages = data["tasks"]["echo"]["stages"] assert stages["identity_schema"] == 3 @@ -508,6 +512,8 @@ def test_a_refused_resume_fails_the_dry_run( assert code == 1 data = out["data"] assert "inference" in data["tasks"]["echo"]["error"] + # What a check needs is known before planning, so a refusal keeps it. + assert data["tasks"]["echo"]["metrics"]["names"] == ["correct", "total"] assert [c["name"] for c in data["checks"] if not c["ok"]] == ["resume_plan"] def test_a_fresh_run_infers_everything( @@ -532,6 +538,7 @@ def test_a_fresh_run_infers_everything( "to_infer": 3, "to_grade": 3, "needs_inference": True, + "metrics": {"names": ["correct", "total"], "patterns": []}, } assert not (tmp_path / "results").exists() @@ -600,6 +607,8 @@ def test_a_report_only_change_rereports_and_infers_nothing( assert echo["stages"]["rerun_from"] == "report" assert echo["stages"]["stages"]["report"] == "differs" assert echo["stages"]["stages"]["feedback"] == "same" + # The patched report writes `patched`, which the class never declared. + assert echo["metrics"] is None assert data["needs_inference"] is False # The run re-reports on a node with no GPU: no engine, no request. diff --git a/tests/unit/cli/test_output.py b/tests/unit/cli/test_output.py index 37bd0389..416bb2ed 100644 --- a/tests/unit/cli/test_output.py +++ b/tests/unit/cli/test_output.py @@ -434,6 +434,7 @@ def test_each_model_and_task_gets_a_line(self) -> None: "stages": {"rerun_from": "feedback"}, "to_infer": 1, "to_grade": 2, + "metrics": {"names": ["score"], "patterns": []}, }, "fresh": { "start": "fresh", @@ -450,7 +451,10 @@ def test_each_model_and_task_gets_a_line(self) -> None: "task echo: resume (2 kept, 1 regraded), from feedback, 1 to infer, " "2 to grade" ) in lines - assert "task fresh: fresh, 3 to infer, 3 to grade" in lines + # No declared report metrics: a check on them cannot run. + assert ( + "task fresh: fresh, 3 to infer, 3 to grade, report metrics unknown" + ) in lines assert "needs inference: yes" in lines def test_a_refused_task_is_named_on_a_failed_dry_run(self) -> None: @@ -464,6 +468,7 @@ def test_a_refused_task_is_named_on_a_failed_dry_run(self) -> None: "resume_plan": _outcomes(kept=3), "to_infer": 0, "to_grade": 0, + "metrics": {"names": ["score"], "patterns": []}, }, }, needs_inference=False, diff --git a/tests/unit/cli/test_run.py b/tests/unit/cli/test_run.py index dc30ea94..bd1785a8 100644 --- a/tests/unit/cli/test_run.py +++ b/tests/unit/cli/test_run.py @@ -35,6 +35,7 @@ WellKnownRole, deployment_plan_projection, ) +from sieval.tasks import MATH500ZeroShotGenTask runner = CliRunner() @@ -108,6 +109,19 @@ def _a_gpu_node(monkeypatch: pytest.MonkeyPatch) -> None: ) +class _UndeclaredTask(MATH500ZeroShotGenTask): + """A plugin task whose own report declares no metrics.""" + + async def report(self, finals, fails): + return {"mine": 1.0} + + +class _MalformedTask(MATH500ZeroShotGenTask): + """A plugin task whose declaration is not a ``ReportMetrics``.""" + + report_metrics = frozenset({"score"}) + + class TestRunCommand: def test_run_help(self): result = runner.invoke(app, ["run", "--help"]) @@ -320,12 +334,13 @@ async def _dry_run( n_errors: int = 0, resume: bool = True, plan_tasks: AsyncMock | None = None, + tasks: dict[str, object] | None = None, ) -> dict[str, object]: """``_run_dry_run`` with ``model_a`` served and ``api`` not.""" from sieval.cli.run import _run_dry_run config_path = tmp_path / "cfg.yaml" - config_path.write_text(yaml.safe_dump({"models": {}, "tasks": {}})) + config_path.write_text(yaml.safe_dump({"models": {}, "tasks": tasks or {}})) plan_dicts = {"model_a": {"backend": "vllm", "assignments": []}} checks = [{"name": "schema", "ok": not n_errors}] with ( @@ -384,6 +399,54 @@ async def test_needs_inference_is_whether_an_engine_starts( assert tasks["t_a"]["needs_inference"] is False assert tasks["t_a"]["stages"] == {"rerun_from": None} + @pytest.mark.anyio + async def test_each_task_reports_its_metric_names(self, tmp_path: Path) -> None: + tasks = { + "math": {"class": "MATH500ZeroShotGenTask"}, + "mmlu": {"class": "sieval.tasks.mmlu_kshot_clp.MMLUFewShotCLPTask"}, + "mine": {"class": "tests.unit.cli.test_run._UndeclaredTask"}, + # Advisory: a class this cannot resolve, or none, is unknown. + "gone": {"class": "no_such_plugin.Task"}, + "bare": {"dataset": "d"}, + } + plans = AsyncMock(return_value=({}, {name: self._plan(0) for name in tasks})) + result = await self._dry_run(tmp_path, plan_tasks=plans, tasks=tasks) + found = { + name: entry["metrics"] + for name, entry in cast(dict, result["tasks"]).items() + } + assert {"score", "pass@1", "n_truncated"} <= set(found["math"]["names"]) + assert found["math"]["patterns"] == [] + # A clp task has no rollouts for the runner to count. + assert "n_truncated" not in found["mmlu"]["names"] + assert found["mmlu"]["patterns"] == ["score_{category}"] + assert found["mine"] is None + assert (found["gone"], found["bare"]) == (None, None) + assert result["n_errors"] == 0 + assert "report_metrics" not in {c["name"] for c in cast(list, result["checks"])} + + @pytest.mark.anyio + async def test_a_malformed_declaration_fails_the_dry_run( + self, tmp_path: Path + ) -> None: + # Reading it as unknown would let a check pass that the author broke. + tasks = { + "bad": {"class": "tests.unit.cli.test_run._MalformedTask"}, + "math": {"class": "MATH500ZeroShotGenTask"}, + } + plans = AsyncMock(return_value=({}, {name: self._plan(0) for name in tasks})) + result = await self._dry_run(tmp_path, plan_tasks=plans, tasks=tasks) + entries = cast(dict, result["tasks"]) + assert entries["bad"]["metrics"] is None + assert entries["math"]["metrics"] is not None + (check,) = [ + c for c in cast(list, result["checks"]) if c["name"] == "report_metrics" + ] + assert check["ok"] is False + assert check["detail"].startswith("bad: ") + assert "ReportMetrics or None" in check["detail"] + assert result["n_errors"] == 1 + @pytest.mark.anyio async def test_a_fresh_run_starts_every_served_engine(self, tmp_path: Path) -> None: # `_models_to_infer` does not plan a run where nothing resumes. diff --git a/tests/unit/core/tasks/test_report_metrics.py b/tests/unit/core/tasks/test_report_metrics.py new file mode 100644 index 00000000..dbe95f5e --- /dev/null +++ b/tests/unit/core/tasks/test_report_metrics.py @@ -0,0 +1,168 @@ +"""``ReportMetrics`` and how a task class finds its declaration. + +AI-Generated Code - Claude Opus 5.5 (Anthropic) +""" + +from typing import ClassVar + +import pytest + +from sieval.core.tasks.report_metrics import ( + RUNNER_GEN_METRICS, + ReportMetrics, + class_path, + declared_report_metrics, + report_metrics, +) + + +class TestReportMetrics: + def test_a_name_matches_as_a_name(self) -> None: + declared = report_metrics("score", patterns=("score_{category}",)) + assert declared.match("score") == "name" + + def test_a_pattern_matches_a_group_from_the_data(self) -> None: + declared = report_metrics("score", patterns=("score_{category}",)) + assert declared.match("score_stem") == "pattern" + assert declared.match("score_social_sciences") == "pattern" + + def test_a_placeholder_is_not_empty(self) -> None: + assert report_metrics(patterns=("score_{category}",)).match("score_") is None + + def test_a_pattern_matches_the_whole_name(self) -> None: + declared = report_metrics(patterns=("score_{category}",)) + assert declared.match("xscore_stem") is None + assert declared.match("n_problems") is None + + def test_literal_parts_are_not_regex(self) -> None: + declared = report_metrics(patterns=("pass@1_{language}",)) + assert declared.match("pass@1_cpp") == "pattern" + assert declared.match("passX1_cpp") is None + + def test_two_placeholders(self) -> None: + declared = report_metrics(patterns=("turn_{turn}_{grade}_accuracy",)) + assert declared.match("turn_1_strict_accuracy") == "pattern" + + @pytest.mark.parametrize("pattern", ["{category}", "score", ""]) + def test_a_pattern_needs_a_literal_and_a_placeholder(self, pattern: str) -> None: + with pytest.raises(ValueError, match="literal part"): + report_metrics(patterns=(pattern,)) + + def test_an_empty_name_is_refused(self) -> None: + with pytest.raises(ValueError, match="non-empty"): + report_metrics("") + + def test_union_keeps_both_and_drops_repeated_patterns(self) -> None: + a = report_metrics("score", patterns=("score_{x}",)) + b = report_metrics("fails", patterns=("score_{x}", "n_{x}")) + assert (a | b) == ReportMetrics( + frozenset({"score", "fails"}), ("score_{x}", "n_{x}") + ) + assert (a | {"n"}).names == {"score", "n"} + + def test_union_with_a_str_is_refused(self) -> None: + # Iterating a str would add one name per character. + with pytest.raises(TypeError, match="wrap the name"): + report_metrics("fails") | "score" + + def test_json_sorts_names_and_keeps_pattern_order(self) -> None: + declared = report_metrics("score", "fails", patterns=("b_{x}", "a_{x}")) + assert declared.to_json() == { + "names": ["fails", "score"], + "patterns": ["b_{x}", "a_{x}"], + } + + +async def _report(_self, _finals, _fails): + return {} + + +class _Base: + tags: ClassVar[frozenset[str]] = frozenset({"gen"}) + report_metrics = report_metrics("score") + report = _report + + +class _Inherits(_Base): + pass + + +class _OwnReport(_Base): + async def report(self, finals, fails): + return {"other": 1.0} + + +class _OwnReportDeclared(_Base): + report_metrics = report_metrics("other") + + async def report(self, finals, fails): + return {"other": 1.0} + + +class _Unknown(_Base): + report_metrics = None + + +class _Clp: + tags: ClassVar[frozenset[str]] = frozenset({"clp"}) + + async def report(self, finals, fails): + return {} + + +class TestDeclaredReportMetrics: + def test_a_subclass_inherits_a_declaration_with_the_report(self) -> None: + declared = declared_report_metrics(_Inherits) + assert declared is not None + assert declared.names == {"score"} | RUNNER_GEN_METRICS + + def test_a_report_below_the_declaration_is_unknown(self) -> None: + # Its keys are its own; the declaration above never saw them. + assert declared_report_metrics(_OwnReport) is None + + def test_a_report_declared_beside_it_is_known(self) -> None: + declared = declared_report_metrics(_OwnReportDeclared) + assert declared is not None and "other" in declared.names + + def test_none_declares_unknown(self) -> None: + assert declared_report_metrics(_Unknown) is None + + def test_a_table_entry_declares_the_class_it_names(self) -> None: + table = {class_path(_Clp): report_metrics("acc")} + declared = declared_report_metrics(_Clp, table) + # No rollouts, so the runner adds nothing. + assert declared == report_metrics("acc") + + def test_the_class_attribute_wins_over_the_table(self) -> None: + table = {class_path(_Base): report_metrics("from_table")} + declared = declared_report_metrics(_Base, table) + assert declared is not None and "score" in declared.names + + def test_no_declaration_is_unknown(self) -> None: + assert declared_report_metrics(_Clp) is None + + def test_a_patched_report_is_unknown(self, monkeypatch: pytest.MonkeyPatch) -> None: + # A report defined in another module than the class: `setattr` patching. + import json + + monkeypatch.setattr(_Base, "report", json.dumps) + assert declared_report_metrics(_Base) is None + + def test_a_declaration_that_is_not_report_metrics_is_refused(self) -> None: + class Wrong(_Base): + report_metrics = frozenset({"score"}) + + with pytest.raises(TypeError, match="ReportMetrics or None"): + declared_report_metrics(Wrong) + + def test_a_malformed_declaration_is_refused_below_a_nearer_report(self) -> None: + # The report shadows it, but the author still wrote a broken declaration. + class Wrong(_Base): + report_metrics = frozenset({"score"}) + + class Below(Wrong): + async def report(self, finals, fails): + return {} + + with pytest.raises(TypeError, match="ReportMetrics or None"): + declared_report_metrics(Below) diff --git a/tests/unit/meta/test_report_metrics.py b/tests/unit/meta/test_report_metrics.py new file mode 100644 index 00000000..5daec3e0 --- /dev/null +++ b/tests/unit/meta/test_report_metrics.py @@ -0,0 +1,91 @@ +"""The built-in report metric table, against the registry and the identities. + +Each task test's report is checked against its entry as it runs +(``tests/unit/tasks/conftest.py``); this file holds what no single report shows: +every built-in task is declared, no entry is dead, and declaring moves no +stage identity. + +AI-Generated Code - Claude Opus 5.5 (Anthropic) +""" + +import importlib +import pkgutil + +import pytest + +import sieval.tasks +from sieval.core.tasks.meta import _TASK_CLASSES +from sieval.core.tasks.report_metrics import RUNNER_GEN_METRICS, class_path +from sieval.core.tasks.run_meta import GRADER_PACKAGES +from sieval.core.tasks.task import Task +from sieval.meta.report_metrics import BUILTIN_REPORT_METRICS, task_report_metrics + +type Registry = dict[str, type[Task]] + + +@pytest.fixture(scope="module") +def registry() -> Registry: + for module in pkgutil.walk_packages(sieval.tasks.__path__, "sieval.tasks."): + importlib.import_module(module.name) + return dict(_TASK_CLASSES) + + +def test_every_registered_task_is_declared(registry: Registry) -> None: + unknown = sorted( + name for name, cls in registry.items() if task_report_metrics(cls) is None + ) + assert unknown == [] + + +def test_every_entry_names_a_class_that_owns_a_report( + registry: dict[str, type], +) -> None: + used = { + class_path(klass) + for cls in registry.values() + for klass in cls.__mro__ + if class_path(klass) in BUILTIN_REPORT_METRICS + } + assert sorted(set(BUILTIN_REPORT_METRICS) - used) == [] + + +def test_every_task_declares_its_headline(registry: Registry) -> None: + """``score`` is what a smoke criterion names first; only T-Eval has none.""" + headless = sorted( + name + for name, cls in registry.items() + if (declared := task_report_metrics(cls)) is not None + and "score" not in declared.names + ) + assert headless == ["t_eval_before_calling_0shot_gen"] + + +def test_a_generative_task_gains_the_runner_keys( + registry: Registry, +) -> None: + for name, cls in registry.items(): + declared = task_report_metrics(cls) + assert declared is not None + assert (declared.names >= RUNNER_GEN_METRICS) == ("gen" in cls.tags), name + + +def test_the_table_is_outside_every_stage_identity( + registry: Registry, +) -> None: + """A task module's source is in its stage identities; this module is not. + + Declaring on each class would rerun every resumed directory from + ``postprocess``. The grader walk follows only :data:`GRADER_PACKAGES`. + """ + assert not any( + root == "sieval.meta" or "sieval.meta".startswith(root + ".") + for root in GRADER_PACKAGES + ) + # And no built-in declares on its class, which the walk would digest. + declaring = sorted( + class_path(klass) + for cls in registry.values() + for klass in cls.__mro__ + if "report_metrics" in vars(klass) + ) + assert declaring == [] diff --git a/tests/unit/tasks/conftest.py b/tests/unit/tasks/conftest.py index 35922ea8..91ef1b08 100644 --- a/tests/unit/tasks/conftest.py +++ b/tests/unit/tasks/conftest.py @@ -38,12 +38,17 @@ and the task tests. Deleting the variable alone is not enough: google-auth then probes the GCE metadata server, which on a host off GCP costs ~12s of timeouts. +**Report metric names.** Every ``report()`` a test here runs must write only +numeric keys its declaration (``sieval/meta/report_metrics.py``) names or matches. + AI-Generated Code - Claude Fable 5 (Anthropic) """ import contextlib +import functools import importlib import os +import pkgutil import tempfile from pathlib import Path @@ -131,3 +136,57 @@ def _nothing_configured(*_args, **_kwargs): raise RuntimeError("no application default credentials") monkeypatch.setattr(google_auth, "default", _nothing_configured) + + +@functools.cache +def _owners() -> tuple[type, ...]: + """Each built-in class that defines ``report``, from every task module.""" + import sieval.tasks + from sieval.core.tasks.task import Task + + for module in pkgutil.walk_packages(sieval.tasks.__path__, "sieval.tasks."): + with contextlib.suppress(ImportError): + importlib.import_module(module.name) + seen: list[type] = [] + pending = list(Task.__subclasses__()) + while pending: + cls = pending.pop() + pending += cls.__subclasses__() + if "report" in vars(cls) and cls.__module__.startswith("sieval.tasks."): + seen.append(cls) + return tuple(seen) + + +def _numeric(value: object) -> bool: + return isinstance(value, int | float) and not isinstance(value, bool) + + +@pytest.fixture(autouse=True) +def _report_writes_only_declared_metrics(monkeypatch: pytest.MonkeyPatch) -> None: + """Fail a test whose task report writes a metric its declaration lacks.""" + from sieval.core.tasks.task import Task + from sieval.meta.report_metrics import task_report_metrics + + for owner in _owners(): + + def checked(report, owner=owner): + @functools.wraps(report) + async def wrapper(self, finals, fails): + out = await report(self, finals, fails) + cls = type(self) if isinstance(self, Task) else owner + declared = task_report_metrics(cls) + if isinstance(out, dict) and declared is not None: + undeclared = sorted( + key + for key, value in out.items() + if _numeric(value) and declared.match(key) is None + ) + assert not undeclared, ( + f"{cls.__qualname__}.report() wrote {undeclared}, which " + "sieval/meta/report_metrics.py does not declare" + ) + return out + + return wrapper + + monkeypatch.setattr(owner, "report", checked(vars(owner)["report"]))