[Feature]: report each task's metric names in the dry-run, before the run #223

🚀 The feature, motivation and pitch

A platform that runs SiEval checks the result against criteria an author wrote, such as a smoke pass of score: {min: 0}, or a leaderboard column that reads pass@1. If the name is not one the task reports, because of a typo or because a variant reports under another name, the key is just missing from report.json. The platform can only say the verdict is unknown, after the run, and that is easy to miss.

Which keys a report can write is fixed by the task's code, so it can be known before the run. The proposal is to report it per task in the dry-run JSON:

"tasks": {
  "math_500.math1k-n4": {
    "metrics": {"names": ["avg@n", "fails", "k", "n", "pass@1", "score"], "patterns": []}
  },
  "mmlu": {
    "metrics": {"names": ["fails", "n_problems", "score"], "patterns": ["score_{category}"]}
  }
}

• names are the numeric keys the report may write.
• patterns cover keys built from the data (score_{category}, turn_{turn}_csr). A group that the run's data lacks is not written, so a name matching a pattern can still be absent. A checker should treat a pattern match as plausible, not certain.
• null means unknown. A checker should not refuse in that case.

Declared, not derived. Running report() on synthetic final records at dry-run time would need a stub of each task's feedback_result shape, and many keys only appear when a group or budget is present (n > 1, a subset, a language). A stub would also miss exactly the data-dependent keys. A declaration is cheap to read. To keep it honest, every task unit test checks the keys its report writes against the declaration.

Where the declarations live. A task module's source is part of its stage identities, so a class attribute on each built-in task would change every stage identity and rerun every resumed directory from postprocess. The built-in declarations therefore live in a table outside the grader walk, keyed by the class that defines report. A plugin task declares on its own class. A subclass that defines its own report, or a report patched in from another module, does not inherit its parent's declaration and stays unknown until it declares one.

Alternatives

• Derive the names by running report() on stub records. This was rejected for the reasons above, but the per-test check gives most of what it would.
• A class attribute on each built-in task. This was rejected because it moves every stage identity.

Additional context

This builds on the dry-run JSON from #217 and #220.