[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Rebased onto main after #215, #216 and #211 landed; one commit.

• sieval run --dry-run [--resume [DIR]] -o json reports what the run would do. The existing checks are unchanged. A resume_plan check is added, and so are these keys:

{
  "gpu": "NVIDIA H100 80GB HBM3:8:81559",
  "models": {
    "subject": {
      "plan": {"checkpoint": "…", "backend": "sglang", "assignments": ["…"], "deterministic": false, "seed": 0},
      "launch": [{"role": "full", "argv": ["sglang", "serve", "--model-path", "…", "--port", "30000", "…"]}],
      "needs_inference": true
    }
  },
  "tasks": {
    "echo": {
      "model": "subject",
      "start": "resume",
      "recorded": "…/results/echo",
      "samples": 3,
      "resume_plan": {"kept": 2, "rereported": 0, "regraded": 0, "repostprocessed": 0, "restarted": 0, "dropped": 0},
      "new": 1,
      "to_infer": 1,
      "to_grade": 1
    }
  },
  "needs_inference": true
}

• models.<m>.plan is the plan as infer_plans.yaml records it. launch lists each process the launch starts (argv, plus env and cwd when set). needs_inference says whether this run starts that engine.
• tasks.<t>.start is fresh, resume or reuse. resume_plan is the RunPlan.outcomes from feat(core)!: split the grader identity along postprocess, feedback and report #211, the counts runs[].resume will record. It is null for a fresh start. new counts samples with no record.
• The top-level needs_inference says whether any engine starts. A task that infers through an api_base model counts in its to_infer but starts no engine.
• gpu, models, tasks and needs_inference are always present. When the checks fail or planning raises, tasks and every needs_inference are null, meaning unknown rather than false.
• A task whose resume would be refused carries error. That task, or a planning error, fails resume_plan, so the dry-run exits 1.
• The same call as the run. The task plans come from _plan_tasks, and the per-model flags come from _models_to_launch. These are the calls sieval run launches by. A test checks that the plan the dry-run reports is the one the run then records.
• Cost, now stated. No engine starts and no result directory is written. --resume directories are read in place. Planning sets the session up as the run does: it loads every dataset, downloading any that are not yet cached, and starts each vendored service a task declares. The --dry-run help text, the guide and the CHANGELOG now say this.
• Text output gains a short summary: one line per model (whether its engine starts) and per task, then needs inference: yes|no.

Identity impact. None. The dry-run computes and writes nothing a run records.

Related Issues

Closes #214. Builds on #215, #216 and #211.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit tests pass (pdm run pytest)

pytest tests/unit tests/integration -n 8: 9557 passed, 1 skipped (Linux). ty check (repo-wide), scripts/sync_package_stubs.py --check, scripts/check_preflight.py and the pre-commit hooks are clean.

tests/integration/test_lazy_launch.py::TestDryRunReportsWhatARunWouldDo runs the real CLI with SIEVAL_TARGET_GPU set and detection reporting no GPU:

• A resume with one new sample: the full shape above. nvidia-smi is never asked, no engine starts, and the result directory is byte-identical afterwards. The run that follows records the reported plan.
• A task-argument change: needs_inference: false, resume_plan = 2 repostprocessed.
• A finished run: start: reuse, resume_plan = 3 kept.
• --resume DIR from another session: reads DIR in place, copies nothing, creates no result directory.
• A refused resume: exit 1, the task carries error, and only resume_plan fails.
• A fresh run: the whole task entry, including resume_plan: null, with every sample to infer and nothing written.

Unit (tests/unit/cli/test_run.py, test_output.py):

• needs_inference is false when only an api_base task has samples to infer. A fresh run starts every served engine.
• Failed checks, a planning error and a missing config each leave every key present, with null where unknown.
• The text summary for resume, fresh, reuse and refused tasks.
• Reverting either needs_inference fix, or the null guard in the renderer, fails these tests.

Manual

• On a Mac without nvidia-smi, with SIEVAL_TARGET_GPU='NVIDIA H100 80GB HBM3:8:81559', sieval run eval.yaml --dry-run --resume over a finished stub run printed model subject: engine not started, task echo: reuse (3 kept), 0 to infer, 0 to grade and needs inference: no, and passed. This was run before the rebase.

🤖 Generated with Claude Code