feat(cli): sieval run --dry-run reports what the run would do

CHANGELOG.md

@@ -25,6 +25,7 @@ Planned as 0.12.0.
 - **`SIEVAL_TARGET_GPU` declares the GPUs a serving plan is resolved for**, as `<name>:<count>:<memory MiB>` with the name and memory `nvidia-smi` reports, e.g. `NVIDIA H100 80GB HBM3:8:81559`. Recipe resolution reads it instead of `nvidia-smi`, on any node, so a CPU node resolves the plan, and the inference identity, a node with those GPUs would. Unset, nothing changes. Declared, an engine launches only on a node with exactly those GPUs. `infer_plans.yaml` gains `gpu`, the GPUs its plans were resolved for, outside every identity; a `sieval run --resume` declaring the same GPUs reuses the recorded plan of each served model whose `path`, `infer`, capability layer and `deterministic` are unchanged, unless the plan held an engine secret, which `infer_plans.yaml` records redacted (#215, closes #212).
 - `sieval.core.runners.task_resume_plan` / `plan_resume`: the per-stage resume plan, without running anything.
 - `TaskRunner.plan_run` / `MultiTaskRunner.plan_run` (`RunPlan`): the runner's per-sample plan of a run, `to_infer` included, read-only; the one `sieval run` launches by.
+- **`sieval run --dry-run` reports what the run would do** (closes #214). `-o json` adds `gpu`, `models` (each served model's plan as `infer_plans.yaml` would record it, each process its launch starts, and `needs_inference`, whether this run starts it), `tasks` (each task's `start`, `resume_plan` with the counts `runs[].resume` will record, `null` for a fresh start, `new`, `to_infer` and `to_grade`) and `needs_inference`, whether any engine starts. The plans come from the call `sieval run` launches by. A resume that would be refused, or planning that raises, fails the new `resume_plan` check; when the dry-run stops before planning, `tasks` and `needs_inference` are `null`. It launches no engine, writes no result directory, reads `--resume` directories in place, and runs on a node without GPUs given `SIEVAL_TARGET_GPU`. Planning loads the datasets and starts each vendored service a task declares, as the run does, so a dry-run that passes its checks costs a session setup. The existing checks are unchanged.
 
 ## [0.11.0] - 2026-10-23
 

docs/guide/configuration.md

@@ -321,6 +321,67 @@ A directory recorded before stage identities is compared through the session's
 `effective_config.yaml`. Without one, `--resume DIR` is refused and an in-place
 `--resume` trusts it.
 
+### What a run would do
+
+`sieval run --dry-run [--resume [DIR]] -o json` reports what the run would do,
+without launching an engine or writing a result directory. It reads the
+`--resume` directories in place. On a node without GPUs, declare the target ones
+in `SIEVAL_TARGET_GPU` ([Resolving for declared GPUs](infer.md#resolving-for-declared-gpus)).
+
+```json
+{
+  "checks": [{"name": "resume_plan", "ok": true, "detail": "1 task(s): 1 sample(s) to infer, 1 to grade"}],
+  "n_errors": 0,
+  "n_warnings": 0,
+  "gpu": "NVIDIA H100 80GB HBM3:8:81559",
+  "models": {
+    "subject": {
+      "plan": {"checkpoint": "/models/Qwen3-4B", "backend": "sglang", "assignments": []},
+      "launch": [{"role": "full", "argv": ["sglang", "serve", "--model-path", "/models/Qwen3-4B"]}],
+      "needs_inference": true
+    }
+  },
+  "tasks": {
+    "echo": {
+      "model": "subject",
+      "start": "resume",
+      "recorded": "outputs/run/echo",
+      "samples": 3,
+      "resume_plan": {"kept": 2, "rereported": 0, "regraded": 0, "repostprocessed": 0, "restarted": 0, "dropped": 0},
+      "new": 1,
+      "to_infer": 1,
+      "to_grade": 1
+    }
+  },
+  "needs_inference": true
+}
+```
+
+- `checks` are the config checks, then `resume_plan`, which fails when a task's
+  resume would be refused; that task's entry then holds `error` instead.
+- `models` has one entry per model `sieval run` serves: the plan as
+  `infer_plans.yaml` would record it, each process the launch starts (`argv`,
+  and `env` and `cwd` when set), and `needs_inference`, whether this run starts
+  its engine. `gpu` is what the plans were resolved for.
+- `tasks.<task>.start` is `fresh`, `resume` (stage by stage) or `reuse` (the
+  recorded `report.json` stands, recomputed when only `report` changed).
+  `resume_plan` counts the recorded samples as `runs[].resume` will, and is
+  `null` for a fresh start; `new` counts samples with no record. `to_infer`
+  counts the samples sent to the model under test, `to_grade` those run through
+  a grading stage.
+- `needs_inference` is whether the run starts any engine. A task that infers
+  through an `api_base` model counts in its `to_infer` but starts none.
+
+`gpu`, `models`, `tasks` and `needs_inference` are always present. When the
+config checks fail, or planning raises (the `resume_plan` check then names
+why), `tasks` and every `needs_inference` are `null`: unknown, not false.
+
+Planning sets the session up as the run does. It loads every dataset,
+downloading one not yet cached, and starts each vendored service a task
+declares; neither writes under the result directory. A sample that `feedback`
+sends to another iteration after a rerun from `feedback` is not counted, since
+the plan cannot know it.
+
 Deprecated, removed next release: `--resume-from DIR`, `runner_config.auto_resume`
 and `runner_config.resume_from` (and the same-named Python fields and
 arguments) map to `resume` and warn once; one that disagrees with `resume` is

sieval/cli/output.py

@@ -348,6 +348,33 @@ def _render_text_task_reports(result: CommandResult) -> None:
         log_user("{}", task_data.get("report", ""))
 
 
+def _render_dry_run_plan(data: dict) -> None:
+    """What ``sieval run --dry-run`` found the run would do, when it got that far."""
+    tasks = data.get("tasks")
+    if not isinstance(tasks, dict) or not tasks:
+        return
+    log_user("")
+    for name, entry in (data.get("models") or {}).items():
+        starts = "starts" if entry.get("needs_inference") else "not started"
+        log_user("model {}: engine {}", name, starts)
+    for name, entry in tasks.items():
+        if "error" in entry:
+            log_user("task {}: refused", name)
+            continue
+        outcomes = ", ".join(
+            f"{n} {outcome}" for outcome, n in (entry["resume_plan"] or {}).items() if n
+        )
+        log_user(
+            "task {}: {}{}, {} to infer, {} to grade",
+            name,
+            entry["start"],
+            f" ({outcomes})" if outcomes else "",
+            entry["to_infer"],
+            entry["to_grade"],
+        )
+    log_user("needs inference: {}", "yes" if data.get("needs_inference") else "no")
+
+
 def _render_text_dry_run(result: CommandResult) -> None:
     if not result.ok:
         if not isinstance(result.data, dict):
@@ -363,6 +390,7 @@ def _render_text_dry_run(result: CommandResult) -> None:
                 log_user("✗ {}", check.get("detail", check["name"]))
             for w in check.get("warnings", []):
                 log_user("⚠ {}", w)
+        _render_dry_run_plan(result.data)
         n_err = result.data.get("n_errors", 0)
         n_warn = result.data.get("n_warnings", 0)
         log_user("\nDry-run failed: {} error(s), {} warning(s).", n_err, n_warn)
@@ -382,6 +410,7 @@ def _render_text_dry_run(result: CommandResult) -> None:
             log_user("✗ {}", check.get("detail", check["name"]))
         for w in check.get("warnings", []):
             log_user("⚠ {}", w)
+    _render_dry_run_plan(result.data)
     n_warn = result.data.get("n_warnings", 0)
     if n_warn:
         log_user("\nDry-run passed with {} warning(s).", n_warn)

sieval/cli/run.py

@@ -8,7 +8,7 @@
 import shlex
 import sys
 import time
-from collections.abc import Mapping
+from collections.abc import Iterable, Mapping
 from pathlib import Path
 from typing import Annotated, Any, cast
 
@@ -341,32 +341,79 @@ async def _plan_tasks(
     )
 
 
+def _models_to_launch(
+    served: Iterable[str],
+    task_models: Mapping[str, str],
+    plans: Mapping[str, RunPlan | Exception],
+) -> set[str]:
+    """The *served* models some task still has samples to infer with.
+
+    A task whose plan raises infers nothing: its run raises the same before
+    its first sample.
+    """
+    return {
+        task_models[task]
+        for task, plan in plans.items()
+        if not isinstance(plan, Exception)
+        and plan["to_infer"]
+        and task_models.get(task) in served
+    }
+
+
 async def _models_to_infer(
     config_path: Path,
     config: Mapping[str, Any],
     prepared_launches: Mapping[str, tuple[DeploymentPlan, list[BackendCommand]]],
     plan_dicts: dict[str, dict[str, Any]],
     **session: Any,
 ) -> set[str]:
-    """The served models some task still has samples to infer with.
-
-    A run where nothing resumes infers everything, so it is not planned. A
-    task whose plan raises infers nothing: its run raises the same before its
-    first sample.
-    """
+    """The served models to launch: all of them for a run where nothing
+    resumes, which is not planned; else :func:`_models_to_launch`."""
     if not prepared_launches:
         return set()
     if not _resumes(session["resume"], config):
         return set(prepared_launches)
     task_models, plans = await _plan_tasks(
         config_path, prepared_launches, plan_dicts, **session
     )
+    return _models_to_launch(prepared_launches, task_models, plans)
+
+
+def _launch_json(command: BackendCommand) -> dict[str, object]:
+    """One process a launch starts: its role, argv and the env it adds."""
+    out: dict[str, object] = {"role": command.role, "argv": list(command.cli_args)}
+    if command.env:
+        out["env"] = dict(command.env)
+    if command.working_dir:
+        out["cwd"] = command.working_dir
+    return out
+
+
+def _plan_summary(plans: Mapping[str, RunPlan | Exception]) -> str:
+    """One line for the ``resume_plan`` check: what the tasks would do."""
+    ok = [p for p in plans.values() if not isinstance(p, Exception)]
+    to_infer = sum(p["to_infer"] for p in ok)
+    to_grade = sum(p["to_grade"] for p in ok)
+    return f"{len(ok)} task(s): {to_infer} sample(s) to infer, {to_grade} to grade"
+
+
+def _task_json(model: str | None, plan: RunPlan | Exception) -> dict[str, object]:
+    """A task's entry in the dry-run report.
+
+    ``resume_plan`` is the plan's ``outcomes``, the counts ``runs[].resume``
+    records; ``None`` for a fresh start, which has no recorded samples.
+    """
+    if isinstance(plan, Exception):
+        return {"model": model, "error": cli_error_message(plan)}
     return {
-        task_models[task]
-        for task, plan in plans.items()
-        if not isinstance(plan, Exception)
-        and plan["to_infer"]
-        and task_models.get(task) in prepared_launches
+        "model": model,
+        "start": plan["start"],
+        "recorded": plan["recorded"],
+        "samples": plan["samples"],
+        "resume_plan": plan["outcomes"],
+        "new": plan["new"],
+        "to_infer": plan["to_infer"],
+        "to_grade": plan["to_grade"],
     }
 
 
@@ -379,7 +426,13 @@ async def _run_dry_run(
     deterministic: bool | None = None,
     cli_model: CliModel | None = None,
 ) -> dict[str, object]:
-    """Resolve the same managed batch and return the pure prelaunch plan."""
+    """Report what the run would do, launching no engine.
+
+    The checks, then each served model's plan and launch argv, and each task's
+    plan; see ``docs/guide/configuration.md``. Reads ``--resume`` directories in
+    place and writes no result directory. Planning loads the datasets and
+    starts each vendored service a task declares, as the run does.
+    """
 
     from sieval.cli.validation import run_dry_run
 
@@ -404,8 +457,14 @@ def checks(
             )
         )
 
+    def unplanned(result: dict[str, object]) -> dict[str, object]:
+        # Every report key is present; ``None`` where the dry-run stopped first.
+        for key in ("gpu", "models", "tasks", "needs_inference"):
+            result.setdefault(key, None)
+        return result
+
     if not config_path.exists():
-        return checks()
+        return unplanned(checks())
 
     with open(config_path) as stream:
         loaded = yaml.safe_load(stream)
@@ -414,15 +473,84 @@ def checks(
     try:
         apply_cli_model(loaded, cli_model, model)
     except ValueError:
-        return checks()  # reported as the failed cli_model check
+        return unplanned(checks())  # reported as the failed cli_model check
     effective_deterministic = resolve_deterministic(deterministic, loaded)
     plan_dicts, prepared_launches = await _prepare_launch_batch(
         config_path,
         loaded,
         effective_deterministic=effective_deterministic,
         recorded=_recorded_plans(resume, result_dir or loaded.get("result_dir")),
     )
-    return checks(plan_dicts or None, frozenset(prepared_launches))
+    result = checks(plan_dicts or None, frozenset(prepared_launches))
+    # ``needs_inference`` stays ``None`` until the tasks are planned.
+    models: dict[str, dict[str, object]] = {
+        name: {
+            "plan": plan_dicts[name],
+            "launch": [_launch_json(command) for command in commands],
+            "needs_inference": None,
+        }
+        for name, (_, commands) in prepared_launches.items()
+    }
+    result["gpu"] = await _plan_gpu_text() if prepared_launches else None
+    result["models"] = models
+    if result["n_errors"]:
+        return unplanned(result)
+
+    # What the run would do: each task's plan, from the call the run makes.
+    # Planning sets the session up as the run does: it loads every dataset
+    # (downloading one not yet cached) and starts each vendored service a task
+    # declares, writing nothing under the result directory.
+    checks_list = cast(list[dict[str, object]], result["checks"])
+    try:
+        task_models, plans = await _plan_tasks(
+            config_path,
+            prepared_launches,
+            plan_dicts,
+            model=model,
+            resume=resume,
+            result_dir=result_dir,
+            deterministic=deterministic,
+            invocation=shlex.join(sys.argv),
+            cli_model=cli_model,
+        )
+    except Exception as exc:
+        # Raised before any task would start: the run fails the same way.
+        detail = cli_error_message(exc)
+        checks_list.append({"name": "resume_plan", "ok": False, "detail": detail})
+        result["n_errors"] = cast(int, result["n_errors"]) + 1
+        return unplanned(result)
+    refused = {t: p for t, p in plans.items() if isinstance(p, Exception)}
+    result["tasks"] = {
+        task: _task_json(task_models.get(task), plan) for task, plan in plans.items()
+    }
+    if refused:
+        checks_list.append(
+            {
+                "name": "resume_plan",
+                "ok": False,
+                "detail": "; ".join(
+                    f"{task}: {cli_error_message(exc)}"
+                    for task, exc in sorted(refused.items())
+                ),
+            }
+        )
+        result["n_errors"] = cast(int, result["n_errors"]) + 1
+    else:
+        checks_list.append(
+            {"name": "resume_plan", "ok": True, "detail": _plan_summary(plans)}
+        )
+    # The launch decision `_models_to_infer` makes for the run.
+    launches = (
+        _models_to_launch(prepared_launches, task_models, plans)
+        if _resumes(resume, loaded)
+        else set(prepared_launches)
+    )
+    for name, entry in models.items():
+        entry["needs_inference"] = name in launches
+    # Whether an engine starts. A task inferring through an `api_base` model
+    # needs no engine, so it does not count.
+    result["needs_inference"] = bool(launches)
+    return result
 
 
 async def _run_all(
@@ -730,7 +858,14 @@ def run(
         ] = False,
         dry_run: Annotated[
             bool,
-            typer.Option("--dry-run", help="Validate config without running"),
+            typer.Option(
+                "--dry-run",
+                help=(
+                    "Validate the config, then report each model's plan and each "
+                    "task's resume plan, launching no engine. Loads the datasets "
+                    "and starts any vendored service, as the run does."
+                ),
+            ),
         ] = False,
         output: Annotated[
             OutputFormat,