feat(cli): sieval run starts an engine only for samples left to infer

CHANGELOG.md

@@ -18,11 +18,13 @@ Planned as 0.12.0.
 
 - **A resume reruns from the earliest stage that changed** (#210). An extraction change reruns from `postprocess`, a scoring change (judge, prompt, sampling, service, `feedback` code) from `feedback`, and a `report`-only change recomputes `report.json` from the records with no engine or judge call. Before, each of these regraded every sample. Code is attributed to stages by a static walk; a run-time lookup (`getattr(self, <expr>)`, `eval`, `importlib`) falls back to all grading code. No built-in task falls back.
 - `runs[].resume` counts what a resume did with each recorded sample.
+- **`sieval run` starts an engine only for a model with samples to infer** (closes #213). A run that may resume (`--resume`, or `resume` in a `runner_config`) first plans each task's resume, read-only, through the runner's own resume planning (`TaskRunner.plan_run`), and launches only the engines whose models some task still has samples to infer with. A resume that reruns only grading stages, or re-reports, starts no engine and runs on a node without GPUs; when some sample needs inference and `nvidia-smi` finds no GPU, the run is refused before anything starts. The plans, `infer_plans.yaml`, the inference identity and its request digest are what an eager launch records. A fresh run is not planned and launches as before. A sample that `feedback` sends to another iteration after a rerun from `feedback` fails at `infer` when its engine was not started, and a resume that starts it retries it.
 
 ### Added
 
 - **`SIEVAL_TARGET_GPU` declares the GPUs a serving plan is resolved for**, as `<name>:<count>:<memory MiB>` with the name and memory `nvidia-smi` reports, e.g. `NVIDIA H100 80GB HBM3:8:81559`. Recipe resolution reads it instead of `nvidia-smi`, on any node, so a CPU node resolves the plan, and the inference identity, a node with those GPUs would. Unset, nothing changes. Declared, an engine launches only on a node with exactly those GPUs. `infer_plans.yaml` gains `gpu`, the GPUs its plans were resolved for, outside every identity; a `sieval run --resume` declaring the same GPUs reuses the recorded plan of each served model whose `path`, `infer`, capability layer and `deterministic` are unchanged, unless the plan held an engine secret, which `infer_plans.yaml` records redacted (#215, closes #212).
 - `sieval.core.runners.task_resume_plan` / `plan_resume`: the per-stage resume plan, without running anything.
+- `TaskRunner.plan_run` / `MultiTaskRunner.plan_run` (`RunPlan`): the runner's per-sample plan of a run, `to_infer` included, read-only; the one `sieval run` launches by.
 
 ## [0.11.0] - 2026-10-23
 

docs/guide/infer.md

@@ -155,6 +155,23 @@ Models with a `path` field (and no `api_base`) or an `infer` section in the YAML
 
 A top-level `api_base` turns auto-serve off. An `args.api_base` next to `path` or `infer.checkpoint` is refused before any engine starts: keep one or the other.
 
+A run that may resume (`--resume`, or `resume` in a `runner_config`) plans each
+task's resume before it starts any engine, with the same per-sample planning
+the run uses, and launches one only for a model some task still has samples to
+infer with. A resume that keeps every sample, or reruns only from
+`postprocess`, `feedback` or `report`, starts none, so it runs on a node
+without GPUs; its `infer_plans.yaml`, the inference identity and `realized_deployments`
+are what a launch would record, and no engine log is written. When some sample
+needs inference and `nvidia-smi` finds no GPU, the run is refused before
+anything starts. A judge or other external model is called only when a sample
+is graded. A recorded context overflow the resume scores from its record still
+starts the engine: a task that calls the model more than once asks again.
+
+A sample that a rerun `feedback` sends to another iteration needs the model
+again, which planning cannot foresee. With no engine started it fails at
+`infer`, before any request, and a later `--resume` that starts the engine
+retries it.
+
 ### Anthropic Messages
 
 To call the native Anthropic Messages API, select the dialect explicitly and

sieval/cli/run.py

@@ -48,6 +48,7 @@
 from sieval.cli.resolution import resolve_config_model_types
 from sieval.cli.services import SERVICE_LOG_DIR
 from sieval.cli.session import (
+    config_resumes,
     holds_secret,
     read_persisted_config,
     redact_secrets,
@@ -56,11 +57,16 @@
 )
 from sieval.cli.signals import terminate_as_interrupt
 from sieval.core.models import Deployment
+from sieval.core.runners import RunPlan
 from sieval.core.utils.logging import configure_logging, log_user
 from sieval.infer.backends import get_translator
 from sieval.infer.backends.translator import BackendCommand, inject_user_env
 from sieval.infer.config import InferHandle
-from sieval.infer.deployer import DEFAULT_READY_TIMEOUT, LocalDeployer
+from sieval.infer.deployer import (
+    DEFAULT_READY_TIMEOUT,
+    LocalDeployer,
+    unlaunched_deployment,
+)
 from sieval.infer.introspect import (
     check_launch_gpu,
     declared_target_gpu,
@@ -289,6 +295,81 @@ async def _prepare_launch_batch(
     return plan_dicts, prepared_launches
 
 
+def _resumes(resume: bool | str | Path | None, config: Mapping[str, Any]) -> bool:
+    """Whether some task may resume: ``--resume``, or a ``runner_config`` key."""
+    return bool(resume) or config_resumes(config)
+
+
+def _unlaunched(
+    prepared_launches: Mapping[str, tuple[DeploymentPlan, list[BackendCommand]]],
+    names: set[str],
+) -> dict[str, Deployment]:
+    return {
+        name: unlaunched_deployment(*prepared_launches[name]) for name in sorted(names)
+    }
+
+
+async def _plan_tasks(
+    config_path: Path,
+    prepared_launches: Mapping[str, tuple[DeploymentPlan, list[BackendCommand]]],
+    plan_dicts: dict[str, dict[str, Any]],
+    *,
+    model: str | None,
+    resume: bool | str | Path | None,
+    result_dir: str | None,
+    deterministic: bool | None,
+    invocation: str,
+    cli_model: CliModel | None,
+) -> tuple[dict[str, str], dict[str, RunPlan | Exception]]:
+    """Each task's served model and resume plan, with no engine launched.
+
+    The one call that turns resume planning into a launch decision.
+    """
+    from sieval.cli.session import aplan_session
+
+    return await aplan_session(
+        config_path,
+        model=model,
+        resume=resume,
+        result_dir=result_dir,
+        deterministic=deterministic,
+        infer_plans=plan_dicts or None,
+        invocation=invocation,
+        self_managed_endpoints=frozenset(prepared_launches),
+        unlaunched_deployments=_unlaunched(prepared_launches, set(prepared_launches)),
+        cli_model=cli_model,
+    )
+
+
+async def _models_to_infer(
+    config_path: Path,
+    config: Mapping[str, Any],
+    prepared_launches: Mapping[str, tuple[DeploymentPlan, list[BackendCommand]]],
+    plan_dicts: dict[str, dict[str, Any]],
+    **session: Any,
+) -> set[str]:
+    """The served models some task still has samples to infer with.
+
+    A run where nothing resumes infers everything, so it is not planned. A
+    task whose plan raises infers nothing: its run raises the same before its
+    first sample.
+    """
+    if not prepared_launches:
+        return set()
+    if not _resumes(session["resume"], config):
+        return set(prepared_launches)
+    task_models, plans = await _plan_tasks(
+        config_path, prepared_launches, plan_dicts, **session
+    )
+    return {
+        task_models[task]
+        for task, plan in plans.items()
+        if not isinstance(plan, Exception)
+        and plan["to_infer"]
+        and task_models.get(task) in prepared_launches
+    }
+
+
 async def _run_dry_run(
     config_path: Path,
     *,
@@ -447,9 +528,27 @@ async def _run_all(
                 f"Model name {SERVICE_LOG_DIR!r} is reserved for "
                 f"<result_dir>/logs/{SERVICE_LOG_DIR}/; rename the model."
             )
-        if prepared_launches:
-            await check_launch_gpu()
+        # Plan the resume first: an engine starts only for a model some task
+        # still has samples to infer with.
+        to_launch = await _models_to_infer(
+            config_path,
+            config,
+            prepared_launches,
+            plan_dicts,
+            model=model,
+            resume=resume,
+            result_dir=result_dir,
+            deterministic=deterministic,
+            invocation=invocation,
+            cli_model=cli_model,
+        )
+        if to_launch:
+            await check_launch_gpu(required=True)
+        for model_name in sorted(set(prepared_launches) - to_launch):
+            log_user("No inference needed for model {}; not starting it", model_name)
         for model_name, (plan, commands) in prepared_launches.items():
+            if model_name not in to_launch:
+                continue
             log_user("Starting inference for model: {}", model_name)
 
             last_status = None
@@ -536,10 +635,14 @@ def _progress(
             resume=resume,
             result_dir=result_dir,
             deterministic=deterministic,
-            self_managed_endpoints=frozenset(realized_deployments),
+            self_managed_endpoints=frozenset(prepared_launches),
             infer_plans=plan_dicts or None,
             invocation=invocation,
             realized_deployments=realized_deployments or None,
+            unlaunched_deployments=(
+                _unlaunched(prepared_launches, set(prepared_launches) - to_launch)
+                or None
+            ),
             cli_model=cli_model,
             infer_gpu=await _plan_gpu_text() if plan_dicts else None,
         )

sieval/cli/session.py

@@ -125,6 +125,7 @@
     LegacyStageMatch,
     MultiTaskRunner,
     ResumeValue,
+    RunPlan,
     TaskRunnerConfig,
     as_resume,
     fold_resume_aliases,
@@ -1273,6 +1274,15 @@ def _runner_config_blocks(cfg: Mapping[str, Any]) -> dict[str, dict[str, Any]]:
     return {k: v for k, v in blocks.items() if isinstance(v, dict)}
 
 
+def config_resumes(cfg: Mapping[str, Any]) -> bool:
+    """Whether some ``runner_config`` of *cfg* sets ``resume`` or an alias of it."""
+    return any(
+        block.get(key)
+        for block in _runner_config_blocks(cfg).values()
+        for key in _RESUME_KEYS
+    )
+
+
 def _drop_resume_keys(cfg: dict[str, Any]) -> None:
     """Remove every resume key in place, and a ``runner_config`` left empty."""
     for block in _runner_config_blocks(cfg).values():
@@ -1671,6 +1681,7 @@ def __init__(
         resume_from: str | Path | None = None,
         cli_model: CliModel | None = None,
         infer_gpu: str | None = None,
+        unlaunched_deployments: Mapping[str, Deployment] | None = None,
     ):
         self.config_path = Path(config_path)
         self.model_override = model_override
@@ -1695,7 +1706,17 @@ def __init__(
         self._legacy_endpoint_map: Mapping[str, str] = legacy_endpoint_map
         self._infer_plans: Mapping[str, dict[str, Any]] | None = infer_plans
         self._infer_gpu = infer_gpu
-        self._realized_deployments = realized_deployment_map
+        unlaunched = dict(unlaunched_deployments or {})
+        if set(unlaunched) & set(realized_deployment_map):
+            raise ValueError(
+                "a root model cannot have both a realized and an unlaunched "
+                "deployment: "
+                + ", ".join(sorted(set(unlaunched) & set(realized_deployment_map)))
+            )
+        # Roots served by no engine: bound like launched ones, so every
+        # identity is the launched one, but a task never infers through them.
+        self._unlaunched_roots = frozenset(unlaunched)
+        self._realized_deployments = {**realized_deployment_map, **unlaunched}
         for root_name, deployment in self._realized_deployments.items():
             if not isinstance(root_name, str) or not root_name:
                 raise TypeError("realized deployment keys must be non-empty strings")
@@ -4870,6 +4891,9 @@ def _setup_tasks(self) -> None:
                 inference_recipe=self._inference_recipe_for(candidate.config_name),
                 resume_config_diff=list(self._resume_config_diff),
                 legacy_match=self._legacy_match_for(task_name, runner_config),
+                unserved_candidate=(
+                    self._root_model(candidate.config_name) in self._unlaunched_roots
+                ),
             )
             logger.info("Added task '{}' with class '{}'", task_name, task_spec)
 
@@ -4934,6 +4958,25 @@ def _legacy_match_for(
             == _stage_config(persisted, recorded_name, "inference"),
         }
 
+    def _root_model(self, model_name: str) -> str:
+        """The root of *model_name*'s ``base`` chain: the model that is served."""
+        models_cfg = self._get_named_config_map("models")
+        if model_name not in models_cfg:
+            return model_name
+        return self._model_config_chain(model_name, models_cfg)[0][0]
+
+    def task_models(self) -> dict[str, str]:
+        """Each task's model under test, as the root model config that serves it.
+
+        Read after :meth:`prepare_prelaunch`.
+        """
+        out: dict[str, str] = {}
+        for task_name, context in self._task_requirement_contexts.items():
+            candidate = context.model_bindings.get("candidate")
+            if isinstance(candidate, NamedModelBinding):
+                out[task_name] = self._root_model(candidate.config_name)
+        return out
+
     def _inference_recipe_for(self, model_name: str) -> dict[str, JSONValue] | None:
         """The serving plan of *model_name*'s root, for the inference identity.
 
@@ -4947,13 +4990,7 @@ def _inference_recipe_for(self, model_name: str) -> dict[str, JSONValue] | None:
         infer_plans = self._infer_plans
         if not infer_plans:
             return None
-        models_cfg = self._get_named_config_map("models")
-        root = (
-            self._model_config_chain(model_name, models_cfg)[0][0]
-            if model_name in models_cfg
-            else model_name
-        )
-        plan = infer_plans.get(root)
+        plan = infer_plans.get(self._root_model(model_name))
         if plan is None:
             return None
         recipe = cast(dict[str, Any], copy.deepcopy(unwrap_proxies(plan)))
@@ -5283,6 +5320,34 @@ async def arun(self) -> dict[str, Any]:
         finally:
             await self._close_owned_model_resources()
 
+    async def aplan(self) -> dict[str, RunPlan | Exception]:
+        """What :meth:`arun` would do with each task, by task name; writes nothing.
+
+        Prepares the session as :meth:`arun` does, vendored services included,
+        and asks each task's runner for its :meth:`TaskRunner.plan_run`. A
+        task whose planning raises maps to the exception its run would raise.
+        Raises what :meth:`arun` would raise before any task starts. Service
+        logs go to ``~/.sieval/logs``, never into a result directory.
+        """
+        try:
+            await run_sync(self.prepare_prelaunch)
+            self._stamp_deterministic_seed_contract()
+            result_dir = self._resolve_result_dir()
+            if self.resume_override is True and result_dir:
+                # What `_persist_effective_config` reads before it rewrites.
+                self._recorded_config = await run_sync(
+                    read_persisted_config, Path(result_dir) / "effective_config.yaml"
+                )
+            async with vendored_services_for(
+                self._required_services, result_dir=lambda: None
+            ):
+                await self._prepare_execution()
+                if self.runner is None:
+                    raise RuntimeError("Runner not initialized")
+                return await self.runner.plan_run()
+        finally:
+            await self._close_owned_model_resources()
+
     async def _close_owned_model_resources(self) -> None:
         """Drain and close every session-owned root once; never close externals."""
 
@@ -5331,6 +5396,7 @@ async def arun_session(
     resume_from: str | Path | None = None,
     cli_model: CliModel | None = None,
     infer_gpu: str | None = None,
+    unlaunched_deployments: Mapping[str, Deployment] | None = None,
 ) -> dict[str, Any]:
     """Run tasks defined in a YAML configuration file asynchronously.
 
@@ -5363,6 +5429,10 @@ async def arun_session(
         infer_gpu: The GPUs *infer_plans* were resolved for, as
             ``SIEVAL_TARGET_GPU`` spells them; recorded in ``infer_plans.yaml``
             as ``gpu``, so a resume can tell whether it may reuse the plans.
+        unlaunched_deployments: For a root model ``sieval run`` serves but did
+            not launch, since no task had anything for it to infer: the
+            deployment launching it would realize, nothing listening. Its
+            tasks bind and resume as if it were launched, and refuse to infer.
 
     Returns:
         A dictionary mapping task names to their reports.
@@ -5381,11 +5451,47 @@ async def arun_session(
         resume_from=resume_from,
         cli_model=cli_model,
         infer_gpu=infer_gpu,
+        unlaunched_deployments=unlaunched_deployments,
     )
 
     return await runner.arun()
 
 
+async def aplan_session(
+    config: str | Path,
+    *,
+    model: str | None = None,
+    resume: bool | str | Path | None = None,
+    result_dir: str | None = None,
+    deterministic: bool | None = None,
+    infer_plans: Mapping[str, dict[str, Any]] | None = None,
+    invocation: str | None = None,
+    self_managed_endpoints: frozenset[str] | set[str] = frozenset(),
+    unlaunched_deployments: Mapping[str, Deployment] | None = None,
+    cli_model: CliModel | None = None,
+) -> tuple[dict[str, str], dict[str, RunPlan | Exception]]:
+    """Plan the run :func:`arun_session` would make with the same arguments.
+
+    Returns each task's model under test (its root model config) and its
+    :meth:`EvalSession.aplan` entry. Every served model passes as one of
+    *unlaunched_deployments*. Reads ``--resume`` directories, writes nothing.
+    """
+    session = EvalSession(
+        config_path=config,
+        model_override=model,
+        resume=resume,
+        result_dir_override=result_dir,
+        deterministic_override=deterministic,
+        infer_plans=infer_plans,
+        invocation=invocation,
+        self_managed_endpoints=self_managed_endpoints,
+        unlaunched_deployments=unlaunched_deployments,
+        cli_model=cli_model,
+    )
+    plans = await session.aplan()
+    return session.task_models(), plans
+
+
 def run_session(
     config: str | Path,
     model: str | None = None,