[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Rebased onto main after #215 and #211 merged.

• sieval run plans the resume before it launches anything. _run_all launched every engine before arun_session, where resume planning happened. As a result, a resume that reused every inference still took a GPU and loaded the model, and could not run on a CPU node.
• How it plans. It builds the session exactly as the run will, with each served model bound to unlaunched_deployment(plan, commands). That is the Deployment the launch would realize: same plan projection, endpoints, engine and topology, with only the deployment_id naming no process. Every binding, inference recipe and identity is therefore the launched one. It then asks each task's TaskRunner.plan_run(). That call and arun's _begin_run share one helper, _plan_recorded (inference check, grading-change detection, per-sample plan, overflow replays, retry rollback, model switch check), so the plan counts exactly the samples the run infers. It runs read-only: a resume=<dir> is read in place, not copied, and nothing writes meta.json. The one call is _plan_tasks → aplan_session.
• RunPlan vs ResumePlan. feat(core)!: split the grader identity along postprocess, feedback and report #211's ResumePlan / plan_resume compare identities alone. The new RunPlan / plan_run is the runner's per-sample plan: start, outcomes (as runs[].resume will record them, ResumeOutcomes), new, to_infer, to_grade. A re-report (only report changed) plans as reuse with rereported set.
• Only engines with work start. The run launches only the models some task has to_infer > 0 for. With none, no engine starts and the run needs no GPU. If some sample needs inference and nvidia-smi finds no GPU, the run is refused before anything starts (check_launch_gpu(required=True), next to feat(infer): resolve serving plans for GPUs declared in SIEVAL_TARGET_GPU #215's declared-GPU check). A fresh run is not planned: it infers everything and launches as before.
• Unlaunched models stay safe. They reach the session as unlaunched_deployments and their runners as unserved_candidate.
  • A run whose directory changed between the plan and the run (so that it would now infer) is refused before it records anything.
  • A sample that a rerun feedback sends to another iteration fails at infer, before any request, instead of calling a dead endpoint. A resume that starts the engine retries it. Planning cannot know such a sample in advance.
• Judges and external models already connect lazily. Their clients are built at bind time, and nothing calls them until a sample is graded.
• Launched engines keep their logs under <result_dir>/logs/<model>/, infer_plans.yaml (it still lists every served model), and realized_deployments. self_managed_endpoints still names every served model.

Review fixes

• Plan and run disagreed on an overflow replay at the max_retries limit. _begin_run gives a scored-from-record overflow its retry back; the plan's copy of the planning steps did not. At max_retries: 0 the plan said to_infer: 0, so no engine launched, and the run then refused with "Did change since?" on every rerun. Both now call _plan_recorded. Regression test: test_context_overflow.py::TestResume::test_the_plan_counts_a_scored_refusal_as_the_run_does[0|3]. It fails with the retry give-back removed.
• _check_served had no test. Added test_plan_run.py::TestUnservedCandidate::test_a_regrade_that_sends_a_sample_back_fails_it_at_infer. It checks that no request is sent, the samples fail at infer on iteration 1, and a resume with the engine retries them. It fails with the guard removed.
• _resumes reuses the session's resume-key scan (config_resumes) instead of a copy of _RESUME_KEYS. getattr(self, "_unlaunched_roots", ()) became a direct attribute read.

Rebase adjustments

• feat(core)!: split the grader identity along postprocess, feedback and report #211 made _fast_resume re-report when only report changed. _reusable_report now returns (report, rereport), and _fast_resume keeps feat(core)!: split the grader identity along postprocess, feedback and report #211's body.
• feat(core)!: split the grader identity along postprocess, feedback and report #211's ResumePlan name was taken, so the runner's plan became RunPlan / plan_run (MultiTaskRunner.plan_run, EvalSession.aplan).
• check_launch_gpu keeps feat(infer): resolve serving plans for GPUs declared in SIEVAL_TARGET_GPU #215's per-field mismatch message and adds the required no-GPU refusal.
• test_lazy_launch.py's regrade test reads identity schema 2: feedback differs, reran == ["postprocess", "feedback", "report"]. A task argument is read by every grading stage.

Identity impact. None. The inference identity reads only the plan, which is unchanged. A stub-engine run through sieval run on main (339f7d0) and on this branch recorded the same identity.inference before the rebase. After it, test_a_resume_with_samples_to_infer_starts_the_engine still asserts a resume's inference identity equals a fresh run's. RESULT_SCHEMA_VERSION stays 1.

Related Issues

Closes #213.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)

• Type check clean (ty check)

• Unit tests pass (pdm run pytest)

• pytest tests/unit tests/integration -n 8: 9543 passed, 1 skipped (Linux).

• ruff check, ruff format --check, ty check (whole repo), check_preflight.py and pre-commit run --from-ref origin/main --to-ref HEAD are clean.

tests/integration/test_lazy_launch.py drives _run_all through the real session, runner and OpenAI client. The engine is a stub, and GPU detection is faked.

• Every sample reused, resumed on a node with no GPU: no engine starts, no request is sent, the inference identity is unchanged, reran == [], and no engine log directory is created.
• A grader argument changed, on a node with no GPU: no engine starts.
• A resume with a new sample: the engine starts, only Q3 is sent, and identity.inference equals a fresh run of all three.
• Two served models where only one task gains a sample: only that model's engine starts.
• Samples to infer on a node with no GPU: refused before anything starts, and meta.json is byte-identical.
• A fresh run is not planned.

tests/unit/core/runners/test_plan_run.py covers plan_run for fresh, reuse, new-sample, failed-inference and past-max_retries cases (each leaving the result directory byte-identical), resume=<dir> copying nothing, refusals, MultiTaskRunner.plan_run mapping a refusal to its task, and both unserved_candidate guards.

🤖 Generated with Claude Code