Skip to content

sieval run --dry-run: report each task's per-stage resume plan #219

Closed

Description

@ethan-scitix

What

sieval run --dry-run (#217) reports each task's resume_plan outcome counts, but not the per-stage comparison that #211 introduced. A reader sees kept: 3, rereported: 3 without being told which stage changed, and has to derive whether a task infers from to_infer.

Proposal

Report, per task, the stage plan the runner decides from:

  • stages: plan_resume's comparison of the recorded identities with this run's (same, differs or unknown per stage, identity_schema, legacy_grader), with rerun_from taken from the runner's own decision, so report names a re-report;
  • needs_inference: whether the task sends any sample to its model under test, by the same test sieval run launches engines by.

A report-only change should read start: reuse, rereported equal to kept, rerun_from: report, to_infer: 0 and needs_inference: false.

The dry-run's needs_inference must agree with what sieval run then launches, for each outcome.

Activity

  1. added a commit that references this issue on Oct 23, 2026
    9bc4b8d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions