You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
sieval run --dry-run: report each task's per-stage resume plan #219
sieval run --dry-run (#217) reports each task's resume_plan outcome counts, but not the per-stage comparison that #211 introduced. A reader sees kept: 3, rereported: 3 without being told which stage changed, and has to derive whether a task infers from to_infer.
Proposal
Report, per task, the stage plan the runner decides from:
stages: plan_resume's comparison of the recorded identities with this run's (same, differs or unknown per stage, identity_schema, legacy_grader), with rerun_from taken from the runner's own decision, so report names a re-report;
needs_inference: whether the task sends any sample to its model under test, by the same test sieval run launches engines by.
A report-only change should read start: reuse, rereported equal to kept, rerun_from: report, to_infer: 0 and needs_inference: false.
The dry-run's needs_inference must agree with what sieval run then launches, for each outcome.
Activity
feat(cli): sieval run --dry-run reports each task's stage plan (#220)