fix(eval): close incremental rollout submission - #629
Conversation
Prevent early rollout failures from finalizing a partially submitted Taskset job.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Close jobs after scheduling completes and preserve pre-trace launch failures in the final job result.
End submission as soon as scheduling finishes and keep telemetry draining outside the lifecycle boundary.
Ensure interrupted taskset runs close their owned platform job as failed, including cancellation during initial job registration.
Keep the incremental lifecycle regression compatible with canonical record links added on main.
Keep multi-call jobs open until an explicit finish, and provide an async context lifecycle that reports failed exits reliably.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e8fb56b. Configure here.
| _exc: BaseException | None, | ||
| _tb: TracebackType | None, | ||
| ) -> None: | ||
| await self.finish(failed=exc_type is not None) |
There was a problem hiding this comment.
Context exit ignores rollout errors
High Severity
Job.__aexit__ reports failed only from a raised exception, so a caller-owned job whose rollouts returned Run.failed still exits as successful. Taskset.run and the cookbooks treat job.errors as failure; rollout and HostedRuntime.run convert launch and mid-run failures into error runs without raising. baseline.py uses this context manager, so hosted and launch failures can mark the job successful.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit e8fb56b. Configure here.
| ) | ||
| history.append(row) | ||
| out.write_text(json.dumps(history, indent=1)) | ||
| await session.finish(failed=bool(session.errors)) |
There was a problem hiding this comment.
Training jobs stay open on crash
Medium Severity
session.finish runs only after a successful loop. Job.start now registers is_open=True, so a raise from taskset.run or the trainer leaves the platform job open. In daytona-rl/train.py this is especially visible: finish sits inside the try while finally only drops the pool.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit e8fb56b. Configure here.
| async def close(self, *, failed: bool = False) -> None: | ||
| """Close the platform job after the final conversation turn.""" | ||
| if self.job is not None: | ||
| await self.job.finish(failed=failed) |
There was a problem hiding this comment.
Chat close leaves jobs unfinished
Medium Severity
Chat.send opens a job via Job.start (is_open=True) and raises on an error turn without closing it. The new close method is opt-in, does not clear self.job, and defaults failed=False even when job.errors is set, so later send calls attach traces to an already-exited job and failed conversations can stay open or report success.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e8fb56b. Configure here.
ChinmayK0607
changed the title
1 participant
What changed
Taskset.runjobs as open while rollouts are being submitted.finally, including cancellation, hosted launch failures, and exceptional exits.Job.finish(), with async-context-manager support.Chat.close()and close long-running cookbook sessions after their final batch.Why
The SDK submitted rollouts incrementally but did not tell the platform when submission was still in progress or definitively finished. An early terminal trace could therefore make a partial benchmark look complete, while pre-trace hosted failures could leave expected work unrepresented.
Impact
Taskset jobs remain open until scheduling ends, failed submissions settle accurately, and multi-call jobs have an explicit safe completion lifecycle.
Depends on https://github.com/hud-evals/hud-monorepo/pull/1499
Validation
54 passedacrosshud/eval/tests/test_job.pyandhud/eval/tests/test_rollout.pyty checkpassed forjob.py,chat.py, andtaskset.pyHUD-2748