[dadamsncsa]

Part of #101: its backend half (the kit half was #103). #101 stays open for the UI's missing Stop on a failed deployment and for Delta's single-node false FAILED (cxip_ep_close ... Unable to free EP object, Qwen2.5-32B on 2 GPUs). Part of #60.

Problem

ModelService.shutdown_deployment returned early for every failed row. vec-inf reports FAILED from a line in a job's log while Slurm still runs the job. A DELETE on that row then cancelled nothing, and the job held its nodes to the time limit: 3277706 ran to TIMEOUT, and 3282264 was still RUNNING after its shutdown request.

What changes

When the DELETE lands on a failed row with a Slurm job id, the backend asks sacct for the job's state:

• if sacct says it ended (COMPLETED, or one of the states the status sync already counts as failed: FAILED, CANCELLED, TIMEOUT, NODE_FAIL, …), the row is left as it is;
• otherwise the job is cancelled through the same impersonated scancel as a running row, and the row becomes shutdown.

Two more cases:

• if sacct gives no state (an error, no row, or no answer in 30 s), the job is cancelled anyway, with a warning. If scancel then says the job is already gone, the row keeps its failure reason; any other cancel failure is recorded on the row, as for a running one;
• a failed row that never had a job (failed/awaiting-download) is left alone, and Slurm is not asked.

Only a DELETE does this. The status sync never cancels, so a false FAILED cannot kill a serving job. The UI still offers no Stop for a failed deployment; API callers and the kit's launch-test send the DELETE.

Evidence (DeltaAI gh-login03, at 2f5edfa)

• tests/test_shutdown_failed_deployment.py: 16 passed. Each test sends the real DELETE through ModelService and LLMInferenceClient in impersonate mode, with a recording impersonation wrapper and a fake sacct.
  • A failed row whose job is RUNNING, PENDING, CONFIGURING, SUSPENDED or REQUEUE_HOLD is cancelled with the same wrapper call as the running control (--no-login-shell <user> -- scancel <job>). It ends shutdown and releases its GPUs.
  • A failed row whose job is FAILED, CANCELLED, TIMEOUT, COMPLETED or NODE_FAIL is not cancelled, and keeps its status and message.
  • sacct erroring, or hanging past the timeout, still cancels the job and logs the warning. A cancel scancel refuses for a gone job leaves a failed row's reason as it was; one that fails otherwise (an impersonation error on a live job) is recorded.
• Removal records: each removal applied to a git archive of the head, and its tests exit 1 (the unedited control exits 0):
  • failed restored to the early return: 5 failed;
  • the ended-state check removed: 5 failed;
  • unknown treated as ended: 2 failed;
  • the shutdown timeout dropped: 1 failed;
  • the no-job guard removed: 1 failed;
  • the reason kept when the job is gone, and another cancel failure recorded: 1 failed each.
• Full backend suite: 215 passed, 1 skipped. black, isort and flake8 are clean.
• Reviews: /simplify, /code-review, a re-review, a fresh reader and two more /code-review rounds; each finding is fixed or answered. 2f5edfa (the fix for the last review's findings) has not been reviewed by an agent.

Rollback

Revert the PR. It has no config or schema change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HnSD7fZZkKzxYacTgvKLL2