fix(controller): preserve timeout reason across report retries

[Yusef Syed (YusefSyed)]

A local subprocess timeout is initially reported as local subprocess timed out. If that terminal PATCH fails after the process exits, the next reconciliation previously reported the generic signal exit instead, or success if the process exited with code zero during the kill/wait race.

Preserve the timeout result across report retries using the existing killed marker. Keep all completed process records as controller-side launch history. The marker is interpreted within the existing lifecycle: shutdown performs its final reconciliation before shutdown-originated kills, with no later reconciliation in that run.

Regression coverage exercises zero, one, and two failed timeout reports, post-kill exit codes -9 and 0, unchanged ordinary completion, record retention after terminal rollouts leave the query, shutdown ordering, and timeout retries during final reconciliation.

Validation: 49 controller/server/package tests passed. Scoped Ruff, formatting, Pyright, and git diff --check passed. On the unchanged upstream source, the new test file fails the five expected timeout-retry cases and passes the other 14. Processes and HTTP responses are mocked; no GPU or Kubernetes cluster is required.

This change preserves the existing retention behavior; it does not address historical-record growth or repeated scans.

AI assistance: Codex helped investigate, implement, and validate this change. ChatGPT Pro provided design critique that was checked against source and local tests.

[Copilot]

🟡 Changes recommended

Timeout retries can incorrectly replace the timeout error with a generic signal-exit error.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Removes completed local subprocess records after successful terminal status updates while retaining failed updates for retry.

Changes:

• Cleans up exited and timed-out process records.
• Adds reconciliation and retry coverage.

File summaries

File	Description
agentlightning/controller/local_reconciler.py	Removes successfully reported terminal processes.
tests/controller/test_local_reconciler.py	Tests exits, timeouts, retries, and running retention.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Yusef Syed (YusefSyed)]

Preserved the timeout outcome through the existing killed marker. The retry regression now models the completed kill (returncode=-9) and asserts that both PATCH attempts carry local subprocess timed out. A separate case verifies that a natural signal exit without the marker still reports its exit code. The controller/server/package selection passes 42 tests; Ruff and Pyright pass.

[Sylvester Kaczmarek (sylvesterkaczmarek)]

The normal-exit cleanup is correct: retaining the process only until the terminal patch succeeds gives reconciliation a durable retry boundary. There is one timeout retry edge to fix, though. If the process is killed successfully but the FAILED: local subprocess timed out patch fails, the next pass goes through _finish_proc with returncode == -9 and can replace the original timeout reason with a generic signal-exit error. Please preserve the timeout terminal state/reason across a failed patch so retries report the same failure that actually caused the kill.

[Sylvester Kaczmarek (sylvesterkaczmarek)]

Thanks — using the killed marker to preserve the timeout terminal reason across a failed PATCH is exactly the retry invariant I was looking for. Keeping a natural -9 exit on the generic signal path preserves the distinction cleanly.

[Zhiyuan He (hzy46)]

Hi Yusef Syed (@YusefSyed) Thank you for this PR.

Keeping the terminated process in the dictionary is a design choice. It makes the state machine clear, as all the recent information is kept in both agent lightning server and controller. If we delete some rollout in controller, it will make it hard to check whether the process has been launched or not in the controller side. I understand the current logic is right, but breaking the assumption that "controller has all recent information" is somewhat dangerous to me.

[Yusef Syed (YusefSyed)]

Thanks for clarifying. I've dropped the deletion changes and kept the completed records as controller-side launch history. The PR now focuses only on preserving the timeout reason across failed terminal PATCHes.

That retry issue reproduces on current upstream: after a timeout kill and failed status update, later reconciliation changes local subprocess timed out to a generic signal-exit error, or success if the process finishes with exit code zero during the kill/wait race. The revised tests cover both exit codes, repeated reporting failures, retained records, and timeout retries during shutdown's final reconciliation.

The shutdown ordering was checked as well: its own kills happen after the final reconciliation, so those records do not feed back through timeout classification in the normal lifecycle. All 49 controller/server/package tests pass on the revised head, along with scoped lint, formatting, and type checks.

If you'd prefer this timeout-only change in a separate PR, I'm happy to split it. The retention and repeated-scan concern remains a separate design discussion.

AI assistance: Codex and ChatGPT Pro helped investigate and prepare this follow-up.