Skip to content

Keep managed Codex turns alive across server updates - #201

Open
albrand wants to merge 26 commits into
oursfrom
fix/p7-swap-survive-pool-503
Open

albrand wants to merge 26 commits into
oursfrom
fix/p7-swap-survive-pool-503

Conversation

@albrand

@albrand albrand commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

What was wrong

During shutdown, plugin teardown could make the account-pool route return terminal 503s while the old HTTP listener was still serving. Waiting for open HTTP or upgraded WebSocket connections before stopping plugins could also deadlock. Managed Codex retries were too short for a full restart window. The nested shutdown schedule did not fit inside the desktop's 6-second grace, and Linux AppImage had a separate 4-second process-group kill cap.

The launcher had 500 ms of daemon detach head start in this branch, below the deliberate 3-second phase from get-bb#3143. Read-only daemon logs measured a 4,736 ms maximum from the shutdown marker through the final host-plugin worker stop; this is a proxy, not a direct detach-ack timestamp.

What changed

The server stops accepting HTTP requests before plugin teardown, drains ordinary requests for at most 500 ms, bounds WebSocket and listener close waits, destroys remaining tracked sockets before stopping plugins, and runs periodic update-check stop concurrently with plugin stop inside a 3-second bound. An open streaming request is cut at the drain deadline and becomes a transport failure that Codex can retry.

Managed Codex uses request_max_retries=11 and stream_max_retries=11. The codex-rs retry formula gives 409.4 seconds nominal cumulative wait, within the requested 180–600 second range. Installed Codex 0.162.1 also recovered against a local-only stub for both request 503s and dropped SSE streams.

The daemon head start is now 6,000 ms: the observed 4,736 ms proxy maximum plus a 1,250 ms margin, rounded up. Shared shutdown budgets derive the server bound as 3,700 ms, the server SIGKILL deadline as 4,200 ms, daemon termination as 10,200 ms, launcher maximum as 10,700 ms, and desktop grace as 11,500 ms. Desktop grace increased from 6,000 ms to 11,500 ms to fit that schedule. The AppImage supervisor now uses the same 11,500 ms grace instead of its hardcoded 4,000 ms SIGKILL timer. The schedule assertion and a real-child test cover the chain. No daemon/server wire fields changed; protocol version remains 240.

How you verified

The pooled HTTP request reproduces the 503 at f09e8bf and passes with the shutdown fix. The hanging shutdown test is red at f09e8bf; upgraded-socket, stalled-updater and socket-order regressions pass with the bounded listener-first shutdown. Red/green/mutant log paths and the read-only 18:55 live incident timeline are in /tmp/p7-swap503/READY.

After a plain merge of origin/ours 7a45943dfd94f1a65264f52c72805df6bea48070 (#206), exact head is 3b599c819bac233f1cc9e36631e9c4551c8c41a8. HSRPC nested shutdown, seven server shutdown, and AppImage-grace tests pass; the 500 ms head-start mutant fails. Exact-head CI run 38750103412 passed, including macOS and Ubuntu Package Smoke and the Ubuntu AppImage lifecycle smoke. The PR's dev-stack restart reproduction is prepared at /tmp/p7-swap503/repro-p7-swap503.sh and is not run; see READY for the environment prerequisite.

Fixes #

AGENT GENERATED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant