fix(queue): do not receive a new message after stop - #262
Merged
Conversation
The Swoole consume loops checked the stop flag only at the top of the loop, before blocking on the concurrency slot. At maxCoroutines=1 the push blocks for the whole handler, so a SIGTERM that landed mid-job was first noticed only after the slot came free, and by then the loop had already gone back to receive() and claimed the next message. Every handler process took one more job after being told to stop, so a rolling restart drained for two job durations per process, and a long job accepted after SIGTERM was SIGKILLed at the pod's grace period with its claim stranded. Re-check the flag once the slot is held and leave without receiving. The restart test now publishes a second slow job behind the in-flight one and asserts it is still on the queue after the drain; seen red on both SIGTERM and SIGINT before this change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
loks0n
requested review from
ChiragAgg5k,
abnegate and
lohanidamodar
as code owners
September 11, 2026 17:37
Contributor
|
…roker The restart fixture takes a queue count, so the drain test runs once against the single-queue loop and once against the multi-queue one; the multi-queue cases go red when only the run() check is removed. Remaining depth is read through Broker\Redis::getQueueSize() rather than the list key it happens to use. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Benchmark resultsqueue — workload shapes across both concurrency axes (4 cores, 600 messages, median of 3)
Shared CI runners — treat absolute numbers as rough, compare modes within a run. Commit c123375. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Both Swoole consume loops (
consumeBound()andrun()) checkisStopped()only at the top of the loop, before blocking on the concurrency slot. AtmaxCoroutines=1the push blocks for the whole handler, so a SIGTERM that lands mid-job is first noticed after the slot comes free, and by then the loop has already gone back toreceive()and claimed the next message.Measured on cloud's compose stack with
_APP_WORKERS_NUM=3and six 5s jobs queued: SIGTERM during the first three, and each process still took one more, so the drain ran for 10s instead of 5s. Every handler process accepts one job after being told to stop. With cloud's 30sterminationGracePeriodSecondsand noreap()on most Redis queues, a long build or delete accepted after SIGTERM is SIGKILLed and its claim is stranded onprocessing.Fix
Re-check the flag once the slot is held; pop it and leave without receiving. A message that
receive()already returned before the flag flipped is still processed and committed as before.Test
SwooleRestartTest::testShutdownDrainsJobWithoutRestartingWorkersnow publishes a second slow job behind the in-flight one and asserts it is still on the queue after the drain. Seen red on both SIGTERM and SIGINT before the fix (actual size 2 matches expected size 1), green after.bin/monorepo test queuepasses; Pint passes.Supersedes utopia-php/queue#88, closed.
🤖 Generated with Claude Code