Summary
When the workers that process ML results stop consuming, nothing says so. The platform keeps accepting results, queues them, and then quietly throws them away, while the jobs that produced them are marked failed for "no progress". From the outside it looks like the ML pipeline is broken; in fact inference succeeded and the results were discarded on arrival.
That is what happened in a recent production incident. Every result-processing worker had lost its connection to the message broker and never reconnected. The container runtime still reported them as running with a restart count of zero. Results were POSTed successfully — the ML workers logged result posts: 62 (success: 62, failed: 0) — accepted by the API, queued, and never processed. Two large jobs (roughly 37,000 and 11,000 images) were destroyed, and capacity had been silently halving for two and a half days before the last consumer died.
The infrastructure-side alert for this is being handled separately. This issue is about the part the application owns: making the condition observable and not silently destructive.
List of Changes
| Change (user effect) |
How (implementation) |
| Operators can see that results are arriving but not being processed |
Expose queue depth and consumer count for the result queue as a health signal (management command, health endpoint, or existing monitoring hook — design open) |
| A job is not marked failed when its results are merely queued rather than lost |
Have the stalled-job check distinguish "no results arriving" from "results arriving but not consumed" before it reaps |
| A late result cannot be silently discarded without trace |
Count and surface discards, rather than only logging them at INFO |
Detailed Description
Three separate things combined to make this invisible.
Container liveness means nothing here. All eight result-processing workers reported running with RestartCount=0 throughout. The only external symptom was that their logs stopped. Any check based on process or container state passed.
Results are accepted unconditionally. The API takes the POST and queues the work (Queued pipeline result: task_id=...). Nothing verifies that anything is on the other end of the queue. A worker that has lost its broker connection is indistinguishable, from the producer's side, from a healthy one.
The stalled-job check then blames the job. With results queued but unconsumed, process stays at 0% and jobs_health_check reaps the job for no progress. The resulting message points at the job, not at the missing consumers, which sends investigation in the wrong direction. In this incident it also destroyed work that had genuinely completed.
There is a fourth, quieter cost. Once a job is terminal, its late-arriving results hit the discard branch and are logged at INFO as result arrived after the job already finished ...; ignoring. During recovery, roughly 9,200 such messages were consumed and thrown away. That is correct behaviour in isolation, but nothing aggregates it, so a large volume of discarded inference produces no signal at all.
Suggested direction, not a settled design
- A queue-health signal is the strongest single indicator, because "messages ready > 0 and consumers == 0" states the problem directly and cannot be faked by a worker that is alive but not working.
- The reaper change is the higher-value correctness fix but needs care: it must not simply refuse to reap, or genuinely stuck jobs will accumulate. Distinguishing the two cases probably requires knowing whether results for that job are sitting unconsumed.
- Discard counting is cheap and worth doing regardless.
What we still need to verify
- Whether the application can read broker consumer counts cleanly in this deployment, or whether that inspection belongs entirely to infrastructure monitoring.
celery inspect proved unreliable in this incident: it returned empty output when workers were wedged, which is precisely when it was needed.
- What the reaper should do when it detects queued-but-unconsumed results — hold the job, extend the deadline, or fail it with an accurate reason. Each has different consequences for a genuinely stuck job.
- Whether surfacing this belongs in an existing health surface rather than a new one, to avoid adding another thing that itself needs monitoring.
Related
Summary
When the workers that process ML results stop consuming, nothing says so. The platform keeps accepting results, queues them, and then quietly throws them away, while the jobs that produced them are marked failed for "no progress". From the outside it looks like the ML pipeline is broken; in fact inference succeeded and the results were discarded on arrival.
That is what happened in a recent production incident. Every result-processing worker had lost its connection to the message broker and never reconnected. The container runtime still reported them as running with a restart count of zero. Results were POSTed successfully — the ML workers logged
result posts: 62 (success: 62, failed: 0)— accepted by the API, queued, and never processed. Two large jobs (roughly 37,000 and 11,000 images) were destroyed, and capacity had been silently halving for two and a half days before the last consumer died.The infrastructure-side alert for this is being handled separately. This issue is about the part the application owns: making the condition observable and not silently destructive.
List of Changes
Detailed Description
Three separate things combined to make this invisible.
Container liveness means nothing here. All eight result-processing workers reported
runningwithRestartCount=0throughout. The only external symptom was that their logs stopped. Any check based on process or container state passed.Results are accepted unconditionally. The API takes the POST and queues the work (
Queued pipeline result: task_id=...). Nothing verifies that anything is on the other end of the queue. A worker that has lost its broker connection is indistinguishable, from the producer's side, from a healthy one.The stalled-job check then blames the job. With results queued but unconsumed,
processstays at 0% andjobs_health_checkreaps the job for no progress. The resulting message points at the job, not at the missing consumers, which sends investigation in the wrong direction. In this incident it also destroyed work that had genuinely completed.There is a fourth, quieter cost. Once a job is terminal, its late-arriving results hit the discard branch and are logged at INFO as
result arrived after the job already finished ...; ignoring.During recovery, roughly 9,200 such messages were consumed and thrown away. That is correct behaviour in isolation, but nothing aggregates it, so a large volume of discarded inference produces no signal at all.Suggested direction, not a settled design
What we still need to verify
celery inspectproved unreliable in this incident: it returned empty output when workers were wedged, which is precisely when it was needed.Related