Skip to content

Fix startup deadlock and event loss in discovery - #62

Merged
umputun merged 1 commit into
umputun:masterfrom
paskal:fix/discovery-event-delivery
Aug 20, 2026
Merged

umputun merged 1 commit into
umputun:masterfrom
paskal:fix/discovery-event-delivery

Conversation

@paskal

@paskal paskal commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Two defects in app/discovery around event delivery, both reproduced by the tests added here.

Startup deadlock with more than 100 running containers. Previously, NewEventNotif created eventsCh with a fixed buffer of 100 and then emitted the initial snapshot into it from inside the constructor, while the only consumer (runEventLoop) cannot start until the constructor returns. With 101 or more containers passing the filters the 101st send blocked forever, so do() never reached the event loop and nothing was logged at all. After this change the snapshot is collected into a slice first and the channel is created with max(100, len(initial)), so the whole initial batch fits. emitRunningContainers became runningContainerEvents returning the events instead of publishing them; error handling and the exported API are unchanged.

Events dropped when the consumer is busy. The channel passed to AddEventListener was unbuffered, and go-dockerclient publishes to listeners with a non-blocking send, vendor/github.com/fsouza/go-dockerclient/event.go:340-345:

for _, listener := range eventState.listeners {
    select {
    case listener <- event:
    default:
    }
}

Any event arriving while activate was busy handling the previous one was therefore discarded by the client, leaving a started container without a log stream or a stopped one with a stale streamer. The listener channel is buffered now, which removes the loss for bursts up to the buffer size.

Note that this narrows the loss window rather than closing it completely: if the outgoing channel stays full long enough for the listener buffer to fill as well, the client will start dropping again, and events fired between ListContainers and AddEventListener are still missed. Both need a larger change (a Since timestamp via AddEventListenerWithOptions, which alters the exported DockerClient interface), so I have described them separately in an issue rather than deciding the direction here.

Previously, NewEventNotif emitted the initial container snapshot into a channel
with a fixed buffer of 100 while the consumer could not start until the
constructor returned, so with more than 100 allowed running containers the send
blocked forever and no logging began. After this change the snapshot is
collected first and the events channel is sized to fit it.

The docker events listener channel was unbuffered, and the client drops events
for any listener which is not immediately ready, so events were lost whenever
the consumer was busy. The listener channel is buffered now.
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 32267610063

Coverage decreased (-0.1%) to 79.226%

Details

  • Coverage decreased (-0.1%) from the base build.
  • Patch coverage: 16 of 16 lines across 1 file are fully covered (100%).
  • 2 coverage regressions across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

2 previously-covered lines in 1 file lost coverage.

File Lines Losing Coverage Coverage
app/main.go 2 83.17%

Coverage Stats

Coverage Status
Relevant Lines: 491
Covered Lines: 389
Line Coverage: 79.23%
Coverage Strength: 27.01 hits per line

💛 - Coveralls

@umputun
umputun merged commit 4b6e206 into umputun:master Aug 20, 2026
3 of 4 checks passed
@paskal
paskal deleted the fix/discovery-event-delivery branch August 20, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants