Skip to content

Do not lose streamed log entries - #164

Merged
rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:fix-logs-streaming
Nov 12, 2026
Merged

rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:fix-logs-streaming

Conversation

@milan-zededa

Copy link
Copy Markdown
Contributor

Streaming subscribers of /admin/device/<uuid>/logs (and the other streamed endpoints) could silently miss entries:

  1. stream.publish delivered into a fixed 100-entry channel and dropped data when it was full. A device uploading a large log chunk (hundreds of lines, e.g. a kernel panic stack trace) makes newLogsProcess publish all lines in a tight loop while the HTTP handler flushes one entry at a time, so the tail of the chunk was dropped even though it was written to the store.
  2. The stream handler wrote and flushed the response headers before calling subscribe. A client that treats the response as "stream is open" could miss entries published between the flush and the subscription.

Changes:

  • Give each subscriber an unbounded queue drained into its output channel by a dedicated goroutine, so publish never blocks and never drops for live subscribers.
  • Subscribe before writing headers (and set Content-type before WriteHeader, otherwise it was a no-op).
  • Add a unit test publishing a burst of 5000 entries and checking in-order delivery and channel close on unsubscribe.

Found via recurring EVE evetest TestTelemetrySuite failures where log lines present in Adam's store were never delivered to a live log watch. Verified with a locally built Adam image.

Note: the stream still only carries entries published after subscription; it does not replay what is already stored.

Streaming subscribers could miss entries in two ways. Publishing used a
fixed 100-entry channel and silently dropped data when it was full, so a
large log chunk uploaded by a device (hundreds of lines published in a
tight loop) lost its tail. In addition, the stream handler flushed the
response headers before subscribing, so entries published between the
flush and the subscription were lost even though the client already
considered the stream open.

Give each subscriber an unbounded queue drained by its own goroutine,
and subscribe before the headers are written.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rene
rene merged commit f48e9f6 into lf-edge:master Nov 12, 2026
2 checks passed
@milan-zededa milan-zededa mentioned this pull request Nov 12, 2026
4 of 7 tasks
milan-zededa added a commit to lf-edge/eve that referenced this pull request Nov 12, 2026
Brings the latest Adam patch (lf-edge/adam#164), which stops the log
stream from silently losing entries. Previously Adam dropped published
entries once a subscriber's 100-entry channel was full, so a large log
chunk uploaded by the device (e.g. the hundreds of lines of a kernel
panic stack trace) could lose its tail even though it was stored. Adam
also flushed the stream response before subscribing, so entries
published in between could be missed.

Affected tests in the telemetry suite: TestKernelPanicReport and
TestDeviceLogs. Both failed intermittently because log lines present
in Adam's store never reached the live log watch.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
eriknordmark pushed a commit to lf-edge/eve that referenced this pull request Nov 14, 2026
Brings the latest Adam patch (lf-edge/adam#164), which stops the log
stream from silently losing entries. Previously Adam dropped published
entries once a subscriber's 100-entry channel was full, so a large log
chunk uploaded by the device (e.g. the hundreds of lines of a kernel
panic stack trace) could lose its tail even though it was stored. Adam
also flushed the stream response before subscribing, so entries
published in between could be missed.

Affected tests in the telemetry suite: TestKernelPanicReport and
TestDeviceLogs. Both failed intermittently because log lines present
in Adam's store never reached the live log watch.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit fc84110)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants