Skip to content

newlogd: keep early-boot logs until vector is up - #6839

Merged
eriknordmark merged 2 commits into
lf-edge:masterfrom
eriknordmark:newlogd-vector-wait
Oct 26, 2026
Merged

eriknordmark merged 2 commits into
lf-edge:masterfrom
eriknordmark:newlogd-vector-wait

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

When vector is enabled, newlogd loses the device logs that follow the first 1000
entries of every boot, until vector is ready to receive them. Vector creates its
input sockets only once its topology is up. On an EVE-k boot that took about
20 s, during which the services and pillar log their startup. newlogd queues
1000 entries per socket and silently drops the rest. On one EVE-k boot the
device log had a gap of 866 entries right after entry 1000. Everything nodeagent
logged before it published its status was lost, including the kernel panic stack
from #6822, while the reboot reason still arrived.

For the first 60 s after it starts writing to vector, newlogd now waits for room
in the queue instead of dropping. While it waits it stops reading from memlogd,
and memlogd's ring buffer holds the backlog. newlogd already blocks the same way
while it waits for its own vector sinks to appear. After the window a full queue
drops as before, so a vector that never starts, or stalls later, holds up
logging for at most that long.

How to test and validate this PR

TestWriteWaitsForFirstConnection writes 50 entries into a 5-entry queue before
the socket exists, then checks that all 50 arrive in order once it appears.
TestWriteDropsAfterStartupWait checks that dropping resumes when the window
ends. Both fail against the current writer.

On a device, take the device log of one boot from /persist/newlog and check
its msgid sequence. Without this change an EVE-k boot shows a gap starting at
1001; with it there is none. Check the device's copy rather than what the
controller received: on a fresh install newlogd uploads only info and above
until it has the global config, so the uploaded log of the first boot also
lacks some debug entries.

Changelog notes

Device logs from the first seconds of boot are no longer lost while the log
pipeline starts up.

PR Backports

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.34%. Comparing base (a22f9fe) to head (fb61868).
⚠️ Report is 86 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6839      +/-   ##
==========================================
+ Coverage   28.04%   28.34%   +0.29%     
==========================================
  Files         534      545      +11     
  Lines       97710    99447    +1737     
==========================================
+ Hits        27399    28184     +785     
- Misses      68080    68841     +761     
- Partials     2231     2422     +191     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

Tested on top of #6822 (image 0.0.0-pstore-newlogd-test-2d0f5999) with TestKernelPanicReport, which needs the kernel panic stack nodeagent logs early in the boot after the panic to reach the controller.

without this PR (48ab50f2) with this PR
EVE-k (kubevirt) FAIL: panic header never uploaded PASS (800 s)
kvm PASS PASS (511 s)

Per-boot msgid sequence of the logs adam received, EVE-k:

boot without this PR with this PR
first gap 1001–1199 no gap at 1001
after the panic gap 1001–1866 no gaps

Both kvm boots have no gaps. The first EVE-k boot still lacks about 145 msgids between 1199 and 1344, which the run without this PR also lacks. They are absent only from the uploaded stream, so they may be entries below the remote log level rather than drops; the device's keep stream was not collected to tell them apart.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

The missing first-boot msgids are not drops. Rerun on EVE-k (image 0.0.0-pstore-newlogd-test-6a9465b7, this PR on top of #6822 at 8bffe1db7; TestKernelPanicReport PASS, 602 s), this time with all of /persist/newlog copied off the device:

boot on the device received by adam
first (fresh install) msgid 1–8573, no gaps 8092
after the panic msgid 1–5333, no gaps 5333

All 481 entries adam lacks are debug, from msgid 1200–2941, about 5 s early in the first boot (nim 334, diag 66, zedrouter 51, …). No debug entry precedes 1200, and all 3533 from 2942 on were uploaded. On a fresh install there is no logLevelsCache.json yet, so newlogd's remote log level is info until it gets the global config, while the agents already log at debug. The second boot starts from the cache and has no gap.

Vector creates its input sockets only once its topology is up, about
20 s into an EVE-k boot, while the services and pillar log their
startup. newlogd queued 1000 entries per socket and silently dropped
the rest, so the first 1000 log entries of a boot were followed by a
gap: 866 entries on one EVE-k boot, including everything nodeagent
logged before it published its status.

For the first 60 s after it starts writing to vector, newlogd now
waits for room in the queue instead of dropping. That stops it reading
from memlogd, whose ring buffer holds the backlog meanwhile. Once the
window ends a full queue drops as before, so a vector that never comes
up or stalls later cannot hold up logging for longer than that.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eriknordmark
eriknordmark marked this pull request as ready for review October 18, 2026 08:16
@eriknordmark eriknordmark added stable Should be backported to stable release(s) bug Something isn't working labels Oct 19, 2026
@rene

rene commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 2m 12s —— View job


Code review

I reviewed the change against the pubsub/watchdog flow in newlogd. The approach is sound and the tests cover both the wait-and-deliver and the drop-after-window behaviors.

Correctness / safety checks that pass:

  • The 60 s block in sendLogsToVector is bounded and cannot trip the watchdog: ps.StillRunning is kicked from the main Run loop and doMoveCompressFile, both independent of the blocked goroutine. Back-pressure only stalls loggerChan/memlogd's ring buffer, which is the intended mechanism.
  • waitDeadline uses time.Now().Add(...) / time.Until(...), which carry Go's monotonic clock reading, so the window is immune to the wall-clock jumps common during early boot (e.g. NTP sync).
  • slices.Clone(p) is now hoisted before the select and the same msg is reused across both send attempts — correct, FIFO ordering preserved; the only cost is one extra clone on the pure-drop path, which is negligible.
  • Deadline is anchored at writer creation, and the writers are created only after the sink sockets exist, so "60 s after it starts writing to vector" matches the code.

One finding (posted inline): a pre-existing data race on msgsDropped — written from both the Write caller goroutine and the run() goroutine (on socket write error) and read by ReportDroppedMsgs, with no synchronization. Not introduced by this PR, but it sits directly on the path you're editing and go test -race can surface it; worth converting to an atomic while here.

No blocking issues.
· branch newlogd-vector-wait

Comment thread pkg/newlog/cmd/bufferedSocketWriter.go Outdated
case <-timer.C:
}
}
sw.msgsDropped++

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

msgsDropped is written here from the caller's goroutine (via Write) and also from the run() goroutine on a socket write error (sw.msgsDropped++), and read by ReportDroppedMsgs, all without synchronization — a data race that go test -race can flag. This is pre-existing rather than introduced here, but this PR touches exactly this path, so it's worth closing out (e.g. an atomic.Int64) while you're here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in fb61868: msgsDropped is now an atomic.Int64.

The dropped-message counter of the vector socket writer is incremented
by the logging goroutine when the buffer is full, by the writer
goroutine when a socket write fails, and read when deciding whether to
report drops, with no synchronization. Make it an atomic.Int64 so the
race detector stays quiet and the count stays exact.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eriknordmark
eriknordmark merged commit 0b6761e into lf-edge:master Oct 26, 2026
61 of 66 checks passed
@eriknordmark
eriknordmark deleted the newlogd-vector-wait branch October 27, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants