Repository navigation
newlogd: keep early-boot logs until vector is up - #6839
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6839 +/- ##
==========================================
+ Coverage 28.04% 28.34% +0.29%
==========================================
Files 534 545 +11
Lines 97710 99447 +1737
==========================================
+ Hits 27399 28184 +785
- Misses 68080 68841 +761
- Partials 2231 2422 +191 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Tested on top of #6822 (image
Per-boot
Both kvm boots have no gaps. The first EVE-k boot still lacks about 145 msgids between 1199 and 1344, which the run without this PR also lacks. They are absent only from the uploaded stream, so they may be entries below the remote log level rather than drops; the device's keep stream was not collected to tell them apart. |
|
The missing first-boot msgids are not drops. Rerun on EVE-k (image
All 481 entries adam lacks are |
Vector creates its input sockets only once its topology is up, about 20 s into an EVE-k boot, while the services and pillar log their startup. newlogd queued 1000 entries per socket and silently dropped the rest, so the first 1000 log entries of a boot were followed by a gap: 866 entries on one EVE-k boot, including everything nodeagent logged before it published its status. For the first 60 s after it starts writing to vector, newlogd now waits for room in the queue instead of dropping. That stops it reading from memlogd, whose ring buffer holds the backlog meanwhile. Once the window ends a full queue drops as before, so a vector that never comes up or stalls later cannot hold up logging for longer than that. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fb97b8c to
5d001ea
Compare
|
Claude finished @rene's task in 2m 12s —— View job Code reviewI reviewed the change against the pubsub/watchdog flow in Correctness / safety checks that pass:
One finding (posted inline): a pre-existing data race on No blocking issues. |
| case <-timer.C: | ||
| } | ||
| } | ||
| sw.msgsDropped++ |
There was a problem hiding this comment.
msgsDropped is written here from the caller's goroutine (via Write) and also from the run() goroutine on a socket write error (sw.msgsDropped++), and read by ReportDroppedMsgs, all without synchronization — a data race that go test -race can flag. This is pre-existing rather than introduced here, but this PR touches exactly this path, so it's worth closing out (e.g. an atomic.Int64) while you're here.
There was a problem hiding this comment.
Done in fb61868: msgsDropped is now an atomic.Int64.
The dropped-message counter of the vector socket writer is incremented by the logging goroutine when the buffer is full, by the writer goroutine when a socket write fails, and read when deciding whether to report drops, with no synchronization. Make it an atomic.Int64 so the race detector stays quiet and the count stays exact. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Description
When vector is enabled, newlogd loses the device logs that follow the first 1000
entries of every boot, until vector is ready to receive them. Vector creates its
input sockets only once its topology is up. On an EVE-k boot that took about
20 s, during which the services and pillar log their startup. newlogd queues
1000 entries per socket and silently drops the rest. On one EVE-k boot the
device log had a gap of 866 entries right after entry 1000. Everything nodeagent
logged before it published its status was lost, including the kernel panic stack
from #6822, while the reboot reason still arrived.
For the first 60 s after it starts writing to vector, newlogd now waits for room
in the queue instead of dropping. While it waits it stops reading from memlogd,
and memlogd's ring buffer holds the backlog. newlogd already blocks the same way
while it waits for its own vector sinks to appear. After the window a full queue
drops as before, so a vector that never starts, or stalls later, holds up
logging for at most that long.
How to test and validate this PR
TestWriteWaitsForFirstConnectionwrites 50 entries into a 5-entry queue beforethe socket exists, then checks that all 50 arrive in order once it appears.
TestWriteDropsAfterStartupWaitchecks that dropping resumes when the windowends. Both fail against the current writer.
On a device, take the device log of one boot from
/persist/newlogand checkits
msgidsequence. Without this change an EVE-k boot shows a gap starting at1001; with it there is none. Check the device's copy rather than what the
controller received: on a fresh install newlogd uploads only
infoand aboveuntil it has the global config, so the uploaded log of the first boot also
lacks some debug entries.
Changelog notes
Device logs from the first seconds of boot are no longer lost while the log
pipeline starts up.
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
🤖 Generated with Claude Code