Repository navigation
Record kernel panics from pstore instead of kdump - #6822
Conversation
5320e37 to
47ab887
Compare
|
In both runs:
|
ddfc190 to
48ab50f
Compare
|
The EVE-k failure was newlogd dropping the device logs after the first 1000 of the boot until vector was up, which took the panic stack with it while |
48ab50f to
8bffe1d
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6822 +/- ##
==========================================
+ Coverage 28.18% 28.63% +0.45%
==========================================
Files 535 547 +12
Lines 97789 99824 +2035
==========================================
+ Hits 27562 28588 +1026
- Misses 67986 68785 +799
- Partials 2241 2451 +210 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
|
|
Claude finished @rene's task in 3m 25s —— View job Review completeI reviewed the new
Findings: one edge-case correctness issue, posted inline on Verified as OK:
Overall the change is well-structured and thoroughly tested. The rotation finding is narrow (requires a backend that keeps no per-record time and more than 10 accumulated dumps) and does not block the common amd64 efi-pstore/ERST path. |
8bffe1d to
89c0e41
Compare
89c0e41 to
2d79ea1
Compare
|
Hardware test on a Supermicro E300 (ERST pstore backend, ZFS
The first test found that same-second Oops/Panic records were split into one dump per record; the grouping now keeps each dump together. That fix, a same-second comparison between the panic and the stored reboot reason (ERST times records to the second), and the rotation fix from the review comment are squashed into the nodeagent commit, each with a unit test that fails on the previous code. |
|
|
A kernel panic reaches EVE today only through kdump, which needs a crash kernel and its memory reservation, exists only on amd64, and fails on 2-16 GB devices because 128 MB is too little for it. When no crash kernel is loaded the panicking kernel writes the tail of its log to pstore (efi-pstore or ERST) instead. On the next boot nodeagent now reads those records before it handles the reboot reason. It reassembles each dump, logs it the way a Go fatal stack trace is logged, with the kernel log lines on a kernel_panic_stacks log object, and saves it under /persist/kcrashes. The newest panic since the previous boot becomes the reboot reason, BootReasonKernel and the reboot stack, unless a reboot reason stored after the panic or by kdump takes precedence. A panic while EVE was already rebooting, for example in a driver after the watchdog fired, is reported as "kernel panic after <stored reason>" and keeps the boot reason EVE stored, since zedagent picks the configuration to start from by it. Records older than the previous boot are saved and logged but do not set the reason. A record is erased only after its copy on /persist has been fsynced, so a reboot at any point loses nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
With nodeagent reading panic records from pstore, the crash kernel is no longer needed. Dropping crashkernel= returns the 128-512 MB it reserved on amd64 to EVE and applications, and removing the kexec load lets the panicking kernel write its pstore record, which it skips while a crash kernel is loaded. The kdump and kexec packages go, 000-kexec only disables kexec load when the kernel offers it, and storage-init no longer checks for a crash kernel. KERNEL-DUMPS.md now describes the pstore records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Add TestKernelPanicReport to the telemetry suite. It panics an onboarded device with sysrq and checks that, after the reboot, the controller reports BOOT_REASON_KERNEL with the panic as the reboot reason and the tail of the kernel log as the reboot stack, that the dump is logged on kernel_panic_stacks log objects and reaches the controller without a second [dmesg #N] copy, that it is saved in /persist/kcrashes and /persist/newlog, and that pstore is left empty. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2d79ea1 to
87ea903
Compare
Description
EVE records a kernel panic by booting a crash kernel (kdump). This PR records
it from pstore instead and removes the crash kernel.
Motivation
Memory. On amd64 GRUB reserves
crashkernel=2G-16G:128M,16G-128G:256M,128G-:512Mat boot. EVE and itsapplications never get that memory back, and pillar's
TotalMemoryMBexcludesit. Yet the 128 MB tier is too small: the crash kernel keeps about 75 MB for
itself, and on a 4 GB device it runs out of memory before
kdump.shruns. So2–16 GB devices pay for the reservation and still record no panic. Making the
crash kernel work there would mean raising that tier, which costs 3% of RAM on
a 4 GB device. pstore needs no reserved memory. arm64 and riscv64 never load a
crash kernel, so they record no panic at all today.
Getting the panic to the controller. With kdump the controller gets the
reboot reason
kernel panic, dmesg collected: /persist/kcrashes/dmesg-<ts>.log,which names a file on the device but not the panic, and the next boot logs the
crashed kernel's whole dmesg as untagged
[dmesg #N]lines. With this PR thepanic message is in the reboot reason, and the dump is logged like a Go fatal
stack trace: a header naming the panic, every kernel log line on a
kernel_panic_stackslog object, and an end marker. Finding kernel panics inthe controller's logs is then a search for
kernel_panic_stacks. Panics in theinitrd and early boot are caught too; the crash kernel is loaded only once the
rootfs is up.
Example
An
echo c > /proc/sysrq-triggeron EVE-k (TestKernelPanicReport). On thenext boot the controller receives:
lastBootReason:BOOT_REASON_KERNELlastRebootReason:kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crashlastRebootStack: the first 1600 characters of the log from the panic lineonward, which takes in the call trace through
do_syscall_64Whatever the kernel printed before the panic, such as an oops report or a
dm-verity error, is in the same lines ahead of the panic line, as far back as
the pstore records reach; here, to 5.6 s after boot.
lastRebootStackleavesthose lines out so that the panic message and the innermost frames fit.
Question for reviewers: keep the crash kernel as well?
Should EVE keep kdump next to pstore instead of removing it, for example only
on devices with 16 GB or more? A loaded crash kernel takes the panic before
pstore is written, unless
crash_kexec_post_notifiersis set, so keeping bothmeans setting it. kdump adds the crashed kernel's whole dmesg rather than about
the last 25 KB; it keeps no vmcore. It costs the reservation above, kexec in
the kernel, and a crash kernel booting EVE's init on a machine that just
crashed. If we keep it, the second commit changes rather than drops kdump, and
the 2–16 GB tier needs raising or removing.
Hardware support
pstore needs a firmware or hardware store that survives the reboot. The kernel
registers one backend per boot, and nodeagent reads whichever one it is.
dmesg-erst-*records. Not yet run with this PRCONFIG_PSTORE_RAM)CONFIG_ACPI_APEI=yselectsCONFIG_PSTOREand builds ERST. efi-pstore isbuilt in on the core kernel and a module (
CONFIG_EFI_VARS_PSTORE=m) on thehwe kernel that EVE-k runs; the EVE-k test used
efi_pstore. The genericarm64 kernel (v6.1.155) gets the same through
CONFIG_ACPI_APEI=yand theEFI_VARS_PSTOREdefault ofy. This is derived from its defconfig andKconfig, not from a built
.config. The NVIDIA and i.MX kernels have not beenchecked.
SetVariable()at runtime for efi-pstore. Thekernel registers efi-pstore only if the firmware's
EFI_RT_PROPERTIES_TABLEdoes not markSetVariableunsupported.EBBR, the embedded Arm boot spec, does not require runtime
SetVariable.When U-Boot's UEFI keeps variables in a file on the ESP it refuses runtime
writes; its
EFI_RT_VOLATILE_STOREoption accepts them but only into RAM,which does not survive the reboot. So efi-pstore records nothing on those
boards. Arm servers with EDK2 firmware may support it; no arm64 device has
been tried.
12.4 KB compressed. OVMF's 256 KB store held 19 dumps. The kernel then stops
writing (
efi_query_variable_store()keeps 5 KB free) rather thanoverwriting, so the first panic of a loop survives. nodeagent erases records
once they are saved, which frees the store. The size of a real firmware's
variable store and of an ERST store varies by platform and has not been
measured.
What changes
Without a crash kernel the panicking kernel writes the tail of its log to
pstore (see Hardware support above): about 25 KB, which holds the panic message
and the whole call trace. On the next boot nodeagent
reads the records before it handles the reboot reason, and:
/persist/kcrashes/pstore-<time>.log, keeping the 10 newest dumps. Recordsthe running kernel cannot decompress (
.enc.z) are copied as they are anddo not count toward that limit;
lastRebootReason("kernel panic, pstore: Kernel panic - not syncing: …"),
BOOT_REASON_KERNELandlastRebootStack. A reboot reason stored after thepanic stands. A panic during an EVE-initiated reboot is reported as "kernel
panic after " and keeps the boot reason EVE stored, since
zedagent picks the configuration to start from by it: a watchdog reboot
still starts from the backup configuration;
counter) as stale, for example ones an older EVE left in firmware storage.
These are saved and logged but do not set the reason;
/persisthas been fsynced, so areboot at any point loses nothing.
The first commit adds the reader. The second drops
crashkernel=, the kdumpand kexec packages and the
kexec -pload.000-kexecstill disables kexecload when the kernel supports it. The third adds an evetest end-to-end test.
Configuring kexec out of
eve-kernel(CONFIG_KEXEC,CONFIG_KEXEC_FILE,CONFIG_CRASH_DUMP) is a separate eve-kernel change and does not block thisPR.
PR dependencies
up, which loses the kernel panic stack this PR logs.
TestKernelPanicReportfails on EVE-k without it.
How to test and validate this PR
Unit tests:
pkg/pillar/cmd/nodeagent/pstore_test.gocovers dump groupingacross a second boundary, partial dumps, stale and compressed records, each
reboot-reason case, rotation, keeping records whose copy failed, the kernel
stack trace logging including the lines before the panic, and the reboot
stack cut.
End to end:
make evetest NAME=TestKernelPanicReport(addEVETEST_HYPERVISOR=kubevirtfor EVE-k) runs steps 2 and 3 below on anonboarded QEMU device with OVMF and swtpm.
On an onboarded amd64 UEFI device (or QEMU with OVMF and swtpm):
cat /proc/cmdlinehas nocrashkernel=, andMemTotalin/proc/meminfois higher than on the previous image by the size of the oldreservation.
echo c > /proc/sysrq-trigger. The device reboots after120 s.
kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash,boot reason kernel, and a reboot stack starting at the panic message and
holding the call trace;
Kernel panic stack trace from pstore, the[kernel panic #N]lines withobj_typekernel_panic_stacksholdingthe call trace, and
Kernel panic stack trace: end of stack, and no[dmesg #N]copy of the same trace;/persist/kcrashes/pstore-<time>.logexists;/sys/fs/pstoreis empty.corrupted rootfs that panics on every boot. Once a boot reaches nodeagent,
the reason should say "(N panics)", and every dump should be in
/persist/kcrashes.controller. The reason should read "kernel panic after Reboot from agent …"
and the boot reason should stay
BOOT_REASON_REBOOT_CMD.Repeat steps 2–3 on a server with ACPI ERST, where the files are named
dmesg-erst-*.Changelog notes
EVE no longer reserves memory for a crash kernel on amd64 (128 MB on 2–16 GB
devices, 256 MB up to 128 GB, 512 MB above). Kernel panics are now reported to
the controller from firmware pstore on the next boot, including on 2–16 GB
devices where they previously went unrecorded.
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
🤖 Generated with Claude Code