Skip to content

Record kernel panics from pstore instead of kdump - #6822

Merged
rene merged 3 commits into
lf-edge:masterfrom
eriknordmark:pstore-panic-records
Oct 25, 2026
Merged

rene merged 3 commits into
lf-edge:masterfrom
eriknordmark:pstore-panic-records

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

EVE records a kernel panic by booting a crash kernel (kdump). This PR records
it from pstore instead and removes the crash kernel.

Motivation

Memory. On amd64 GRUB reserves
crashkernel=2G-16G:128M,16G-128G:256M,128G-:512M at boot. EVE and its
applications never get that memory back, and pillar's TotalMemoryMB excludes
it. Yet the 128 MB tier is too small: the crash kernel keeps about 75 MB for
itself, and on a 4 GB device it runs out of memory before kdump.sh runs. So
2–16 GB devices pay for the reservation and still record no panic. Making the
crash kernel work there would mean raising that tier, which costs 3% of RAM on
a 4 GB device. pstore needs no reserved memory. arm64 and riscv64 never load a
crash kernel, so they record no panic at all today.

Getting the panic to the controller. With kdump the controller gets the
reboot reason kernel panic, dmesg collected: /persist/kcrashes/dmesg-<ts>.log,
which names a file on the device but not the panic, and the next boot logs the
crashed kernel's whole dmesg as untagged [dmesg #N] lines. With this PR the
panic message is in the reboot reason, and the dump is logged like a Go fatal
stack trace: a header naming the panic, every kernel log line on a
kernel_panic_stacks log object, and an end marker. Finding kernel panics in
the controller's logs is then a search for kernel_panic_stacks. Panics in the
initrd and early boot are caught too; the crash kernel is loaded only once the
rootfs is up.

Example

An echo c > /proc/sysrq-trigger on EVE-k (TestKernelPanicReport). On the
next boot the controller receives:

  • lastBootReason: BOOT_REASON_KERNEL
  • lastRebootReason: kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash
  • lastRebootStack: the first 1600 characters of the log from the panic line
    onward, which takes in the call trace through do_syscall_64
  • in the device log, 366 lines, abridged here:
error   Kernel panic stack trace from pstore (Panic#1 at 2026-10-16T11:47:56Z, from the previous boot or later) with 364 lines: Kernel panic - not syncing: sysrq triggered crash
warning [kernel panic #0]: <6>[    5.617018][    T1] usb usb10: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.18
…
warning [kernel panic #316]: <6>[  422.366323][  T107] cni0: port 12(veth9e5a380e) entered forwarding state
warning [kernel panic #317]: <6>[  423.523524][T11905] sysrq: Trigger a crash
warning [kernel panic #318]: <0>[  423.526094][T11905] Kernel panic - not syncing: sysrq triggered crash
warning [kernel panic #319]: <4>[  423.530990][T11905] CPU: 3 UID: 0 PID: 11905 Comm: sh Tainted: P           O        6.18.55-hwe-d6791352e9c5 #1 PREEMPT(voluntary)
warning [kernel panic #320]: <4>[  423.540189][T11905] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
warning [kernel panic #321]: <4>[  423.545029][T11905] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
warning [kernel panic #322]: <4>[  423.550569][T11905] Call Trace:
warning [kernel panic #323]: <4>[  423.552210][T11905]  <TASK>
warning [kernel panic #324]: <4>[  423.553546][T11905]  dump_stack_lvl+0x5f/0x90
warning [kernel panic #325]: <4>[  423.558591][T11905]  dump_stack+0x10/0x18
warning [kernel panic #326]: <4>[  423.560479][T11905]  vpanic+0x130/0x370
warning [kernel panic #327]: <4>[  423.562396][T11905]  panic+0x67/0x67
warning [kernel panic #328]: <4>[  423.563922][T11905]  sysrq_handle_crash+0x1a/0x20
warning [kernel panic #329]: <4>[  423.566152][T11905]  __handle_sysrq.cold+0x20/0x140
warning [kernel panic #330]: <4>[  423.569122][T11905]  write_sysrq_trigger+0x62/0x90
warning [kernel panic #331]: <4>[  423.571333][T11905]  proc_reg_write+0x69/0xa0
warning [kernel panic #332]: <4>[  423.575739][T11905]  vfs_write+0xd4/0x490
…
warning [kernel panic #362]: <4>[  423.661470][T11905]  </TASK>
warning [kernel panic #363]: <0>[  423.663540][T11905] Kernel Offset: 0x14600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
error   Kernel panic stack trace: end of stack

Whatever the kernel printed before the panic, such as an oops report or a
dm-verity error, is in the same lines ahead of the panic line, as far back as
the pstore records reach; here, to 5.6 s after boot. lastRebootStack leaves
those lines out so that the panic message and the innermost frames fit.

Question for reviewers: keep the crash kernel as well?

Should EVE keep kdump next to pstore instead of removing it, for example only
on devices with 16 GB or more? A loaded crash kernel takes the panic before
pstore is written, unless crash_kexec_post_notifiers is set, so keeping both
means setting it. kdump adds the crashed kernel's whole dmesg rather than about
the last 25 KB; it keeps no vmcore. It costs the reservation above, kexec in
the kernel, and a crash kernel booting EVE's init on a machine that just
crashed. If we keep it, the second commit changes rather than drops kdump, and
the 2–16 GB tier needs raising or removing.

Hardware support

pstore needs a firmware or hardware store that survives the reboot. The kernel
registers one backend per boot, and nodeagent reads whichever one it is.

Platform Backend Status
amd64 UEFI efi-pstore: EFI variables in the firmware's variable store Tested in QEMU with OVMF (kvm and EVE-k); lab Supermicros hold efi-pstore records
amd64 servers with an ACPI ERST table ERST: the platform's error record store, through APEI ERST registers instead of efi-pstore; the lab device sc-sm-e300-10 holds dmesg-erst-* records. Not yet run with this PR
arm64 UEFI (generic kernel) efi-pstore, or ERST where ACPI provides it Kernel side built; depends on the firmware; untested
arm64 U-Boot, riscv64 ramoops would be needed (CONFIG_PSTORE_RAM) Not built, so these platforms record no panic, as with kdump today
  • Kernel configuration. Both amd64 kernels build both backends.
    CONFIG_ACPI_APEI=y selects CONFIG_PSTORE and builds ERST. efi-pstore is
    built in on the core kernel and a module (CONFIG_EFI_VARS_PSTORE=m) on the
    hwe kernel that EVE-k runs; the EVE-k test used efi_pstore. The generic
    arm64 kernel (v6.1.155) gets the same through CONFIG_ACPI_APEI=y and the
    EFI_VARS_PSTORE default of y. This is derived from its defconfig and
    Kconfig, not from a built .config. The NVIDIA and i.MX kernels have not been
    checked.
  • The firmware must support SetVariable() at runtime for efi-pstore. The
    kernel registers efi-pstore only if the firmware's
    EFI_RT_PROPERTIES_TABLE does not mark SetVariable unsupported.
    EBBR, the embedded Arm boot spec, does not require runtime SetVariable.
    When U-Boot's UEFI keeps variables in a file on the ESP it refuses runtime
    writes; its EFI_RT_VOLATILE_STORE option accepts them but only into RAM,
    which does not survive the reboot. So efi-pstore records nothing on those
    boards. Arm servers with EDK2 firmware may support it; no arm64 device has
    been tried.
  • Capacity. efi-pstore keeps about 25 KB of log per panic: 15–16 variables,
    12.4 KB compressed. OVMF's 256 KB store held 19 dumps. The kernel then stops
    writing (efi_query_variable_store() keeps 5 KB free) rather than
    overwriting, so the first panic of a loop survives. nodeagent erases records
    once they are saved, which frees the store. The size of a real firmware's
    variable store and of an ERST store varies by platform and has not been
    measured.

What changes

Without a crash kernel the panicking kernel writes the tail of its log to
pstore (see Hardware support above): about 25 KB, which holds the panic message
and the whole call trace. On the next boot nodeagent
reads the records before it handles the reboot reason, and:

  • reassembles each dump and logs it as shown above;
  • saves each dump, its records reassembled into one file, to
    /persist/kcrashes/pstore-<time>.log, keeping the 10 newest dumps. Records
    the running kernel cannot decompress (.enc.z) are copied as they are and
    do not count toward that limit;
  • turns the newest panic since the previous boot into lastRebootReason
    ("kernel panic, pstore: Kernel panic - not syncing: …"),
    BOOT_REASON_KERNEL and lastRebootStack. A reboot reason stored after the
    panic stands. A panic during an EVE-initiated reboot is reported as "kernel
    panic after " and keeps the boot reason EVE stored, since
    zedagent picks the configuration to start from by it: a watchdog reboot
    still starts from the backup configuration;
  • treats records older than the previous boot (the mtime of the restart
    counter) as stale, for example ones an older EVE left in firmware storage.
    These are saved and logged but do not set the reason;
  • erases a record only once its copy on /persist has been fsynced, so a
    reboot at any point loses nothing.

The first commit adds the reader. The second drops crashkernel=, the kdump
and kexec packages and the kexec -p load. 000-kexec still disables kexec
load when the kernel supports it. The third adds an evetest end-to-end test.

Configuring kexec out of eve-kernel (CONFIG_KEXEC, CONFIG_KEXEC_FILE,
CONFIG_CRASH_DUMP) is a separate eve-kernel change and does not block this
PR.

PR dependencies

How to test and validate this PR

Unit tests: pkg/pillar/cmd/nodeagent/pstore_test.go covers dump grouping
across a second boundary, partial dumps, stale and compressed records, each
reboot-reason case, rotation, keeping records whose copy failed, the kernel
stack trace logging including the lines before the panic, and the reboot
stack cut.

End to end: make evetest NAME=TestKernelPanicReport (add
EVETEST_HYPERVISOR=kubevirt for EVE-k) runs steps 2 and 3 below on an
onboarded QEMU device with OVMF and swtpm.

On an onboarded amd64 UEFI device (or QEMU with OVMF and swtpm):

  1. cat /proc/cmdline has no crashkernel=, and MemTotal in
    /proc/meminfo is higher than on the previous image by the size of the old
    reservation.
  2. Trigger a panic: echo c > /proc/sysrq-trigger. The device reboots after
    120 s.
  3. After the reboot, check:
    • the controller shows the last reboot reason
      kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash,
      boot reason kernel, and a reboot stack starting at the panic message and
      holding the call trace;
    • the device log has Kernel panic stack trace from pstore, the
      [kernel panic #N] lines with obj_type kernel_panic_stacks holding
      the call trace, and Kernel panic stack trace: end of stack, and no
      [dmesg #N] copy of the same trace;
    • /persist/kcrashes/pstore-<time>.log exists;
    • /sys/fs/pstore is empty.
  4. Panic a few times in a row before nodeagent starts, or keep a deliberately
    corrupted rootfs that panics on every boot. Once a boot reaches nodeagent,
    the reason should say "(N panics)", and every dump should be in
    /persist/kcrashes.
  5. Panic while EVE is rebooting, for example after a reboot from the
    controller. The reason should read "kernel panic after Reboot from agent …"
    and the boot reason should stay BOOT_REASON_REBOOT_CMD.

Repeat steps 2–3 on a server with ACPI ERST, where the files are named
dmesg-erst-*.

Changelog notes

EVE no longer reserves memory for a crash kernel on amd64 (128 MB on 2–16 GB
devices, 256 MB up to 128 GB, 512 MB above). Kernel panics are now reported to
the controller from firmware pstore on the next boot, including on 2–16 GB
devices where they previously went unrecorded.

PR Backports

  • 17.0-stable: No, behavior change.
  • 16.0-stable: No, behavior change.
  • 14.5-stable: No, behavior change.
  • 13.4-stable: No, behavior change.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

@eriknordmark

Copy link
Copy Markdown
Contributor Author

TestKernelPanicReport passes on both flavors, built from 47ab887c4 (0.0.0-pstore-panic-records-47ab887c), on an onboarded QEMU device with OVMF and swtpm. The pstore backend was efi_pstore.

Flavor Result Reboot reason Saved dump kernel_panic_stacks lines in /persist/newlog
kvm PASS (474 s) kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash /persist/kcrashes/pstore-2026-10-16T12:03:28Z.log 356
k PASS (674 s) same /persist/kcrashes/pstore-2026-10-16T12:16:29Z.log 390

In both runs:

  • The controller reported BOOT_REASON_KERNEL, with the panic's time as lastRebootTime.
  • The controller received the Kernel panic stack trace from pstore header, the [kernel panic #N] line with the panic message, and the end marker, and no [dmesg #N] lines.
  • lastRebootStack was the tail of the saved dump.
  • /sys/fs/pstore was empty afterwards.

lastRebootStack is limited to the last 1600 characters (maxRebootStackSize, unchanged here). On EVE-k the call trace has more ? frames, so that tail does not reach sysrq_handle_crash. The full trace is in the device log and in /persist/kcrashes.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

TestKernelPanicReport results:

image EVE-k (kubevirt) kvm
48ab50f2 (this PR) FAIL PASS
48ab50f2 + #6839 PASS (800 s) PASS (511 s)

The EVE-k failure was newlogd dropping the device logs after the first 1000 of the boot until vector was up, which took the panic stack with it while lastRebootReason arrived correctly; #6839 fixes that. With it, both hypervisors report kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash through the efi_pstore backend, save the dump to /persist/kcrashes/pstore-<time>.log, and upload the stack (435 and 358 kernel_panic_stacks lines).

@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.10345% with 78 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.63%. Comparing base (a407274) to head (87ea903).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/cmd/nodeagent/pstore.go 82.40% 29 Missing and 15 partials ⚠️
pkg/pillar/utils/file/file.go 0.00% 21 Missing ⚠️
pkg/pillar/agentlog/agentlog.go 0.00% 10 Missing ⚠️
pkg/pillar/cmd/nodeagent/interfaces.go 50.00% 2 Missing ⚠️
pkg/pillar/cmd/nodeagent/nodeagent.go 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6822      +/-   ##
==========================================
+ Coverage   28.18%   28.63%   +0.45%     
==========================================
  Files         535      547      +12     
  Lines       97789    99824    +2035     
==========================================
+ Hits        27562    28588    +1026     
- Misses      67986    68785     +799     
- Partials     2241     2451     +210     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

TestKernelPanicReport on EVE-k with 8bffe1db7 (boot reason kept on a panic during reboot) + #6839, image 0.0.0-pstore-newlogd-test-6a9465b7: PASS (602 s). lastRebootReason is kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash through efi_pstore, the dump is saved to /persist/kcrashes/pstore-<time>.log, and 371 kernel_panic_stacks lines were uploaded. kvm was not rerun on this commit.

@eriknordmark
eriknordmark marked this pull request as ready for review October 18, 2026 08:16

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rene

rene commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 3m 25s —— View job


Review complete

I reviewed the new pstore.go reader, the agentlog/file.go/nodeagent supporting changes, the removal of kdump/kexec, and the shell-script changes.

  • Read PR diff and changed files
  • Review pstore.go (reader, rotation, reboot-reason logic)
  • Review file.go, agentlog.go, nodeagent.go changes
  • Review shell scripts (000-kexec, storage-init, storage-resize)
  • Post inline comments & summary

Findings: one edge-case correctness issue, posted inline on pstore.go — saveKcrashFiles's newest-first sort is a no-op for timeless pstore backends, so rotation can keep the oldest dumps and erase the newest (deciding) dump without persisting it.

Verified as OK:

  • dom0-ztools image hash is bumped consistently across the two packages that reference it (pillar, vtpm), so check-docker-hashes-consistency should still pass.
  • KEXEC_TAG/KDUMP_TAG are removed consistently from rootfs.yml.in and parse-pkgs.sh.
  • handlePstoreRecords runs before handleLastRebootReason (so restartCounterFile mtime still marks the previous boot), erases records only after their copy is fsynced, and the stored boot reason is preserved for a panic-during-reboot — matching the PR description.
  • WriteKernelRebootReason writes reboot-reason last so its presence implies the stack/boot-reason are complete.

Overall the change is well-structured and thoroughly tested. The rotation finding is narrow (requires a backend that keeps no per-record time and more than 10 accumulated dumps) and does not block the common amd64 efi-pstore/ERST path.
· branch pstore-panic-records

Comment thread pkg/pillar/cmd/nodeagent/pstore.go Resolved
@github-actions
github-actions Bot requested a review from rene October 18, 2026 21:08
@eriknordmark

Copy link
Copy Markdown
Contributor Author

Hardware test on a Supermicro E300 (ERST pstore backend, ZFS /persist), EVE-k built from this PR + #6839 with the core kernel (0.0.0-pstore-newlogd-k-core-a2bd35ae-k-amd64, test-only branch pstore-newlogd-k-core).

Test Saved to /persist/kcrashes Erased from pstore Logged Reboot reason Boot reason
Nine stale ERST records (Oops#1 + Panic#2 written in the same second, twice, plus one Oops) ✓ ✓ ✓ marked stale unchanged (baseos-update) unchanged
echo c > /proc/sysrq-trigger ✓ one 53 KB dump ✓ ✓ 687 [kernel panic #N] lines kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash BootReasonKernel
Controller reboot, panic 28 s after AllDomainsHalted (2 s before nodeagent's reboot) ✓ ✓ ✓ 683 lines kernel panic after NORMAL: controller reboot at … BootReasonRebootCmd (kept)

The first test found that same-second Oops/Panic records were split into one dump per record; the grouping now keeps each dump together. That fix, a same-second comparison between the panic and the stored reboot reason (ERST times records to the second), and the rotation fix from the review comment are squashed into the nodeagent commit, each with a unit test that fails on the previous code.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

TestKernelPanicReport on the current head (2d79ea1ba) plus #6839, images 0.0.0-pstore-newlogd-k-core-a2bd35ae (EVE-k with the core kernel):

Hypervisor pstore backend Reboot reason Dump in /persist/kcrashes kernel_panic_stacks lines in newlog Result
kvm efi_pstore kernel panic, pstore: Kernel panic - not syncing: sysrq triggered crash pstore-2026-10-19T17:22:14Z.log 358 PASS (495 s)
kubevirt efi_pstore same pstore-2026-10-19T17:34:20Z.log 400 PASS (579 s)

eriknordmark and others added 3 commits October 24, 2026 10:42
A kernel panic reaches EVE today only through kdump, which needs a
crash kernel and its memory reservation, exists only on amd64, and
fails on 2-16 GB devices because 128 MB is too little for it. When no
crash kernel is loaded the panicking kernel writes the tail of its log
to pstore (efi-pstore or ERST) instead.

On the next boot nodeagent now reads those records before it handles
the reboot reason. It reassembles each dump, logs it the way a Go
fatal stack trace is logged, with the kernel log lines on a
kernel_panic_stacks log object, and saves it under /persist/kcrashes.
The newest panic since the previous boot becomes the reboot reason,
BootReasonKernel and the reboot stack, unless a reboot reason stored
after the panic or by kdump takes precedence. A panic while EVE was
already rebooting, for example in a driver after the watchdog fired,
is reported as "kernel panic after <stored reason>" and keeps the boot
reason EVE stored, since zedagent picks the configuration to start
from by it. Records older than the previous boot are saved and logged
but do not set the reason. A record is erased only after its copy on
/persist has been fsynced, so a reboot at any point loses nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
With nodeagent reading panic records from pstore, the crash kernel is
no longer needed. Dropping crashkernel= returns the 128-512 MB it
reserved on amd64 to EVE and applications, and removing the kexec
load lets the panicking kernel write its pstore record, which it skips
while a crash kernel is loaded.

The kdump and kexec packages go, 000-kexec only disables kexec load
when the kernel offers it, and storage-init no longer checks for a
crash kernel. KERNEL-DUMPS.md now describes the pstore records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Add TestKernelPanicReport to the telemetry suite. It panics an
onboarded device with sysrq and checks that, after the reboot, the
controller reports BOOT_REASON_KERNEL with the panic as the reboot
reason and the tail of the kernel log as the reboot stack, that the
dump is logged on kernel_panic_stacks log objects and reaches the
controller without a second [dmesg #N] copy, that it is saved in
/persist/kcrashes and /persist/newlog, and that pstore is left empty.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rene
rene merged commit fb9772b into lf-edge:master Oct 25, 2026
59 checks passed
@eriknordmark
eriknordmark deleted the pstore-panic-records branch October 27, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants