Repository navigation
Boot-disk repartition for in-field EVE-kvm → EVE-k conversion - #6063
Conversation
6858d0a to
90fa14a
Compare
053e426 to
36977ab
Compare
643a8f4 to
5dbc563
Compare
42fd8e5 to
58e45a6
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6063 +/- ##
==========================================
+ Coverage 26.82% 27.12% +0.29%
==========================================
Files 521 533 +12
Lines 95420 97453 +2033
==========================================
+ Hits 25598 26434 +836
- Misses 67814 68824 +1010
- Partials 2008 2195 +187 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
55c10f6 to
5f6ec99
Compare
b25f0c5 to
ff02428
Compare
Picks up the four new commits on lf-edge#6063 through the fork#6 chain and adds lf-edge#6441. Notes that lf-edge#6440 is an issue rather than a PR, and that lf-edge#6441 states it is not the cure for it. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Picks up the four new commits on lf-edge#6063 through the fork#7 chain and adds lf-edge#6441. Restates that this line only exercises the fault gates when the build passes FAULT_INJECTION, and where the rootfs cap and the storage-resizer pin come from. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Picks up the four new commits on lf-edge#6063 through the fork#6 chain and adds lf-edge#6441. Notes that lf-edge#6440 is an issue rather than a PR, and that lf-edge#6441 states it is not the cure for it. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Picks up lf-edge#6063's docs-only commit through the chain. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Picks up the four new commits on lf-edge#6063 through the fork#7 chain and adds lf-edge#6441. Restates that this line only exercises the fault gates when the build passes FAULT_INJECTION, and where the rootfs cap and the storage-resizer pin come from. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Picks up lf-edge#6063's docs-only commit through the chain. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Picks up lf-edge#6063's docs-only commit through the chain. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
andrewd-zededa
left a comment
There was a problem hiding this comment.
A couple of quick notes
|
Note that yetus incorrectly fails due to an extra line in the output from this PR adding pkg/storage-resizer. |
|
Rebased on master. Seven of the commits in the earlier series have since landed upstream and dropped out, so this is now the ten commits from The yetus |
|
Claude finished @rene's task in 3m 49s —— View job Review — Boot-disk repartition for EVE-kvm → EVE-k conversion
I focused on this PR's own commits ( One finding, posted inline on
Everything else I checked reads as sound and carefully fail-safe:
Note: the manual mirrors of |
| return nil // describes the backup; not a /persist file | ||
| } | ||
| dst := filepath.Join(persist, rel) | ||
| need, err := needsRestore(rel, dst) |
There was a problem hiding this comment.
The digest comparison computed just above (verifyPersistAgainstManifest) is only reported, never fed into the restore decision. restorePersistFiles relies solely on needsRestore, whose per-type validators can pass a partially-copied file: a *.pem that is a mix of old/new bytes but still contains -----END, or a truncated-then-garbage JSON that still happens to parse, returns needsRestore == false and the corrupt live copy is kept — even though the manifest already flagged it as a mismatch.
For the non-volatile identity files (certs/*.pem, .fscrypt/*, OnboardingStatus, VaultConfig), which never legitimately change between backup and this restore, a digest mismatch is unambiguous corruption and the backup is the known-good copy — so restoring on mismatch is always safe and is exactly what the backup exists for. As written, the strongest corruption signal is discarded precisely for the files that matter most.
Consider having restorePersistFiles also restore a file when its digest mismatches the manifest and it is not in volatileBackupPatterns. (In practice the resize2fs flush-before-repoint property makes such partial copies rare, hence the clean fault-injection runs, but the guarantee only holds "as far as the storage stack honors the flush" — the digest was added for exactly the case where it doesn't.)
Mirror the eve-api ZDeviceState CONVERTING value and the ZDeviceSubState enum as types.DeviceState and types.DeviceSubState, with four sub-states: unspecified, rebooting-to-resize, installing, and rebooting-to-target. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Standalone Go binary doing the disk work (check/backup/shrink/grow/ restore/cleanup) via diskfs/partitionresizer, plus a run-watchdog subcommand that holds/pets /dev/watchdog across a long offline op. The --no-pet/--attempt escalating fault-injection mode is implemented but only used by the separate stress-test PR; the product pets. The grow also creates the reserved second EFI System partition (ESP-B) so a converted device reaches the same layout as a fresh EVE-k install. The pre-flight reads the partition table with go-diskfs, the same library the resize itself drives, so the check and the resize cannot disagree about the layout. It refuses a disk whose primary GPT failed CRC validation and was recovered from the backup copy at end-of-disk: the kernel and the bootloader run off the primary, so a plan built from a newer backup would describe a layout the running system is not using, and a primary that fails CRC also means an earlier partition-table commit was interrupted, which the resize step has to finish before its result can be checked. The ext4 superblock is parsed directly so no filesystem has to be mounted. The /config backup protects the identity and connectivity files against a shrink that wipes /persist, but a file left at its original size and structurally valid while holding a partially-copied mix of old and new bytes passes the present/non-empty/PEM/JSON checks. Whether an interrupted relocation can actually produce such a file is not settled: resize2fs flushes each relocated extent before it repoints any inode, but on real hardware that ordering is only as good as the kernel and controller buffering beneath it. So backup records the SHA-256 of each copy while /persist is still intact, and restore re-hashes the live copies and reports the comparison rather than repairing from it -- the saved config and the DPC list can legitimately differ by then, and overwriting one with the stale backup would lose the ssh keys and network config it carries. Which files to restore stays with the per-type validators. The backed-up set includes the vault key mode. An interrupted shrink can leave /persist/status torn, and the e2fsck that repairs it clears the directory's entries wholesale; with status/vaultmgr/VaultConfig gone, vaultmgr infers the key-derivation mode from whether /persist/vault exists, which on an already-encrypted vault always selects the merged form. A device that created its vault TPM-key-only then derives a key fscrypt refuses; the TPM unseal still succeeds, so the failure reads as a broken seal, and it recurs on every later boot because the inference has nothing new to go on -- the device parks in maintenance mode with no way back. Nothing else on the device can reconstruct the mode. The same tear takes the fscrypt metadata with it, and that loss is worse. /persist/.fscrypt/policies holds the vault's master key, random at encrypt time and stored only there, wrapped by a protector key that lives only in /persist/.fscrypt/protectors and is itself wrapped by the TPM key. Nothing can re-derive a lost policy key, so an orphaned directory does not lock the vault, it destroys it -- fscrypt's only way to replace the policy is to mint a new, empty one. Both directories are in the set. Their contents are opaque protobuf, so the restore gate repairs a copy only when it is missing or zero length; a non-empty but torn one is left in place and surfaces as a digest mismatch, like the rest of the set. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Execs the storage-resizer and selects the next conversion step from the pre-flight's verdict, so the agents driving the conversion deal in decisions rather than in subcommands. Unit-tested behind a fake Runner. The accompanying doc records the resizer's per-subcommand contract -- what each step reads, writes and reports, including the pre-shrink digest manifest and the digest comparison restore prints -- so the orchestration and the binary can be changed against a written interface rather than by reading each other. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On a cross-flavor (kvm<->k) base-OS update, run the read-only pre-flight and proceed / grow / shrink / decline accordingly, reboot into the offline shrink when needed, and report ZDEVICE_STATE_CONVERTING with a sub-state. The conversion is driven from the activation path rather than the download, so a cross-flavor image the controller merely pre-stages leaves the boot disk alone and the repartition -- with the /persist shrink it may entail -- follows the request to switch flavor. The download still precedes the repartition, which the offline resize needs because it runs with no network. A unit test covers the download-only cross-flavor case. Every path that abandons a conversion goes through one helper that clears the converting state, records the reason on BaseOsStatus, publishes it and tells the caller the install must not proceed, so that ordering is stated in a single place. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Handle the baseosmgr-requested reboot into the early-boot offline shrink/grow during a kvm<->k conversion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Report ZDEVICE_STATE_CONVERTING and the conversion sub-state to the controller. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Run the offline shrink/grow and post-mount restore/cleanup under storage-init (re-syncing /config so PCR14 stays stable), feed (pet) the hardware watchdog across it, and raise the resize reboot budget. Wire the eve-storage-resizer image into pillar's and storage-init's Dockerfiles. The reboot that applies a committed GPT, and the one that recovers after a clean abort, reset the device with a magic-sysrq write after a sync. A graceful reboot cannot do it: the onboot container has its own PID namespace, so busybox reboot signals that namespace's init instead of the host, and every resize reboot in the fault-injection soak runs was carried out by the sysrq write. Both reboot forms are kept as fallbacks in case sysrq is unavailable, and storage-init halts rather than proceeding with a mid-resize disk if none of them takes effect. The statically linked resizer adds about 1.5MB to the non-k rootfs, which the 291MB early-warning check accommodates. The constraint that check guards is the 300MB rootfs partition on devices installed before 10.2.0, which the image still clears by 9MB. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Document the in-field boot-disk repartition end to end: what triggers it -- a cross-flavor base-OS update the controller has also asked to activate, so a pre-staged download leaves the disk alone -- the read-only pre-flight and its proceed / grow / shrink / insufficient verdicts, the online grow and the offline shrink under storage-init, and the fail-safes the no-brick claim rests on, including the magic-sysrq reset the offline pass ends in because a graceful reboot from an onboot container with its own PID namespace does nothing. It also records what the design assumes of go-diskfs and partitionresizer, so a reader can tell which guarantees are the libraries' and which are EVE's own. On a ZFS device it states what the conversion does to the vault: the move to the EVE-k zvol layout, that a pool too small for a second copy declines the conversion, and that a converted vault ends up smaller than a fresh EVE-k install's -- what an operator needs before converting, with the mechanism left to the vaultmgr doc. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When a watchdog reset interrupts the offline resize2fs shrink of /persist, the next boot's e2fsck reconnects the orphaned subtrees it repairs into /persist/lost+found. That rescued data is no longer owned by any agent (newlogd, the image CAS GC, ...), so nothing ever reclaims it, yet it keeps consuming P3 space and counts against the low-disk threshold. Report the directory's usage so the space is attributable to the controller and a nonzero value flags that a filesystem repair occurred, and add it to the boot-time low-disk cleanup list. It is placed last, since it may hold the only rescued copy of the data, so an emergency low-disk boot reclaims it only after the known-disposable directories did not free enough. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Eight comments across baseosmgr, nodeagent, zedagent, diskconvert and types describe the boot-disk conversion as EVE-kvm <-> EVE-k. Only the kvm to EVE-k direction reaches the repartition: the cross-flavor gate refuses any switch away from EVE-k outright, since a vault migrated to the EVE-k zvol layout has no path back. Naming both directions also makes the volume condition read as if it applied to both, when it qualifies the kvm to EVE-k direction alone. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Boot-disk repartition for in-field EVE-kvm → EVE-k conversion
Long-lived devices installed with EVE's old small boot-disk layout (IMGA/IMGB
as small as 300 MB, a small ESP) cannot hold the EVE-k (Kubernetes) rootfs and
its container/k3s tooling. Upgrading replaces rootfs contents but never
repartitions, so the small geometry persists. This branch repartitions the boot
disk in place to the larger EVE-k geometry at the moment the controller pushes a
cross-flavor base-OS update — growing ESP/IMGA/IMGB, and first shrinking the
/persistfilesystem when that's the only way to free the space — so anin-field device can be converted to EVE-k without re-imaging and without losing
/persistdata or device identity.A read-only pre-flight classifies the disk into
proceed/grow/shrink/insufficient.growhappens online;shrinkbacks up the device identity tothe CONFIG partition and reboots into an offline shrink+grow under storage-init;
insufficientdeclines the conversion cleanly and the device stays on itscurrent flavor. The design is idempotent and fail-safe: no failure mode bricks
the device, and even a power loss mid-shrink recovers to a bootable, remotely
manageable state.
Interrupting the shrink has not corrupted an app volume in fault injection.
About 2200 conversions across the evetest suite, earlier builds of this code, and
a dedicated volume-integrity soak drove more than 5500 watchdog resets into the
shrink, including roughly 500 conversions whose data volume was verified block by
block.
/persistmetadata is less robust. In 35 of those conversions, a resetduring a relocation left an extent tree or a directory that e2fsck had to repair.
Nearly all hit filler or runtime directories. The rest are why the conversion
backs the device-critical files up to CONFIG and checks them by digest
afterwards: in one run fsck orphaned four of them and they were restored from
/config, and in another a damaged superblock made storage-init recreate/persist, with the device keeping its identity. Two runs on earlier builds lostvault state this way; that state is now in the backup set. All of this is
QEMU-based. Buffering in the kernel and in the storage controller can acknowledge
a flush before the bytes reach the medium, and that has not been exercised on
real hardware.
See
docs/STORAGE-RESIZING.mdfor the full design.What's here
pkg/storage-resizer— standalone Go binary doing all the disk work(check/backup/shrink/grow/restore/cleanup) via
diskfs/partitionresizer, keptout of the pillar module.
storage-init— runs the offline shrink/grow and post-mountrestore/cleanup, re-syncing
/configso PCR14 stays stable.pkg/pillar/diskconvert— pillar orchestration that execs the resizer andselects the next step; unit-tested behind a fake Runner.
seam, reboot into the offline shrink when needed, and report
ZDEVICE_STATE_CONVERTINGwith a sub-state./persistacrossthe kvm→k namespace change so pre-staged content isn't re-downloaded; gate the
cross-flavor block on the actual volume set.
to the EVE-k zvol+ext4 layout by staging a copy and swapping it into place,
with the interrupted swap recovered on the next boot.
In-flight items before merge
never reaches the kernel's or the storage controller's buffering), arm64, and
the evetest: kvm→k boot-disk conversion tests + volverify app #6267 tests against this branch alone rather than an integration branch.
cross-flavor image; a download-only pre-stage leaves the boot disk alone. That
is covered by unit tests and code inspection only.
PR dependencies
None.
How to test and validate this PR
Unit:
pkg/pillar/diskconvertorchestration tests (fake Runner) and thepkg/storage-resizercheck/shrink/grow logic.E2E: the evetest
konvertsuite, run against a build carrying this PR, on adevice installed with the old small boot-disk geometry. In master:
TestKvmToKRepartition(EXPECT_DECISION=shrink|grow),TestKvmToKRepartitionGeometry,TestKvmToKRepartitionRefused(
REFUSE_REASON=zfs|too-full),TestPersistWipeRestoreandTestBackupCorruptRestore. Added by evetest: kvm→k boot-disk conversion tests + volverify app #6267:TestKvmToKRepartitionAppVolume(volume data across a watchdog-interrupted shrink),
TestKvmToKZFSVaultMigration(pool too tight for the vault copy) andTestKvmToKZFSVaultPowerCutMidSwap; evetest: kvm→k boot-disk conversion tests + volverify app #6267 also renamesTestKvmToKRepartitiontoTestKvmToKRepartitionNoVolmig.The suite runs continuously in fault-injected soaks on integration builds
carrying this PR. As of 2026-09-27 it has passed 782 of 827 runs across two
hosts. Of the 45 failures:
found: the migration key not staged (18), and EVE-kvm not recovering a
migration cut mid-swap, so it made a fresh vault and key over it (1).
conversion, an outdated power-cut assertion, and an upgrade wait that gave
up on a download error the device had already scheduled a retry for.
outage at the test site. One of those, a first-boot timeout, is attributed
by timing only; its device log was not kept.
Changelog notes
Adds in-field boot-disk repartitioning so a device installed with EVE's old
small boot-disk layout can be converted between EVE-kvm and EVE-k on a
cross-flavor base-OS update, without re-imaging and without losing /persist
data or device identity. Requires that there are no app instances or volume instances
on the device.
On a ZFS device the conversion also moves the vault from a filesystem dataset to
the EVE-k zvol layout. The vault is sized from the pool space that is free at
conversion time, so a converted device ends up with a smaller vault than a fresh
EVE-k install of the same hardware, and the space the old vault frees returns to
the pool instead of growing the vault. A device whose pool cannot hold a second
copy of its vault declines the conversion and stays on its current flavor.
PR Backports
Checklist
And the last but not least:
check them.
🤖 Generated with Claude Code