Repository navigation
evetest: EVE-kvm to EVE-K conversion test suite - #6539
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6539 +/- ##
==========================================
+ Coverage 26.15% 26.46% +0.31%
==========================================
Files 514 524 +10
Lines 94703 96419 +1716
==========================================
+ Hits 24766 25520 +754
- Misses 68026 68804 +778
- Partials 1911 2095 +184 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
See my comments in #6540, apparently they belong this this PR |
There was a problem hiding this comment.
evetest framework changes LGTM. I didn't check the actual tests in detail because its a lot.
Just note my comments in #6540:
milan4zededa/evetest-ubuntu-ctrshould belfedge/evetest-ubuntu-ctrCAPABILITY_EDIT_DEVICE_DISKis not going to be available in the github CI (we will use proxmox provider) -- so all those tests requiring this will be skipped. Not sure mow much this hurt us and if there is a potential solution for proxmox (have Claude check proxmox API)- considering the API and other evetest framework changes, I suggest to increase the
evetest/VERSION
| DefaultValue: defaultInitialEVEVersion, | ||
| Description: evetest.TestParameterDescription{ | ||
| Summary: "Released EVE version the device boots before the conversion", | ||
| Default: "12.1.0", |
There was a problem hiding this comment.
Should we pick some LTS version instead like 13.4?
There was a problem hiding this comment.
Switched to 13.4.3-lts. Any release predating the boot-disk repartition works — mkimage-raw-efi/make-raw lays the boot disk out identically at 12.1.0 and 13.4.3-lts (36 MB EFI, 512 MB rootfs floor, 1 MB config) — and an LTS keeps the baseline available. 14.5 is the first to raise the rootfs floor conditionally, so I stopped short of it. The suite has not been re-run on the new default yet.
There was a problem hiding this comment.
Corrected: 10.1.0, not an LTS after all.
assertSmallGeometry needs an ESP under 256 MiB and IMGA/IMGB under 1 GiB, and releases
do not lay the boot disk out the same way. In mkimage-raw-efi/make-raw the rootfs
partition is a fixed 300 MiB up to 10.1.0 and a fixed 512 MiB at 10.2.0; 12.1.0 turned it
into a floor the actual rootfs may exceed (SIZE=$((SIZE > ROOTFS_PART_SIZE_MIN ? SIZE : ROOTFS_PART_SIZE_MIN))), and 14.5 raises that floor to 1 GiB.
So an LTS is the wrong axis to pick on: 13.4.3-lts does fit the window today, but only as a
floor, so its baseline geometry can drift with rootfs content. 10.1.0 is the smallest layout
inside the window and the last one whose size is fixed, which makes the baseline
deterministic. It is a published release, so availability is not a concern.
@eriknordmark We do support all this, see TestVCom. |
Do we plan to eventually backport the conversion work to 17.0? |
2640e07 to
87d2177
Compare
50e7b9b to
0f9e508
Compare
Right, my mistake — Your #6540 comments are addressed here: the On the initial version: switched to |
|
Yetus is green — the two findings were an unchecked The red |
@milan-zededa No, that doesn't make sense. 18.0 is it. |
0f9e508 to
f62a27a
Compare
Four additions the EVE-kvm/EVE-K conversion tests need and the framework did not have. A ContentTree can now be declared on its own, with no Volume referencing it. A test whose subject is the image blobs themselves -- that they survive an event, or that a later deployment reuses them -- wants them on the device without a PVC or a CDI import in the picture, which going through AddVolume would drag in. A powered-off device's boot disk can be edited in place: grown, leaving the added space unallocated past the last partition, or made to lose the filesystem on one of its partitions. The first is the precondition for EVE's in-field repartition to grow into slack rather than shrink /persist, and cannot be arranged at creation time because EVE's image generator sizes partitions to fill whatever disk it is given; the second is the data loss a backup-and-restore path has to survive. Both rewrite the file the device boots from, so the broker requires the device to be powered off rather than letting qemu-img discover it, and advertises CAPABILITY_EDIT_DEVICE_DISK only for the overlay disk strategy, where it still owns that file. RAM and vCPU count join the shared test parameters, since sizing a device is not particular to these tests. Finally, a base-OS update can be applied that EVE is expected to reject. A rejection is not a failed upgrade: EVE refuses in place, before it writes a partition and before it reboots, so there is nothing to wait for and no reboot to account for -- and the reboot audit is an equality check, so booking one that cannot happen fails the run at teardown. The expected outcome is now explicit inside the upgrade path, and UpgradeEVE's own signature is unchanged. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
A base-OS update that changes hypervisor flavor is normally refused: /persist is laid out differently for EVE-K, so baseosmgr blocks the switch rather than meet a layout it cannot read. Allowing it while the device holds no volume is the first step towards the in-field conversion, and this is the happy path of it -- both images released, no repartition, the flavor the only variable. The reverse is then pushed and must be refused. EVE-K is a one-way door: docs/EVE-K.md states an upgrade from HV=k to another HV type is not supported, and baseosmgr blocks it unconditionally, with none of the conditions that let the forward direction through. Asserting that keeps a future relaxation from going unnoticed. Carries the suite and the helpers the rest of the conversion tests build on, which is why it is larger than the tests that follow it. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Image blobs already on the device must outlive the flavor switch, and a deployment afterwards must reuse them rather than download again. Staged as a bare ContentTree rather than a volume so the answer is about the blobs alone. A volume would pull in the PVC and CDI machinery EVE-K stands up behind it, and a failure there would look identical to the blobs having been lost -- a different and much less interesting answer. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Deleting an application before the switch and redeploying it after is the recipe that works today: carrying a live app through depends on machinery a single-node EVE-K cannot yet schedule, while delete-and-redeploy needs only the cached blobs to outlive the gap. They do, but only because the deferred content delete is stretched past the length of the gap first. Otherwise EVE reclaims them as soon as the app that referenced them is gone, and the redeploy quietly pays for a full download -- which would still pass a test that only checked the app came back. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Drives the conversion the way a fielded device reaches it: a released small-geometry EVE-kvm image, a same-flavor hop that lands the conversion code without moving the disk, and only then the flavor change whose cross-flavor seam arms the offline resize. Going straight to EVE-K would exercise a path no device in the field can take. The route to the space is the axis. With a full boot disk /persist must shrink, which is the field layout and the harder path, since resize2fs relocates live blocks; with an unallocated tail the partitions grow into it and /persist is untouched. Both routes end at the same layout, so the route is asserted before the conversion from storage-resizer's own pre-flight decision, and confirmed after by what became of /persist. The shrink variant fills /persist first, so there are real blocks to move rather than free space to discard. Whether the repartition cost the TPM seal is judged from the device's own logs rather than live: the window in which the post-conversion unlock is readable is a minute or two, and /persist carries the record across the reboots anyway. The last boot on the conversion-capable image must have unsealed locally, and no re-seal anywhere may name PCR5 -- the measurement the repartition moves, and the one the seal deliberately excludes. The EVE-K boot falling back to the controller key is expected, since a new rootfs moves other PCRs. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
A conversion that fails halfway leaves a device with a half-written partition table and no way back, so EVE has to work out beforehand that it cannot free the space, refuse, say why, and carry on unchanged. Two ways to get there, failing at different points in the resizer's reasoning: a ZFS /persist cannot be shrunk at all, so with no free tail a shrink is never considered; an ext4 /persist filled past what it could give back is considered and does not fit. The reason is asserted, not just the refusal -- both end at the same verdict, so a refusal-only check would pass on a device that declined for a third reason, and would keep passing once the condition it meant to create had stopped happening. The decline is three assertions that hold together, each ruling out a different unacceptable way to refuse: an error is reported, so it is a decision rather than a silent stall; the device still runs the image it did, so nothing was installed; and it still answers its controller, since losing remote management is the outcome refusing exists to avoid. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Asserts only that the boot disk ends at the full EVE-K layout, whatever it started on, and deliberately deploys nothing: mixing an application in would let a layout regression be masked by, or mistaken for, a storage failure. The target is stated absolutely rather than relative to the starting sizes. A conversion that grew every partition but stopped short would satisfy a relative check and still leave a disk unlike a fresh EVE-K install. The starting layout is observed rather than declared, and only has to differ from the target -- a device already laid out correctly would pass without the conversion having done anything. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
The case the other conversion tests avoid. They delete the application first, because the cross-flavor gate refuses to convert while a volume exists; here the device already has the EVE-K layout, so no repartition is needed, the gate allows the conversion with volumes present, and the volume must be carried over rather than recreated. The marker is read back off the volume's raw block device rather than from where it was written. On EVE-K a container's data volume is not auto-mounted at its MountDir (lf-edge#6145), so a mount-based check would report the data lost when it is merely unmounted -- the exact false failure this test exists to tell apart from a real one. The escript this is ported from also carries a VM application's disk volume across. evetest has no VM application image and no cloud-init plumbing to reach a VM guest, so that half needs a new test app and UserData support first. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
EVE-K cannot create a volume until k3s, Longhorn and CDI are up, roughly a quarter of an hour from a cold boot. A device handed an application in its very first configuration therefore asks for a volume that cannot be satisfied yet, and volumemgr has to defer the request and retry it when storage arrives, with creation idempotent so a retry does not trip over its own earlier attempt. Without that the volume sits at CREATING_VOLUME and the application never starts. The timing is the test: the application is deployed immediately after onboarding, not after waiting for storage, which would test nothing because by then the request is one that can simply be granted. Everything else -- a native EVE-K boot, no conversion, no prior volumes -- exists to make that one window real. A device that reboots its way out of the problem has not solved it either. Nothing here asks for a reboot, and the framework audits observed reboots against expected ones, so that case fails without a separate assertion. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Before the conversion repartitions a boot disk it copies the files a device cannot be itself without -- its UUID, its certificates, its last configuration -- onto the CONFIG partition, and restores them at early boot. These two tests exercise that fail-safe on its own, with no resize: /persist is destroyed outright in one, and in the other each critical file is truncated to half its length. Partial corruption is the harder case. An empty /persist is unambiguous; a /persist full of half-written files is not, and the restore has to judge each file on whether it is still valid for its own type, while files it has no validator for fall back to the copies pillar keeps beside them. Restoring too little strands the device, restoring too much discards state it did not need to. The load-bearing assertion in both is not that files reappeared but that a controller-encrypted credential is decrypted from the restored key: files can come back empty or mismatched and satisfy every other check. It has to happen offline, or the device is simply told everything again, so the controller is cut off with an SDN firewall rule rather than by stopping it -- that isolates one device and leaves the harness working. Restoring the path afterwards waits for the device to check in, since reboots are counted from what it reports and the audit runs at teardown. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
f62a27a to
8f4ac5a
Compare
|
Rebased onto master One fix came out of the re-run. The three flavor-switch tests took their Worth flagging for anyone reproducing: |
8f4ac5a to
0d9656c
Compare
Description
Ports the EVE-kvm → EVE-K conversion tests from lf-edge/eden#1209 to evetest.
They form a suite of their own rather than joining
TestUpgradeSuite, because aconversion leg is not a single upgrade: reaching the repartition takes a
released small-geometry kvm image, a kvm→kvm hop to land the conversion code
without moving the geometry, and only then the kvm→k hop, with an offline resize
and several reboots in between. Folding that in would make a suite worth gating
on as expensive as the scenario it is not about.
TestKvmToKUpgradeTestKvmToKContentTreeTestKvmToKAppRecreateTestKvmToKRepartitionShrinkandGrowTestKvmToKRepartitionRefusedPersistTooFullandZFSPersistTestKvmToKRepartitionGeometryTestKvmToKVolumeMigrationTestFirstBootEVEKAppVolumeTestPersistWipeRestore/persistis destroyedTestBackupCorruptRestore/persistis partly corrupted rather than lostThe first commit carries the framework additions; each later commit adds one
test. The additions are:
AddContentTree/DeleteContentTree), so a blob testdoes not drag in a volume and, on EVE-K, PVC/CDI with it.
CAPABILITY_EDIT_DEVICE_DISK: grow a disk leaving the added spaceunallocated, and destroy a partition's filesystem. Growing relocates the
backup GPT, without which the added space is not usable and the device has
been observed rebooting under its own watchdog.
RAM_SIZE_MBandCPUStest parameters.declined rather than wait out an upgrade that will never happen.
Device sizing matches the eden escripts these come from: 4 vCPUs and 8 GiB of
RAM, eden's own defaults, which those escripts never override. Worth noting
because it is evidence that the 16 GiB / 8 vCPU floor used elsewhere for EVE-K
may be unnecessary — a full EVE-K cluster with Longhorn converges at eden's
sizing here. Raising it would also stop these tests covering the envelope the
escripts cover, so the floors move only through
RAM_SIZE_MB/CPUS.Not ported from eden#1209: the VM-application half of volume migration,
POST_REBOOT_CHECK,RELOCATE_CRITICAL_HIGH, and thetwodisk-ext4/twodisk-zfstopologies. None of these is blocked by the framework — evetestdrives VM guests (
TestVCom) andEdgeDeviceConfigalready carries cloud-initas an encrypted
ProtectedUserDatablock, and the two-disk topologies areunblocked by the follow-up PR. They are simply not done yet.
PR dependencies
None. Note that every test here needs an EVE build carrying the conversion work
(#6036, #6063): on a stock build baseosmgr refuses a kvm↔k base-OS update
outright, so the cross-flavor hop fails before any of this is exercised.
How to test and validate this PR
Automated. The suite is
TestKonvertSuite, or run a single test:The tests reach the other flavor two different ways, and that decides what the
version axes have to be.
The three flavor-switch tests —
TestKvmToKUpgrade,TestKvmToKContentTree,TestKvmToKAppRecreate— hop between the two flavors of one version, and thatversion defaults to the build under test. It cannot be a release: the
relaxation the hop needs is in no release, so a released source image refuses
the update, and no release publishes both flavors anyway (
lfedge/eve:12.1.0-k-amd64does not exist). Nothing needs setting beyond
EVE_VERSION.The other nine start the device on a released kvm image and convert it to the
build under test.
INITIAL_EVE_VERSIONmust be a release whose layout fitsassertSmallGeometry— ESP under 256 MiB and IMGA/IMGB under 1 GiB — or a runfails at the baseline instead of converting anything. It defaults to 10.1.0,
whose 300 MiB IMGx is a fixed size rather than a floor, so the starting geometry
cannot drift with rootfs content. The results below were taken on 12.1.0 (36 MiB
ESP, 512 MiB IMGx), which is in the same window. The repartition variants take
their route from
EXPECT_DECISION(shrink/grow), and the refusal variantstheir reason from
REFUSE_REASON(too-full/zfs).All twelve pass against
0.0.0-resize-allprs-ef3029b1, one run each, noretries:
What a reviewer should look for when validating by hand, beyond the verdicts:
TestKvmToKRepartition/Shrinklogsdecision=shrink, then the resizer'sshrink complete / grow complete / repartition complete on /dev/vda, then theEVE-K geometry with a reserved ESP-B at partition 7.
TestKvmToKRepartition/Growreaches the same end state without touching/persist, taking its space from the free tail instead.shrinkApplicable=truewith "persist is too full" for ext4,
shrinkApplicable=falsewith "persist isZFS" — and leave the partition table untouched.
off from the controller; a device that can reach its controller is simply
told everything again and would pass having recovered nothing.
Changelog notes
No user-facing changes. Test-only, plus the evetest framework support those
tests need.
PR Backports
Checklist
helper file carries package-level documentation of what it covers and why,
and
testsuite_test.godocuments the suite's shape.device through the evetest broker on an amd64 host.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA