Skip to content

evetest: EVE-kvm to EVE-K conversion test suite - #6539

Merged
eriknordmark merged 10 commits into
lf-edge:masterfrom
eriknordmark:evetest-konvert-tests
Sep 18, 2026
Merged

eriknordmark merged 10 commits into
lf-edge:masterfrom
eriknordmark:evetest-konvert-tests

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

Ports the EVE-kvm → EVE-K conversion tests from lf-edge/eden#1209 to evetest.

They form a suite of their own rather than joining TestUpgradeSuite, because a
conversion leg is not a single upgrade: reaching the repartition takes a
released small-geometry kvm image, a kvm→kvm hop to land the conversion code
without moving the geometry, and only then the kvm→k hop, with an offline resize
and several reboots in between. Folding that in would make a suite worth gating
on as expensive as the scenario it is not about.

Test What it asserts
TestKvmToKUpgrade The cross-flavor base-OS upgrade itself, and that the reverse (k→kvm) is refused
TestKvmToKContentTree Content-tree blobs survive the flavor switch and are not re-downloaded
TestKvmToKAppRecreate An app taken across the switch comes back from the carried-over content store
TestKvmToKRepartition The in-field boot-disk repartition, by both routes to the space: Shrink and Grow
TestKvmToKRepartitionRefused The conversion declined, for each reason: PersistTooFull and ZFSPersist
TestKvmToKRepartitionGeometry The resulting partition layout, on its own
TestKvmToKVolumeMigration An app volume carried across the conversion without the app being recreated
TestFirstBootEVEKAppVolume A volume asked for before EVE-K storage exists
TestPersistWipeRestore Identity recovered offline after /persist is destroyed
TestBackupCorruptRestore The same, when /persist is partly corrupted rather than lost

The first commit carries the framework additions; each later commit adds one
test. The additions are:

  • Bare content trees (AddContentTree / DeleteContentTree), so a blob test
    does not drag in a volume and, on EVE-K, PVC/CDI with it.
  • Offline disk editing over a new broker RPC, behind
    CAPABILITY_EDIT_DEVICE_DISK: grow a disk leaving the added space
    unallocated, and destroy a partition's filesystem. Growing relocates the
    backup GPT, without which the added space is not usable and the device has
    been observed rebooting under its own watchdog.
  • RAM_SIZE_MB and CPUS test parameters.
  • A refused-upgrade outcome, so a test can assert the controller's update is
    declined rather than wait out an upgrade that will never happen.

Device sizing matches the eden escripts these come from: 4 vCPUs and 8 GiB of
RAM, eden's own defaults, which those escripts never override. Worth noting
because it is evidence that the 16 GiB / 8 vCPU floor used elsewhere for EVE-K
may be unnecessary — a full EVE-K cluster with Longhorn converges at eden's
sizing here. Raising it would also stop these tests covering the envelope the
escripts cover, so the floors move only through RAM_SIZE_MB / CPUS.

Not ported from eden#1209: the VM-application half of volume migration,
POST_REBOOT_CHECK, RELOCATE_CRITICAL_HIGH, and the twodisk-ext4 /
twodisk-zfs topologies. None of these is blocked by the framework — evetest
drives VM guests (TestVCom) and EdgeDeviceConfig already carries cloud-init
as an encrypted ProtectedUserData block, and the two-disk topologies are
unblocked by the follow-up PR. They are simply not done yet.

PR dependencies

None. Note that every test here needs an EVE build carrying the conversion work
(#6036, #6063): on a stock build baseosmgr refuses a kvm↔k base-OS update
outright, so the cross-flavor hop fails before any of this is exercised.

How to test and validate this PR

Automated. The suite is TestKonvertSuite, or run a single test:

EVETEST_EVE_VERSION=<build carrying #6036 and #6063> \
make evetest NAME=TestKvmToKUpgrade

The tests reach the other flavor two different ways, and that decides what the
version axes have to be.

The three flavor-switch tests — TestKvmToKUpgrade, TestKvmToKContentTree,
TestKvmToKAppRecreate — hop between the two flavors of one version, and that
version defaults to the build under test. It cannot be a release: the
relaxation the hop needs is in no release, so a released source image refuses
the update, and no release publishes both flavors anyway (lfedge/eve:12.1.0-k-amd64
does not exist). Nothing needs setting beyond EVE_VERSION.

The other nine start the device on a released kvm image and convert it to the
build under test. INITIAL_EVE_VERSION must be a release whose layout fits
assertSmallGeometry — ESP under 256 MiB and IMGA/IMGB under 1 GiB — or a run
fails at the baseline instead of converting anything. It defaults to 10.1.0,
whose 300 MiB IMGx is a fixed size rather than a floor, so the starting geometry
cannot drift with rootfs content. The results below were taken on 12.1.0 (36 MiB
ESP, 512 MiB IMGx), which is in the same window. The repartition variants take
their route from EXPECT_DECISION (shrink / grow), and the refusal variants
their reason from REFUSE_REASON (too-full / zfs).

All twelve pass against 0.0.0-resize-allprs-ef3029b1, one run each, no
retries:

TestKvmToKUpgrade                              PASS   502s
TestKvmToKContentTree                          PASS   707s
TestKvmToKAppRecreate                          PASS   821s
TestKvmToKRepartition/Shrink                   PASS  2336s
TestKvmToKRepartition/Grow                     PASS  1482s
TestKvmToKRepartitionRefused/PersistTooFull    PASS   618s
TestKvmToKRepartitionRefused/ZFSPersist        PASS   344s
TestKvmToKRepartitionGeometry                  PASS   867s
TestKvmToKVolumeMigration                      PASS   750s
TestFirstBootEVEKAppVolume                     PASS   478s
TestPersistWipeRestore                         PASS   449s
TestBackupCorruptRestore                       PASS   432s

What a reviewer should look for when validating by hand, beyond the verdicts:

  • TestKvmToKRepartition/Shrink logs decision=shrink, then the resizer's
    shrink complete / grow complete / repartition complete on /dev/vda, then the
    EVE-K geometry with a reserved ESP-B at partition 7.
  • TestKvmToKRepartition/Grow reaches the same end state without touching
    /persist, taking its space from the free tail instead.
  • The refusal variants must give different reasons — shrinkApplicable=true
    with "persist is too full" for ext4, shrinkApplicable=false with "persist is
    ZFS" — and leave the partition table untouched.
  • The restore tests must decrypt a controller-encrypted credential while cut
    off from the controller
    ; a device that can reach its controller is simply
    told everything again and would pass having recovered nothing.

Changelog notes

No user-facing changes. Test-only, plus the evetest framework support those
tests need.

PR Backports

  • 17.0-stable: No — test-only, and it targets conversion work not present there.
  • 16.0-stable: No, as the feature is not available there.
  • 14.5-stable: No, as the feature is not available there.
  • 13.4-stable: No, as the feature is not available there.

Checklist

  • I've provided a proper description
  • I've added the proper documentation — no separate docs; each test and
    helper file carries package-level documentation of what it covers and why,
    and testsuite_test.go documents the suite's shape.
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device — not run on arm64; these drive a QEMU
    device through the evetest broker on an amd64 host.
  • I've written the test verification instructions
  • I've set the proper labels to this PR

🤖 Generated with Claude Code

https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA

@codecov

codecov Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 26.46%. Comparing base (953778f) to head (0d9656c).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6539      +/-   ##
==========================================
+ Coverage   26.15%   26.46%   +0.31%     
==========================================
  Files         514      524      +10     
  Lines       94703    96419    +1716     
==========================================
+ Hits        24766    25520     +754     
- Misses      68026    68804     +778     
- Partials     1911     2095     +184     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene rene added the testing New test or test framework changes label Sep 14, 2026
@milan-zededa

Copy link
Copy Markdown
Contributor

See my comments in #6540, apparently they belong this this PR

@milan-zededa milan-zededa left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

evetest framework changes LGTM. I didn't check the actual tests in detail because its a lot.
Just note my comments in #6540:

  • milan4zededa/evetest-ubuntu-ctr should be lfedge/evetest-ubuntu-ctr
  • CAPABILITY_EDIT_DEVICE_DISK is not going to be available in the github CI (we will use proxmox provider) -- so all those tests requiring this will be skipped. Not sure mow much this hurt us and if there is a potential solution for proxmox (have Claude check proxmox API)
  • considering the API and other evetest framework changes, I suggest to increase the evetest/VERSION

DefaultValue: defaultInitialEVEVersion,
Description: evetest.TestParameterDescription{
Summary: "Released EVE version the device boots before the conversion",
Default: "12.1.0",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we pick some LTS version instead like 13.4?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to 13.4.3-lts. Any release predating the boot-disk repartition works — mkimage-raw-efi/make-raw lays the boot disk out identically at 12.1.0 and 13.4.3-lts (36 MB EFI, 512 MB rootfs floor, 1 MB config) — and an LTS keeps the baseline available. 14.5 is the first to raise the rootfs floor conditionally, so I stopped short of it. The suite has not been re-run on the new default yet.

@eriknordmark eriknordmark Sep 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected: 10.1.0, not an LTS after all.

assertSmallGeometry needs an ESP under 256 MiB and IMGA/IMGB under 1 GiB, and releases
do not lay the boot disk out the same way. In mkimage-raw-efi/make-raw the rootfs
partition is a fixed 300 MiB up to 10.1.0 and a fixed 512 MiB at 10.2.0; 12.1.0 turned it
into a floor the actual rootfs may exceed (SIZE=$((SIZE > ROOTFS_PART_SIZE_MIN ? SIZE : ROOTFS_PART_SIZE_MIN))), and 14.5 raises that floor to 1 GiB.

So an LTS is the wrong axis to pick on: 13.4.3-lts does fit the window today, but only as a
floor, so its baseline geometry can drift with rootfs content. 10.1.0 is the smallest layout
inside the window and the last one whose size is fixed, which makes the baseline
deterministic. It is a published release, so availability is not a concern.

@milan-zededa

Copy link
Copy Markdown
Contributor

Not ported from eden#1209: the VM-application half of volume migration (evetest
has no VM application image and no cloud-init plumbing to reach a VM guest)

@eriknordmark We do support all this, see TestVCom.

@milan-zededa

Copy link
Copy Markdown
Contributor

17.0-stable: No — test-only, and it targets conversion work not present there

Do we plan to eventually backport the conversion work to 17.0?

@eriknordmark

Copy link
Copy Markdown
Contributor Author

We do support all this, see TestVCom

Right, my mistake — tests/security/vcom_test.go has both halves (Alpine cloud image, UserData cloud-config). Description corrected: the VM-application leg of volume migration is a follow-up, not something the framework is missing.

Your #6540 comments are addressed here: the CreatePersistPartition/DeletePartition commit moved into this PR so the capability and its implementation land together, evetest/VERSION is 1.7, and the test app image is lfedge/evetest-ubuntu-ctr. #6540 is now just the test.

On the initial version: switched to 13.4.3-lts.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

Yetus is green — the two findings were an unchecked os.Remove in a defer (errcheck) and an untagged switch on fields["PARTLABEL"] (staticcheck QF1002).

The red test job is not from this PR. It fails in dpcmanager — this run on TestSingleDPC, TestDPCFallback, TestPNACDHCPReacquireMaxRetries, plus worker TestLength — and master's own test job is red the same way (run 34806366868 on 46356b9, dpcmanager TestLPSConfig). The failing set moves between runs, so that package is flaky on master and every PR inherits it. This PR touches evetest/ only.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

17.0-stable: No — test-only, and it targets conversion work not present there

Do we plan to eventually backport the conversion work to 17.0?

@milan-zededa No, that doesn't make sense. 18.0 is it.

eriknordmark and others added 10 commits September 16, 2026 14:48
Four additions the EVE-kvm/EVE-K conversion tests need and the framework did
not have.

A ContentTree can now be declared on its own, with no Volume referencing it. A
test whose subject is the image blobs themselves -- that they survive an event,
or that a later deployment reuses them -- wants them on the device without a
PVC or a CDI import in the picture, which going through AddVolume would drag in.

A powered-off device's boot disk can be edited in place: grown, leaving the
added space unallocated past the last partition, or made to lose the filesystem
on one of its partitions. The first is the precondition for EVE's in-field
repartition to grow into slack rather than shrink /persist, and cannot be
arranged at creation time because EVE's image generator sizes partitions to
fill whatever disk it is given; the second is the data loss a backup-and-restore
path has to survive. Both rewrite the file the device boots from, so the broker
requires the device to be powered off rather than letting qemu-img discover it,
and advertises CAPABILITY_EDIT_DEVICE_DISK only for the overlay disk strategy,
where it still owns that file.

RAM and vCPU count join the shared test parameters, since sizing a device is
not particular to these tests.

Finally, a base-OS update can be applied that EVE is expected to reject. A
rejection is not a failed upgrade: EVE refuses in place, before it writes a
partition and before it reboots, so there is nothing to wait for and no reboot
to account for -- and the reboot audit is an equality check, so booking one that
cannot happen fails the run at teardown. The expected outcome is now explicit
inside the upgrade path, and UpgradeEVE's own signature is unchanged.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
A base-OS update that changes hypervisor flavor is normally refused: /persist is
laid out differently for EVE-K, so baseosmgr blocks the switch rather than meet a
layout it cannot read. Allowing it while the device holds no volume is the first
step towards the in-field conversion, and this is the happy path of it -- both
images released, no repartition, the flavor the only variable.

The reverse is then pushed and must be refused. EVE-K is a one-way door:
docs/EVE-K.md states an upgrade from HV=k to another HV type is not supported,
and baseosmgr blocks it unconditionally, with none of the conditions that let
the forward direction through. Asserting that keeps a future relaxation from
going unnoticed.

Carries the suite and the helpers the rest of the conversion tests build on,
which is why it is larger than the tests that follow it.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Image blobs already on the device must outlive the flavor switch, and a
deployment afterwards must reuse them rather than download again.

Staged as a bare ContentTree rather than a volume so the answer is about the
blobs alone. A volume would pull in the PVC and CDI machinery EVE-K stands up
behind it, and a failure there would look identical to the blobs having been
lost -- a different and much less interesting answer.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Deleting an application before the switch and redeploying it after is the recipe
that works today: carrying a live app through depends on machinery a single-node
EVE-K cannot yet schedule, while delete-and-redeploy needs only the cached blobs
to outlive the gap.

They do, but only because the deferred content delete is stretched past the
length of the gap first. Otherwise EVE reclaims them as soon as the app that
referenced them is gone, and the redeploy quietly pays for a full download --
which would still pass a test that only checked the app came back.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Drives the conversion the way a fielded device reaches it: a released
small-geometry EVE-kvm image, a same-flavor hop that lands the conversion code
without moving the disk, and only then the flavor change whose cross-flavor seam
arms the offline resize. Going straight to EVE-K would exercise a path no device
in the field can take.

The route to the space is the axis. With a full boot disk /persist must shrink,
which is the field layout and the harder path, since resize2fs relocates live
blocks; with an unallocated tail the partitions grow into it and /persist is
untouched. Both routes end at the same layout, so the route is asserted before
the conversion from storage-resizer's own pre-flight decision, and confirmed
after by what became of /persist. The shrink variant fills /persist first, so
there are real blocks to move rather than free space to discard.

Whether the repartition cost the TPM seal is judged from the device's own logs
rather than live: the window in which the post-conversion unlock is readable is
a minute or two, and /persist carries the record across the reboots anyway. The
last boot on the conversion-capable image must have unsealed locally, and no
re-seal anywhere may name PCR5 -- the measurement the repartition moves, and the
one the seal deliberately excludes. The EVE-K boot falling back to the controller
key is expected, since a new rootfs moves other PCRs.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
A conversion that fails halfway leaves a device with a half-written partition
table and no way back, so EVE has to work out beforehand that it cannot free the
space, refuse, say why, and carry on unchanged.

Two ways to get there, failing at different points in the resizer's reasoning: a
ZFS /persist cannot be shrunk at all, so with no free tail a shrink is never
considered; an ext4 /persist filled past what it could give back is considered
and does not fit. The reason is asserted, not just the refusal -- both end at the
same verdict, so a refusal-only check would pass on a device that declined for a
third reason, and would keep passing once the condition it meant to create had
stopped happening.

The decline is three assertions that hold together, each ruling out a different
unacceptable way to refuse: an error is reported, so it is a decision rather than
a silent stall; the device still runs the image it did, so nothing was installed;
and it still answers its controller, since losing remote management is the
outcome refusing exists to avoid.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Asserts only that the boot disk ends at the full EVE-K layout, whatever it
started on, and deliberately deploys nothing: mixing an application in would let
a layout regression be masked by, or mistaken for, a storage failure.

The target is stated absolutely rather than relative to the starting sizes. A
conversion that grew every partition but stopped short would satisfy a relative
check and still leave a disk unlike a fresh EVE-K install.

The starting layout is observed rather than declared, and only has to differ
from the target -- a device already laid out correctly would pass without the
conversion having done anything.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
The case the other conversion tests avoid. They delete the application first,
because the cross-flavor gate refuses to convert while a volume exists; here the
device already has the EVE-K layout, so no repartition is needed, the gate allows
the conversion with volumes present, and the volume must be carried over rather
than recreated.

The marker is read back off the volume's raw block device rather than from where
it was written. On EVE-K a container's data volume is not auto-mounted at its
MountDir (lf-edge#6145), so a mount-based check would report the data lost
when it is merely unmounted -- the exact false failure this test exists to tell
apart from a real one.

The escript this is ported from also carries a VM application's disk volume
across. evetest has no VM application image and no cloud-init plumbing to reach a
VM guest, so that half needs a new test app and UserData support first.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
EVE-K cannot create a volume until k3s, Longhorn and CDI are up, roughly a
quarter of an hour from a cold boot. A device handed an application in its very
first configuration therefore asks for a volume that cannot be satisfied yet, and
volumemgr has to defer the request and retry it when storage arrives, with
creation idempotent so a retry does not trip over its own earlier attempt.
Without that the volume sits at CREATING_VOLUME and the application never starts.

The timing is the test: the application is deployed immediately after onboarding,
not after waiting for storage, which would test nothing because by then the
request is one that can simply be granted. Everything else -- a native EVE-K boot,
no conversion, no prior volumes -- exists to make that one window real.

A device that reboots its way out of the problem has not solved it either.
Nothing here asks for a reboot, and the framework audits observed reboots against
expected ones, so that case fails without a separate assertion.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Before the conversion repartitions a boot disk it copies the files a device
cannot be itself without -- its UUID, its certificates, its last configuration --
onto the CONFIG partition, and restores them at early boot. These two tests
exercise that fail-safe on its own, with no resize: /persist is destroyed
outright in one, and in the other each critical file is truncated to half its
length.

Partial corruption is the harder case. An empty /persist is unambiguous; a
/persist full of half-written files is not, and the restore has to judge each
file on whether it is still valid for its own type, while files it has no
validator for fall back to the copies pillar keeps beside them. Restoring too
little strands the device, restoring too much discards state it did not need to.

The load-bearing assertion in both is not that files reappeared but that a
controller-encrypted credential is decrypted from the restored key: files can
come back empty or mismatched and satisfy every other check. It has to happen
offline, or the device is simply told everything again, so the controller is cut
off with an SDN firewall rule rather than by stopping it -- that isolates one
device and leaves the harness working. Restoring the path afterwards waits for
the device to check in, since reboots are counted from what it reports and the
audit runs at teardown.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
@eriknordmark

Copy link
Copy Markdown
Contributor Author

Rebased onto master 953778f63 and re-ran the whole suite on the rebased tree
against 0.0.0-resize-allprs-ef3029b1: all twelve pass, one run each, no
retries. The description carries the new timings.

One fix came out of the re-run. The three flavor-switch tests took their
other-flavor image from INITIAL_EVE_VERSION, so pointing that at a release —
as this PR's own how-to-test example did — made them ask for
lfedge/eve:12.1.0-k-amd64, which is not published, and they failed at the
image pull. They now default both version axes to the build under test, which
is the only thing that can cross the seam: #6036 is in no release, so a
released source refuses the update, and no release publishes both flavors.
Validated by running all three with neither INITIAL_EVE_VERSION nor
ALT_EVE_VERSION set (PASS 511s / 661s / 734s).

Worth flagging for anyone reproducing: TestPersistWipeRestore and
TestKvmToKVolumeMigration were intermittent on the pre-rebase tree, failing 5
of 7 and 2 of 3 runs on 2026-09-12. Both passed first try here, which is not
enough to call that resolved.

@eriknordmark
eriknordmark merged commit 1cf1c02 into lf-edge:master Sep 18, 2026
38 of 39 checks passed
@eriknordmark
eriknordmark deleted the evetest-konvert-tests branch September 19, 2026 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants