Skip to content

evetest: put /persist on a disk of its own - #6540

Merged
milan-zededa merged 2 commits into
lf-edge:masterfrom
eriknordmark:evetest-disks-layout
Sep 18, 2026
Merged

milan-zededa merged 2 commits into
lf-edge:masterfrom
eriknordmark:evetest-disks-layout

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

Lets a test give a device a /persist on a disk of its own, rather than on the
boot disk where EVE creates it.

That shape exercises a different path through EVE's storage setup, and it is
where the in-field repartition reaches its grow branch: with no /persist on
the boot disk to shrink, the space freed there is the only place to grow into.
The harness could not arrange it, so the twodisk-ext4 and twodisk-zfs
topologies from lf-edge/eden#1209 were out of reach.

Two offline disk edits are added. One writes a GPT on a blank extra disk with a
single partition typed and named as EVE's /persist; the other removes a named
partition from the boot disk. Used together they move /persist to its own
disk, because EVE locates it by partition label across every block device rather
than by which disk it sits on. Disk edits now carry the index of the disk to act
on, so an edit can reach an extra disk; the existing edits keep addressing the
boot disk.

The new partition is deliberately left unformatted. storage-init finds no
filesystem and formats it on the next boot exactly as it would a factory-fresh
disk — with the encryption feature the vault needs. Eden's escripts run
mkfs.ext4 -L PERSIST themselves, but EVE's own storage-init.sh uses
mkfs -t ext4 -F -F -O encrypt followed by tune2fs -O encrypt; reproducing
that by hand in the harness would risk a filesystem that differs from EVE's in
ways a test never thinks to check. Leaving it to EVE also keeps mkfs tooling out
of the broker image.

The second commit adds TestPersistOnSeparateDisk, which drives the sequence
end to end.

PR dependencies

None. The EditDeviceDisk broker RPC and the CAPABILITY_EDIT_DEVICE_DISK
capability these two edits extend are in master, from #6539.

How to test and validate this PR

Automated, by TestPersistOnSeparateDisk (part of TestStorageSuite):

make evetest NAME=TestPersistOnSeparateDisk

It brings a device up normally, confirms /persist is on the boot disk, powers
it off, creates a persist partition on the extra disk and removes the boot
disk's, then powers it back on. Passes at ~256s.

What the assertions are actually pinned on, for a reviewer validating by hand:

  • /persist starts on /dev/vda9 and ends on /dev/vdb1.
  • It comes back ext4. The harness hands over an empty partition, so a device
    reporting a formatted /persist is one that formatted it itself — this is the
    check that the "let EVE do the mkfs" decision above actually holds.
  • The device still takes configuration from its controller. /persist carries
    the identity and the last configuration, and a device that came back with a
    fresh one would answer over SSH exactly like a recovered one while being
    unable to be told anything.
  • The space freed on the boot disk is compared against what the removed
    partition actually occupied, read from the partition table before and after,
    rather than against a size derived from the disk. In the passing run those
    match exactly: free space goes from 2079232 to 42944395264 bytes, a gain of
    42942316032, which is precisely what P3 held.

Changelog notes

No user-facing changes. Test-only, plus the evetest framework support that test
needs. That support extends the broker API, so evetest/VERSION goes to 1.8 —
a broker image advertising CAPABILITY_EDIT_DEVICE_DISK at 1.8 is one that
serves these two edits.

PR Backports

  • 17.0-stable: No — test-only, and the capability it extends is not on the
    stable branches.
  • 16.0-stable: No, as the feature is not available there.
  • 14.5-stable: No, as the feature is not available there.
  • 13.4-stable: No, as the feature is not available there.

Checklist

  • I've provided a proper description
  • I've added the proper documentation — no separate docs; the test and the
    two new RPC messages carry documentation of what they do and why.
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device — not run on arm64; this drives a QEMU
    device through the evetest broker on an amd64 host.
  • I've written the test verification instructions
  • I've set the proper labels to this PR

🤖 Generated with Claude Code

https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA

@codecov

codecov Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 26.55%. Comparing base (1cf1c02) to head (1eefc47).
⚠️ Report is 8 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6540      +/-   ##
==========================================
+ Coverage   26.26%   26.55%   +0.29%     
==========================================
  Files         515      525      +10     
  Lines       94872    96588    +1716     
==========================================
+ Hits        24916    25650     +734     
- Misses      68036    68829     +793     
- Partials     1920     2109     +189     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


// appImageName is the container image the escripts use as their test app
// (eden's eclient equivalent for evetest).
appImageName = "milan4zededa/evetest-ubuntu-ctr"

@milan-zededa milan-zededa Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be lfedge image repo (we do publish this: https://hub.docker.com/r/lfedge/evetest-ubuntu-ctr)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — lfedge/evetest-ubuntu-ctr:1.0. (It lives in #6539, so the fix is there.)

Comment thread evetest/broker/broker.go
// device boots from, which the broker can only do while the file stays
// where it put it. The overlay strategy is exactly that case; a standalone
// disk may since have been handed to a node the broker cannot reach.
if diskStrategy == provider.DiskImageOverlay {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately this means that tests requiring CAPABILITY_EDIT_DEVICE_DISK will not be exercised in the github CI where we intend to use the Proxmox provider.
Are there going to be many tests in the konvert suite that require this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two of eleven tests in the konvert suite: TestKvmToKRepartition (it has to leave a free tail past the last partition, which the image generator will not do at creation) and TestPersistWipeRestore (it destroys the /persist filesystem while the device is off — TestBackupCorruptRestore damages the backup from inside EVE and needs no capability). Plus TestPersistOnSeparateDisk in #6540. The rest run anywhere.

So on Proxmox the suite still covers the conversion itself, the app/volume/blob carry-over and the declined-repartition paths; what it loses is the grow-into-slack route and the /persist-loss recovery.

@milan-zededa

Copy link
Copy Markdown
Contributor

Given all the API and other changes in evetest, I think it makes sense to also increase the version number in evetest/VERSION

Comment thread evetest/edgedevice.go
// own, not a lookalike built by the harness.
//
// The device must be powered off (PowerOff), and the test must have declared
// RequireCapabilities{CAPABILITY_EDIT_DEVICE_DISK}.

@milan-zededa milan-zededa Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If these new methods are part of CAPABILITY_EDIT_DEVICE_DISK, then it should be part of the same PR (and the increased evetest version) as the rest of this capability (#6539). Otherwise we may end up with evetest broker image which claims CAPABILITY_EDIT_DEVICE_DISK (after 6539 merge), but actually does not yet implement CreatePersistPartition and DeletePartition, failing (instead of skipping) tests that depend on them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and moved: the commit adding CreatePersistPartition/DeletePartition and their broker side is now in #6539 alongside the capability and the VERSION bump (1.7). #6540 is down to the test that uses them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: that move did not land. #6539 merged without these two edits, so
CreatePersistPartition/DeletePartition, the disk_index field and their
broker side stay in this PR, and evetest/VERSION is bumped here to 1.8 — so
the broker image that advertises CAPABILITY_EDIT_DEVICE_DISK at 1.8 is the
one that serves these edits, and 1.7 keeps meaning what it means in master.

Happy to split the framework half into its own PR if you would rather review it
separately from the test.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, no need to split.
Since we do not run evetest in the CI yet, it does not matter. Only once we start using evetest from github actions and across multiple stable branches, we will need to be little more careful with versioning and capabilities reported.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

VERSION bumped to 1.7, in #6539 with the rest of the framework change.

@rene rene added the testing New test or test framework changes label Sep 15, 2026
@eriknordmark

Copy link
Copy Markdown
Contributor Author

Given all the API and other changes in evetest, I think it makes sense to also increase the version number in evetest/VERSION

@milan-zededa given a number of pending evetest PRs, in which one (the first to merge, the last?) should we increase the version number?

eriknordmark and others added 2 commits September 17, 2026 21:30
A device whose /persist lives on a separate disk exercises a different
path through EVE's storage setup than the usual single-disk layout, and
the in-field repartition reaches its grow branch there: with no /persist
on the boot disk to shrink, the freed space is the only place to grow
into. The harness could not arrange that shape, so those topologies were
out of reach.

Adds two offline disk edits. One writes a GPT on a blank extra disk with
a single partition typed and named as EVE's /persist; the other removes a
named partition from the boot disk. Used together they move /persist to
its own disk, because EVE locates it by partition label across every
block device rather than by which disk it sits on.

The new partition is left unformatted on purpose: storage-init finds no
filesystem and formats it on the next boot exactly as it would a
factory-fresh disk, encryption feature included. Building the filesystem
in the harness instead would mean reproducing EVE's mkfs by hand, and a
test would not notice where the two had drifted apart.

Disk edits now carry the index of the disk to act on, so an edit can
reach an extra disk; the existing edits keep addressing the boot disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
Covers a storage shape the single-disk layout does not reach: /persist on
a dedicated disk rather than on the boot disk. The device is brought
up normally, powered off while a persist partition is created on its
extra disk and the boot disk's is removed, and then has to come back
using the new one.

That EVE adopts it is only half of what is asserted. The partition is
handed over empty, so a device that comes back with /persist formatted
ext4 is one that formatted it itself. The device also has to still take
configuration from its controller: /persist carries the identity and the
last configuration, and one that came back with a fresh /persist would
answer over SSH exactly like a recovered one while being unable to be
told anything.

The freed space is checked against what the removed partition actually
occupied, read from the partition table before and after, rather than
against a size derived from the disk. The two are not the same question,
and only the partition table's answer says what EVE can allocate.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZEoFvMK2UHByU3LAeXsxA
@milan-zededa

Copy link
Copy Markdown
Contributor

Given all the API and other changes in evetest, I think it makes sense to also increase the version number in evetest/VERSION

@milan-zededa given a number of pending evetest PRs, in which one (the first to merge, the last?) should we increase the version number?

@eriknordmark
Last. But this only concerns PRs that are making changes in the evetest framework. When only adding new tests, version should stay the same.
But this will all really matter only once we start using evetest in the github actions across mutliple EVE branches. At this point nothing really breaks when we forget to version evetest changes properly.

@milan-zededa
milan-zededa merged commit eef344b into lf-edge:master Sep 18, 2026
41 of 45 checks passed
@eriknordmark
eriknordmark deleted the evetest-disks-layout branch September 19, 2026 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants