Skip to content

evetest: kvm→k boot-disk conversion tests + volverify app - #6267

Merged
rene merged 32 commits into
lf-edge:masterfrom
eriknordmark:appvol-verify
Oct 9, 2026
Merged

rene merged 32 commits into
lf-edge:masterfrom
eriknordmark:appvol-verify

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Description

evetest coverage for the in-field EVE-kvm→EVE-K conversion: five new konvert
tests, wider coverage in the suite's existing ones, the test app one of them
needs, and the framework changes they required.

The conversion has two shapes depending on whether an application volume is on
the device when it runs. The suite's existing conversion test deletes the app
before the flavor change, so the repartition has nothing to carry; it is now
named TestKvmToKRepartitionNoVolmig. The new tests:

  • TestKvmToKRepartitionVolmig drives the same three hops — a small-geometry
    EVE-kvm release, a kvm→kvm hop that lands the conversion code, then the kvm→k
    hop that arms the offline shrink — with a container app and its data volume
    kept in place, and asserts the contents come back.
  • TestKvmToKRepartitionAppVolume asks what the shrink does to that data when
    it has to relocate the volume holding it (below).
  • TestKvmToKRepartitionTopology runs the grow route on /persist on a second
    disk (ext4, a ZFS partition or a whole-disk pool) and on ZFS with a free tail
    on the boot disk, by default with a 2 GiB data volume carried across it.
  • TestKvmToKZFSVaultMigration and TestKvmToKZFSVaultPowerCutMidSwap cover
    the ZFS vault migration declining for space and losing power mid-swap (below).

The existing tests gain:

  • TestKvmToKVolumeMigration carries a downloaded and a clear-text volume
    beside the blank one and, with VM_APP (default on), an Alpine VM app. It
    runs on ZFS with PERSIST_FILESYSTEM=zfs, and it asserts the kvm copies are
    removed after the commit. REBOOT_BEFORE_DRAIN reboots EVE-K while those
    copies are still held.
  • NoVolmig, RepartitionVolmig and Topology reboot the converted device and
    re-run their gates (REBOOT_AFTER_CONVERSION, default on). NoVolmig adds a
    local-network-instance variant.
  • NoVolmig, RepartitionVolmig and AppVolume stage the identity- and
    vault-critical /persist files above the shrink boundary (RELOCATE_CRITICAL_HIGH, default on) and records
    where they end up.
  • The restore tests choose with RESIZE_FAILURE between a real resize failure
    and a planted marker.
  • Conversion hops get 45-minute upgrade budgets, 120 for Geometry.

TestKvmToKRepartitionAppVolume fills /persist before the app exists so the
volume is allocated in the blocks the shrink must evacuate, writes a verifiable
pattern into it, runs the conversion — optionally letting the chipset watchdog
interrupt the shrink — and judges the volume with fsck plus a replay of the
pattern. The verdict is taken on the volume itself and does not depend on the app
coming back, because the post-conversion app wedge has repeatedly cost otherwise
complete runs their result. On an integration build carrying #6063, a 24-hour run
of 36 conversions drove 193 watchdog resets into the shrink and every volume
verdict came back clean, which matches resize2fs flushing each relocated extent
before repointing any inode at it. A QEMU watchdog reset is a guest-level reset,
so this says nothing about bare-metal power loss.

Whether a build carries that fault injection is read off the device. Each kvm→k
repartition test classifies the run from the resizer's console output — watchdog
held unfed, fed, no watchdog device, or no resizer output — and logs it with the
number of resize attempts and whether each reset landed in the shrink or the
grow. The tests that claim data survived fail on the last two cases, where a run
that interrupted nothing is indistinguishable from one that came back clean; the
layout-only tests record and move on.

TestKvmToKZFSVaultMigration runs on a device whose /persist is ZFS, where the
flavor change copies the carried-over vault into a zvol and swaps it into place.
It makes the vault too large to fit twice in the pool, so the migration
declines, and asserts the device keeps nothing the migration created: no staging
zvol, no parked backup, no etcd volume, no swap record, the vault still readable,
and the pool's free space restored. A leaked staging zvol can hold all the space
the pool had free, and the EVE-kvm the device falls back to cannot reclaim it.
Growing the vault rather than filling the pool keeps the device clear of low-disk
maintenance mode.

TestKvmToKZFSVaultPowerCutMidSwap cuts power between the swap's two renames,
where the vault path does not exist. EVE-kvm then renames the pre-migration vault
back, discards the copy and clears the swap record, and the next EVE-K boot
redoes the migration. The test follows the vault's contents through each step.
It expects the parked vault to stay until the partition is committed and to be
dropped after that. The wait for the EVE-K pass keys on recovery's decision
rather than the migration declining, because the two sit on opposite sides of
the vault unlock. The EVE-K boot is staged through zboot rather than re-pushed,
because baseosmgr will not retry a baseos it has marked failed.

Framework changes:

  • UpgradeEVE takes an optional timeout. A failed upgrade is reported with the
    device's error, and a retryable error is waited out.
  • The QEMU watchdog resets the guest. The emulated q35 LPC bridge otherwise only
    latches a status bit, which silently disarms any injected watchdog.
  • EdgeDevice.ConsoleOutput returns the serial console, the only view of
    storage-init and the offline resizer. Console output is sanitized at the broker
    because GRUB's CP437 bytes are not valid UTF-8 and fail the whole gRPC
    response, which also repairs end-of-test console collection.
  • Reboot accounting counts from RestartCounter, falling back to the
    BootTime change, and counts a harness-issued power cycle when it is issued.
    EdgeDevice.DisableRebootAccounting lets the power-cut test opt out.
  • A failing framework check no longer hangs the test while holding the
    harness's test lock.

evetest/testapps/volverify writes and replays the pattern: a seeded block
stream with an on-volume committed index, so an interrupted write is
distinguishable from corruption. It has its own unit tests and a loopback-ext4
script for iterating off-device.

The tests that carry a volume across a shrink lay the boot disk out with the
installer by default (USE_INSTALLER), since every other test starts from a
live image. The tests that carry a volume, and the two ZFS vault tests, raise
the device floors to 16 GiB and 8 vCPUs, because EVE-K plus Longhorn does not
converge on the suite's 8 GiB and 4 vCPUs once a volume has to survive. The rest
of the suite keeps those figures.

The commits are the framework, then volverify, then one commit with the changes
to the existing tests and the helpers they introduce, then one commit per new
test, ending with the TestKvmToKVolumeMigration additions.

PR dependencies

None for merging. Running the tests that carry a volume across the conversion
needs an EVE build that lets the update through with volumes present, which
master refuses: TestKvmToKVolumeMigration and Topology's carried volume need
#6658; TestKvmToKRepartitionVolmig and TestKvmToKRepartitionAppVolume shrink
/persist, which #6658 still refuses with volumes present, so they need a build
that relaxes that gate for testing. The drain and ZFS-carry assertions in
TestKvmToKVolumeMigration also need #6658.

How to test and validate this PR

Test-framework only. Against an EVE build as above:

make evetest NAME=TestKvmToKRepartitionVolmig
make evetest NAME=TestKvmToKRepartitionAppVolume
make evetest NAME=TestKvmToKRepartitionTopology
make evetest NAME=TestKvmToKZFSVaultMigration
make evetest NAME=TestKvmToKZFSVaultPowerCutMidSwap
EVETEST_PERSIST_FILESYSTEM=zfs make evetest NAME=TestKvmToKVolumeMigration

RepartitionVolmig and AppVolume assert the pre-flight decision is a shrink, so
they need INITIAL_EVE_VERSION at a small-geometry base (the default, 10.1.0, is
one) and no free tail on the boot disk.

TestKvmToKRepartitionAppVolume takes DATAVOL_MB (default 2048),
FILL_PEAK_PCT, FILL_KEEP_GIB, CDI_SAMPLE_SECS, DEVSIDE_ONLY and
VOLVERIFY_SEED/VOLVERIFY_OPS/VOLVERIFY_IMAGE. A run that finds nothing to
verify is reported as such rather than as a pass.

TestKvmToKZFSVaultPowerCutMidSwap needs an image built with
FAULT_INJECTION=y and skips without one. TestKvmToKZFSVaultMigration starts
on the current EVE-kvm build, whose geometry already fits EVE-K, and asserts on
the device log that the migration really declined.

volverify can be exercised without a device:

go test ./evetest/testapps/volverify/...
./evetest/testapps/volverify/scripts/loopback-ext4-test.sh

Changelog notes

No user-facing changes. Test-framework only (evetest/).

PR Backports

  • 16.0-stable: No — new tests for a feature that is not in that branch.
  • 14.5-stable: No — same.
  • 13.4-stable: No — same.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

@codecov

codecov Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 27.46%. Comparing base (f000d56) to head (07f227d).
⚠️ Report is 39 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6267      +/-   ##
==========================================
+ Coverage   27.17%   27.46%   +0.29%     
==========================================
  Files         524      534      +10     
  Lines       95907    97623    +1716     
==========================================
+ Hits        26058    26809     +751     
- Misses      67770    68551     +781     
- Partials     2079     2263     +184     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread evetest/testapps/volverify/Makefile Resolved
Comment thread evetest/edgedevice.go Outdated
// reboots additional times; the number is known only to the test driving it (a
// plain kvm<->k conversion with no resize reboots differently than a shrink+grow),
// so the test declares it here to keep the teardown reboot-count check accurate.
func (d *EdgeDevice) ExpectAdditionalReboots(n int) {

@milan-zededa milan-zededa Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps additional expected reboots and upgrade timeout can be just optional arguments for EdgeDevice.UpgradeEVE()?

@milan-zededa milan-zededa Aug 7, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that ExpectAdditionalReboots is needed also in other non-upgrade scenarios, so keep it as you have it (e.g. when removing node from cluster and converting it to single-node).
But I think that timeout should be argument of UpgradeEVE like we have in RebootApplication, PurgeApplication and some other methods.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right that RebootApplication and PurgeApplication both take a timeout time.Duration. The timeout is SetUpgradeTimeout on the device handle instead: it also covers the waits UpgradeEVE does internally, and it persists across calls, which the conversion tests need since they upgrade twice. What I want to avoid is a sixth positional argument after two bools — UpgradeEVE(v, hv, ds, true, false, 45*time.Minute) is hard to read at the call site.

There is exactly one existing caller, so switching to an argument is cheap if you would rather have the consistency. Your call — say which and I will make it match.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer timeout as (optional, with default) arg for UpgradeEVE instead of adding another method just for it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — UpgradeEVE takes an optional trailing timeout ...time.Duration; SetUpgradeTimeout is gone and the existing call sites are unchanged.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in that thread: the timeout is resolved inside UpgradeEVE and threaded down as a parameter, so no upgrade state survives the call.

eriknordmark added a commit to eriknordmark/eve that referenced this pull request Aug 14, 2026
The tail commit is evetest-only, so the image built from the previous tip
stays valid.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Aug 14, 2026
The tail commit is evetest-only, so the image built from the previous tip
stays valid.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@eriknordmark
eriknordmark requested a review from rene September 9, 2026 09:59
@eriknordmark

Copy link
Copy Markdown
Contributor Author

The yetus -1 is a single line that no code change can clear:

evetest/testapps/volverify/0 issues.

1 new + 507 unchanged - 0 fixed = 508 total (was 507). The golangcilint
plugin prints one header line per Go module and diffs the result files line by
line, so a new go.mod adds a line the master-side scan cannot have — even
though the module itself is clean. blanks, revive, shellcheck and codespell are
all +1.

@eriknordmark eriknordmark changed the title [WIP] evetest: kvm→k boot-disk conversion tests + volverify app evetest: kvm→k boot-disk conversion tests + volverify app Sep 9, 2026
@eriknordmark
eriknordmark marked this pull request as ready for review September 9, 2026 14:21

@milan-zededa milan-zededa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use parameter for installer vs live image selection instead of inventing a new way of creating test variants.

@eriknordmark eriknordmark added the testing New test or test framework changes label Sep 12, 2026
@eriknordmark

Copy link
Copy Markdown
Contributor Author

Rebased on master.

On the red test job: the failures are dpcmanager TestSingleDPC and the containerd-client tests timing out on /run/containerd/containerd.sock. This PR touches nothing under pkg/pillar — the diff is evetest/ only — so neither is reachable from it.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

Fresh CI, with better evidence on the red test job than my note above: master's own test job is red the same way (run 34806366868 on 46356b9, dpcmanager TestLPSConfig), and the failing set inside dpcmanager moves between runs. That package is flaky on master, so every PR inherits the red check.

The yetus -1 is the golangcilint count artifact, not a finding: 1 new + 507 unchanged, every other plugin +1, and the run posts no inline annotation to act on.

@eriknordmark
eriknordmark force-pushed the appvol-verify branch 2 times, most recently from 4816e47 to 0da32c3 Compare September 14, 2026 19:18
Comment thread evetest/edgedevice.go Outdated
})
}

// ExpectAdditionalReboots tells the harness to expect n more device-initiated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that there already is function ExpectReboots -- either reuse that as it is or rename it to ExpectAdditionalReboots

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — they were identical, both looping incExpectedRebootCount. Reusing
ExpectReboots as it stands and dropping the duplicate, with the conversion case
folded into its doc comment so that reasoning is not lost. The two call sites in
the resize tests now use it, and the change is folded into the commit that
introduced the duplicate rather than added on top.

eriknordmark and others added 8 commits September 30, 2026 22:42
The conversion test that deletes its app before the flavor change is
renamed TestKvmToKRepartitionNoVolmig, since its repartition carries no
volume, and gains a variant on a local network instance. It and the
Geometry test get conversion-sized upgrade budgets (45 and 120 minutes)
instead of the framework's 20-minute default, which an offline shrink
routinely exceeds. Every shrink test now stages the identity- and
vault-critical /persist files above the shrink boundary and records where
they end up, and the conversion tests classify from the serial console
whether the image could interrupt the resize and how many attempts were
cut, so a clean result says which case it covers. NoVolmig reboots the
converted device and re-runs its gates, since only EVE-K's second boot
shows the cluster and Longhorn coming back on an existing node.

The restore tests choose with RESIZE_FAILURE between a real resize failure
and a planted marker, and assert that a later kvm-to-k update is declined
on that marker. Failure dumps are armed before the first hop, sweep newlog
for base-OS errors and capture console alarms, and the package doc states
the device sizing and valid start releases on their own terms.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestKvmToKRepartitionAppVolume asks what the offline boot-disk shrink does
to an application's data when it has to relocate the volume holding it.
/persist is filled before the app exists, so the volume is allocated in
the high blocks the shrink must evacuate, and the volume carries the
volverify pattern rather than a single marker, so what comes back can be
judged file by file. On a build whose resizer runs under a watchdog it
stops feeding, the shrink is interrupted part-way.

The verdict is taken on the volume on the device, before Longhorn ingests
it, because a post-conversion app that never reaches running has cost
otherwise complete runs their result; the app-side half then re-takes it
from inside the guest and adds the filesystem's own check. Placement and
the guest watchdog are asserted rather than assumed, since a volume below
the boundary is never relocated and an unarmed watchdog cannot interrupt
anything, and either would make a clean verify prove nothing.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestKvmToKRepartitionVolmig keeps the app and its data volume in place
across the boot-disk shrink and asserts the contents come back, then
reboots the converted device and checks again on a fresh guest boot. It
needs volume-migration support an EVE build may not have, so it sits
beside NoVolmig rather than replacing it. EVE-K plus Longhorn does not
converge on 8 GiB and 4 vCPUs once a volume has to survive, so the floors
are raised for a test that carries one, and USE_INSTALLER picks whether
the boot disk is laid out by the installer or from the live image.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drive a kvm-to-k update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device keeps nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a readable filesystem dataset and the pool's free space restored. A
leaked staging zvol can hold all the space the pool had free, and the
EVE-kvm the device falls back to cannot reclaim it.

The vault is grown rather than the pool filled, so the pool stays clear
of low-disk maintenance mode, and a log assertion on the decline keeps a
revert for another reason from passing. The pool sits on a 32 GiB disk of
its own beside a 32 GiB boot disk with no P3, as in the eden leg this is
ported from, since where the pool lives decides what the migration
attempts. A failed decline dumps the vault's dataset and mount state, so a
vault that did not unlock can be told from one that lost its data.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Between the vault migration's two renames the vault path does not exist,
and only a real interruption shows what ZFS's rename and the swap record's
fsync do when the machine disappears there. The test arms the fault point
of a FAULT_INJECTION=y image, waits for the device to park in that window
and cuts power. EVE-kvm then renames the pre-migration vault back and
clears the swap record, and the next EVE-K boot, staged through zboot
because baseosmgr will not retry a baseos it marked failed, redoes the
migration. The vault's contents are followed through each step, the
parked vault must remain until the partition is committed and be dropped
after it, and the test skips on images without fault injection.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestKvmToKRepartitionTopology runs the grow route, the one a device with
app volumes may take, on the layouts other than a single ext4 disk:
/persist on a second disk as an ext4 partition, a ZFS partition or a
whole-disk ZFS pool, and ZFS on a boot disk grown to leave a free tail.
By default a volverify app's 2 GiB data volume is carried across and
judged on the device and from inside the app, since a grow relocates no
/persist data and a blank volume is data loss; WITH_APP_VOLUME=false
instead redeploys the app and asserts its blobs were reused. The
converted device is then rebooted and checked again.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestKvmToKVolumeMigration now carries a downloaded volume and a
clear-text volume beside the blank one, since each is carried from a
different place, and an Alpine VM app whose boot disk is a downloaded
qcow2 (VM_APP, default on). PERSIST_FILESYSTEM=zfs runs it on ZFS, where
the volumes are zvols read from the parked kvm vault. A carried PVC
that comes up empty fails fast, nothing may be re-downloaded, and once
the markers are found the test asserts that the kvm copies, and on ZFS
the parked vault, are removed after the conversion is committed.

With REBOOT_BEFORE_DRAIN the test holds off volumemgr's drain and reboots
EVE-K first: the app, the markers and the held volumes must survive, and
on ZFS every carried zvol must get its device node back, which needs the
parked vault's key reloaded.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The QEMU watchdog, the upgrade timeout and error handling, console
output and the reboot-accounting changes on this branch alter the
harness image, so its tag moves past master's 1.9.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
eriknordmark and others added 10 commits October 1, 2026 12:25
When the boot after a repartition left no vault record in newlog, the
seal assertion graded the settle boot's controller-key fallback instead
and failed on the seal. It now grades only records written after the
vault settled. If the settle's records survived and nothing later did,
it fails as incomplete newlog. It also counts boot markers against the
device's restart counter, logs device against host clock at each settle
and at the assertion, and dumps a /persist/newlog inventory when records
are missing or the unlock was not local.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
After /persist moves to the extra disk, a device can connect to the controller and then never fetch its config, and the run fails on the harness's 2-minute config wait with nothing but a timeout. When zedagent has not reported a successful fetch 75 s into that wait, the test now records zedagent's status, host pressure, tasks in uninterruptible sleep with their stacks (sysrq-w, so they reach the kept console), the state and writability of /persist, and zedagent's log records, while the device is still up. The test's pass and fail behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Staging the critical files above the shrink boundary fills /persist to ENOSPC, and a newlogd write failing in that window is fatal: the device reboots, zedagent falls back to a backup config that still holds a deleted app, and the conversion is refused for a reason the test is not about. newlogd is now stopped from the last two chunks of free space until space is freed again, and on a test that filled /persist first the high chunks to free are found before the top-off, so the filesystem stays full only while the tails are written rather than through a sync and a filefrag scan of every chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
The critical-file staging never paused newlogd on a device: pillar's busybox pgrep -x compares the name against /usr/bin/newlogd, so the lookup found nothing and newlogd still died on ENOSPC during the top-off. Match /proc/<pid>/comm instead, and say so in the staging output when no newlogd is found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
A device that onboards quickly can send its first device info before
the harness notices the onboarding and subscribes to the controller's
info stream, which replays nothing. If the test then upgrades at once,
the first info the harness handles is from the boot after the upgrade,
and the reboot check counts zero. setupDevice now waits until the
harness reports a device state, which happens only when it has handled
an info message from the current boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
A freshly onboarded device with no config applied sends no device info
for minutes, so the wait in setupDevice timed out and failed
TestKvmToKRepartitionTopology before it did anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
On a host that overcommits memory, a device VM whose pages were swapped
out could wedge for good: a guest task faulting on such a page sleeps
until KVM signals the page is ready, and in a konvert soak that signal
never came, leaving a Longhorn engine blocked for over 20 minutes and the
app's volume reads failing. The QEMU provider now starts the device VM
with kvm-asyncpf off, so the vCPU simply waits for the page-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
The harness learns a device has onboarded by polling Adam every 15 s and
only then subscribes to its info, while Adam's stream carries only
messages received after it is opened. A device that sent its first-boot
info inside that window and then went quiet until its next reboot left
the reboot accounting to take that post-reboot message as its baseline,
so a test that reboots once reported "expected 1 reboot(s) but observed
0".

Each device's info subscriptions now share one feed in the Adam client.
The feed opens Adam's stream, then reads Adam's stored history, and
keeps the latest message per reported object: the device, and each app,
network instance, volume, content tree, metadata entry and patch
envelope by its UUID. A deletion, being the UUID alone, replaces its
object's entry. A new SubscribeToDeviceInfoMsgsWithLatest hands a
subscriber those latest messages first and every later one after them,
and the harness's device state watcher uses it. The existing
SubscribeToDeviceInfoMsgs stays live-only, since its waiters expect each
message to be news, but gains the feed's other properties: a message
present in both history and stream is delivered once, and on a reconnect
the history read delivers what Adam received while the stream was down.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestKvmToKRepartitionVolmig and TestKvmToKRepartitionAppVolume need a
kvm→k update that shrinks /persist while app volumes exist, which
baseosmgr refuses, so in the suite they can only fail on that refusal.
Comment them out of the konvert suite until such an update is allowed.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clearing kvm-asyncpf alone left async page faults on: the 6.12 guest
kernel enables them when the host advertises the interrupt-delivery
variant, and a konvert soak unit wedged on one with the first flag in
place. The device VM now has both bits cleared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
@rene

rene commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 2m 23s —— View job


Review

Reviewed the framework changes and the new volverify app logic (skipping the large body of tests/konvert/* test scaffolding, which is well-covered by prior review rounds in this thread). This is a solid, carefully-written test-framework PR — the concurrency in controller/infofeed.go, the reboot-accounting state machine in devstate.go/harness.go, and the crash-consistent commit scheme in testapps/volverify are all sound. One low-severity note posted inline.

  • Framework changes (broker, controller, devstate, edgedevice, harness, setup)
  • volverify app logic (engine, commit/ping-pong, block/report)
  • Reboot-accounting leak milan flagged — confirmed reset on both reuse paths (setup.go:1445, setup.go:1541)
  • Posted inline finding

Finding (low)

  • controller/infofeed.go — the seen dedup set grows unbounded for the life of the feed (one hash per distinct info message, never reclaimed). Fine for short tests; for the multi-hour konvert soaks it's a slow memory growth. Suggested bounding it to the reconnect-overlap window or to the latest object set. (inline)

Other observations (non-blocking, not posted inline)

  • engine.go:224-228 checkFile has a dead branch — both arms of the os.IsNotExist check return the same FileAnomaly{Verdict: FileLost}. Harmless.
  • The apply dedup relies on every ZInfoMsg carrying a distinct send timestamp; that assumption is stated in the comment and holds for EVE, so legitimate re-sends aren't swallowed.

The yetus -1 is the known golangcilint line-count artifact from adding a new Go module, not an actionable finding, as discussed in the thread.

if _, dup := f.seen[sum]; dup {
return
}
f.seen[sum] = struct{}{}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The seen dedup set grows without bound for the lifetime of the feed — one 32-byte key (plus map overhead) per distinct info message. A feed is one per device and lives as long as any subscription holds a ref, so for the long konvert soak tests (the PR describes a 24-hour / 36-conversion run) this accumulates every unique ZInfoMsg ever streamed and never reclaims. It won't fail a run, but it's a slow unbounded growth worth bounding.

The dedup is really only needed to absorb the overlap between the freshly-opened stream and the replayed history around a (re)connect. Consider clearing/rebuilding seen in connect() on each reconnect, or capping it (e.g. keep only hashes of the current latest set), so steady-state memory stays proportional to the object count rather than to total messages seen.

After /persist moves to a new disk, a device has twice sat with zedagent
waiting for onboarding through the whole config wait, and the stall dump
left no way to tell why: zedclient logs only to the device's newlog, and
the device is torn down when the wait fails. The dump now also records
zedclient's OnboardingStatus, whether the device can reach the
controller over TLS, and zedclient's newlog records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LvDPKeWJjD3ahjiJKF8b7f
Signed-off-by: eriknordmark <erik@zededa.com>
eriknordmark and others added 2 commits October 9, 2026 03:50
The onboarding stall dump showed a device that reached the controller
but never published an OnboardingStatus. device-steps.sh starts the
onboarding client only after the device certificate, the server wait
and tpmmgr createCerts, so the dump now also records the boot script's
last progress lines, any onboarding processes still running with their
elapsed time, and whether the device certificate exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LvDPKeWJjD3ahjiJKF8b7f
Signed-off-by: eriknordmark <erik@zededa.com>
zedbox names an agent's logger after the binary it was invoked as, so
the onboarding agent logs under source "client"; grepping for
"zedclient" found only other agents subscribing to its topic. The
onboarding stall dump now selects the client's own records, and also
records nim's DeviceNetworkStatus state, which the client waits on
before it registers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LvDPKeWJjD3ahjiJKF8b7f
Signed-off-by: eriknordmark <erik@zededa.com>
@rene
rene merged commit f5f8d04 into lf-edge:master Oct 9, 2026
48 of 49 checks passed
@eriknordmark
eriknordmark deleted the appvol-verify branch October 9, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants