Skip to content

evetest: konvert robustness improvements and diagnostics gathering - #6823

Merged
rene merged 5 commits into
lf-edge:masterfrom
eriknordmark:appvol-verify
Oct 26, 2026
Merged

rene merged 5 commits into
lf-edge:masterfrom
eriknordmark:appvol-verify

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #6267 for the kvm→k konvert tests: robustness fixes and more diagnostics for the partly uploaded PVC case (#6760) and the EVE-k soak failures.

evetest infra: after a power cycle, the first info message of the new boot can carry a restart counter of 0 and the next one, from the same boot, the real value. The harness counted that catch-up as a second reboot, failing tests with "expected 6 reboot(s) but observed 7". A counter advance with an unchanged boot time is now treated as a baseline correction.

evetest tests (konvert):

  • MID_UPLOAD_POWER_CYCLE=true makes TestKvmToKRepartitionTopology cut power 2-8 s into the carried volume's CDI upload, so the next boot finds a partial PVC.
  • The app-volume tests report per run how many PVCs were found with an unfinished upload after a reboot and what EVE did with them.
  • Wedge diagnostics: zedbox fatal-error stacks on a config stall, and KubeVirt CR/pod/operator state when no VMI starts.
  • waitDeviceOnTarget matches the flavor field of the version, so a branch name containing -k- is no longer taken for EVE-k.

How to test and validate this PR

go test ./evetest/ ./evetest/tests/konvert/ runs the new unit tests (TestCountReboots, TestPVCUploadReport). The konvert suite itself runs with make evetest against an EVE-k image; set MID_UPLOAD_POWER_CYCLE=true for the mid-upload power cut.

Changelog notes

None (test-only changes).

PR Backports

  • 17.0-stable: No, test-only change; the konvert suite is not on that branch.
  • 16.0-stable: No, test-only change; evetest is not on that branch.
  • 14.5-stable: No, test-only change; evetest is not on that branch.
  • 13.4-stable: No, test-only change; evetest is not on that branch.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj

@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 27.65%. Comparing base (b73cbcb) to head (4278b49).
⚠️ Report is 107 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6823      +/-   ##
==========================================
+ Coverage   27.34%   27.65%   +0.31%     
==========================================
  Files         524      534      +10     
  Lines       96107    97842    +1735     
==========================================
+ Hits        26280    27060     +780     
- Misses      67731    68490     +759     
- Partials     2096     2292     +196     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark eriknordmark changed the title [WIP] evetest: mid-upload power cut and soak diagnostics konvert robustness improvements and diagnostics gathering Oct 20, 2026
@eriknordmark eriknordmark changed the title konvert robustness improvements and diagnostics gathering evetest: konvert robustness improvements and diagnostics gathering Oct 20, 2026
@eriknordmark
eriknordmark marked this pull request as ready for review October 20, 2026 09:26
eriknordmark and others added 5 commits October 24, 2026 11:31
After the harness power-cycles a device, the first info message of the
new boot sets the reboot baseline. zedagent reports a restart counter of
0 until nodeagent's status reaches it, so that message can carry 0 and
the next one, from the same boot, the real value. The harness counted
that step as a second reboot, and the test then failed with "expected 6
reboot(s) but observed 7" although the device booted only once.

When the restart counter advances but both messages carry the same boot
time, treat it as a baseline correction. A counter advance with no boot
time to compare is still counted.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
Two konvert wedges ended with nothing in the dumps saying why. When
zedbox dies on a Go runtime fatal error during onboarding, the watchdog
reboots the device and the config-v3 wait times out; newlogd keeps
panicStacks only for panics, not for fatal errors. The
config-fetch-stalled dump now greps zedbox's stderr (newlog source
pillar) for the fatal-error or panic header and the stack that follows.

When the KubeVirt CR never becomes Available, no VMI starts and the
app-volume wedge dump showed only the empty VMI list. That dump now also
records the CR's conditions, the kubevirt pods, their warning events
and the virt-operator log, plus the k3s apiserver watch-cache errors
that left virt-operator retrying a CRD create failing with "already
exists" in a 6.12 EVE-k soak.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
A reboot during a carried volume's CDI upload leaves a partly written
PVC behind, and whether EVE-k then adopts it decides if the app sees
missing data. The app-volume conversion tests now read the device log
at the end and report, per run, how many PVCs were looked at after a
reboot with their upload unfinished, how many of those uploads later
completed, and the upload-state verdicts and re-drives that EVE logs
when it checks a PVC it finds after a reboot.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
With MID_UPLOAD_POWER_CYCLE=true, TestKvmToKRepartitionTopology waits
for the CDI upload pod of the carried data volume to run, lets it go
for a random 2-8 s and cuts power, so the next boot finds a partly
uploaded PVC. The pod is polled every second because on kernel 6.12 it
finished the 2 GiB upload within 20 s of being seen running. The test
fails if the cut cannot land mid-upload, and logs the cut as
[MID-UPLOAD]; the existing volume checks then show whether the app
started on a partial volume.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
The wait for the device to run EVE-k matched "-k-" anywhere in the
short version, so a build whose branch name contains "-k-" (such as
kvm-to-k-...) was taken for EVE-k while still on the kvm hop, and
TestKvmToKRepartitionAppVolume checked the geometry before the
conversion had started. Compare the flavor field of
<version>-<flavor>-<arch> instead.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyjigXVFZhuUg5LiRSVwsj
@eriknordmark eriknordmark added the testing New test or test framework changes label Oct 26, 2026

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Merging since this is pure evetest and doesn't require Eden tests.

@rene
rene merged commit 4ba11c2 into lf-edge:master Oct 26, 2026
51 of 63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants