Skip to content

evetest: verify app placement from the newest cluster info - #6889

Merged
milan-zededa merged 2 commits into
lf-edge:masterfrom
andrewd-zededa:evetest-backupdnid-hardening
Oct 28, 2026
Merged

milan-zededa merged 2 commits into
lf-edge:masterfrom
andrewd-zededa:evetest-backupdnid-hardening

Conversation

@andrewd-zededa

@andrewd-zededa andrewd-zededa commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

TestDNIDandBackupDNID can fail after vm2 fails back to edge-dev3, even though the cluster runs vm2 on edge-dev3:

vm2 did not fail back to its own designated node edge-dev3
Expected <string>: edge-dev1 to equal <string>: edge-dev3

The placement check read a stale cluster info report. FindDeviceHostingApp returned the host named by the first device whose stored ZInfoKubeCluster mentioned the app. There are three ways that report can be wrong:

  • It lags: only the stats leader publishes cluster info, at most every 30s.
  • A node that loses the eve-kube-stats-leader lease stops publishing without retracting its last report, so that report stays stored indefinitely.
  • A report can list the VMI being replaced next to its successor.

Changes:

  • evetest infra: FindDeviceHostingApp now trusts only the newest stored report across the cluster's devices, then waits for new reports as before. Finished VMIs are not counted. With excludeDevNames, a report listing both an excluded node and a new one now returns the new one. EdgeCluster.WaitUntilAppIsHostedOn is added for checks after an app moves: it waits until the newest report places the app on the given node and no other. The display-name lookup, device lookup and report watching are shared helpers.
  • evetest: TestDNIDandBackupDNID uses WaitUntilAppIsHostedOn for the failback check and for the check after activation on the backup node, and runs its HTTP readiness check against the device that was verified. The backup-node check now expects edge-dev1 itself rather than "not edge-dev3": edge-dev2 is the tie-breaker and edge-dev3 is down.

Draft until the cluster suite has been run against it.

How to test and validate this PR

make evetest NAME=TestDNIDandBackupDNID
make evetest NAME=TestNodeClusterSuite

FindDeviceHostingApp is used throughout the cluster tests, so run the whole suite, not just the one test. TestAppNodeNames unit-tests the new placement parsing: GOWORK=off go test -run TestAppNodeNames . from evetest/ on Linux.

Changelog notes

None. Test framework only.

PR Backports

  • 17.0-stable: To be backported. TestDNIDandBackupDNID is on 17.0-stable and hits the same race.
  • 16.0-stable: No, evetest is not there.
  • 14.5-stable: No, evetest is not there.
  • 13.4-stable: No, evetest is not there.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

🤖 Generated with Claude Code

@andrewd-zededa

Copy link
Copy Markdown
Contributor Author

Opened after more runs of the TestDNIDandBackupDNID evetest for backport PR: #6867

@andrewd-zededa andrewd-zededa added stable Should be backported to stable release(s) next-17.0.x-rc PR must be present in the next 17.0.x-lts release labels Oct 20, 2026
@andrewd-zededa
andrewd-zededa force-pushed the evetest-backupdnid-hardening branch from e0ac25a to ee7610c Compare October 21, 2026 20:06
andrewd-zededa and others added 2 commits October 26, 2026 14:04
FindDeviceHostingApp returned the host named by the first device whose
stored cluster info mentioned the app. That report can be stale: only
the stats leader publishes cluster info, and a node that loses the
lease stops publishing without retracting its last report. A report can
also list the VMI being replaced next to its successor.

FindDeviceHostingApp now trusts only the newest stored report across
the cluster's devices, then waits for new reports as before. Finished
VMIs are not counted.

Add EdgeCluster.WaitUntilAppIsHostedOn for checks after an app moves:
it waits until the newest report places the app on the given node and
on no other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Andrew Durbin <andrewd@zededa.com>
TestDNIDandBackupDNID failed after vm2 failed back to edge-dev3: the
cluster ran vm2 on edge-dev3, but the placement check read a stored
cluster info report that still placed it on edge-dev1, its backup node.
The check after activation on the backup node has the same race.

Use WaitUntilAppIsHostedOn for both checks, and point the HTTP
readiness check at the device it returns. On the backup node, expect
edge-dev1 itself: edge-dev2 is the tie-breaker and edge-dev3 is down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Andrew Durbin <andrewd@zededa.com>
@andrewd-zededa
andrewd-zededa force-pushed the evetest-backupdnid-hardening branch from ee7610c to df3169b Compare October 26, 2026 20:05
@andrewd-zededa

Copy link
Copy Markdown
Contributor Author

rebased

@andrewd-zededa

Copy link
Copy Markdown
Contributor Author

manual TestDNIDandBackupDNID run against HV=k ZARCH=amd64 passed

@andrewd-zededa
andrewd-zededa marked this pull request as ready for review October 26, 2026 22:04
@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.63%. Comparing base (4937a20) to head (df3169b).
⚠️ Report is 41 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6889      +/-   ##
==========================================
+ Coverage   28.32%   28.63%   +0.30%     
==========================================
  Files         537      548      +11     
  Lines       98110    99857    +1747     
==========================================
+ Hits        27790    28592     +802     
- Misses      68064    68815     +751     
- Partials     2256     2450     +194     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@milan-zededa
milan-zededa merged commit b419630 into lf-edge:master Oct 28, 2026
49 of 50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

next-17.0.x-rc PR must be present in the next 17.0.x-lts release stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants