Skip to content

evetest: upgrade a three-node EVE-k cluster - #6673

Merged
eriknordmark merged 1 commit into
lf-edge:masterfrom
eriknordmark:cluster-upgrade-test
Oct 19, 2026
Merged

eriknordmark merged 1 commit into
lf-edge:masterfrom
eriknordmark:cluster-upgrade-test

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds TestThreeNodesUpgrade, an evetest that forms a three-node EVE-k cluster on an earlier EVE, deploys a container app, and upgrades the nodes one at a time to the version under test, bootstrap node last. After each node's upgrade, a fresh cluster report must show all three nodes Ready with healthy storage before the next one starts, so the cluster runs mixed versions for most of the test. At the end the app must still answer over its port forward.

  • EVETEST_INITIAL_EVE_VERSION (default 0.0.0-master-66475696, the same baseline as TestEVEUpgradeKubevirtToKubevirt) and EVETEST_INITIAL_EVE_REPO (default lfedge/eve) pick the baseline.
  • EVETEST_EVE_VERSION picks the target and defaults to the checked-out HEAD, so a run on master upgrades to master. EVETEST_EVE_LIVE_IMAGE upgrades to the local build instead, as in TestEVEUpgrade.

The test runs in TestUpgradeSuite. It builds the same cluster as TestThreeNodesCluster, with its own copy of newThreeNodeClusterConfig because the cluster test package cannot be imported.

Not covered: a node joining for the first time while the cluster runs mixed versions, which needs a fourth device. The App-Tracker check on each node is left out because it needs #6665's App-Tracker on 127.0.0.1, and will be added once both PRs merge.

How to test and validate this PR

Run the test on its own, or as part of TestUpgradeSuite:

EVETEST_EVE_VERSION=<new> EVETEST_HYPERVISOR=kubevirt make evetest NAME=TestThreeNodesUpgrade
EVETEST_EVE_VERSION=<new> make evetest NAME=TestUpgradeSuite

It passes when every node-<name>-upgraded checkpoint is reached and the app answers after the last upgrade. On a failure, the last checkpoint reached shows which phase timed out and how far the rolling upgrade got.

Changelog notes

None, test only.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

@eriknordmark

Copy link
Copy Markdown
Contributor Author

@andrewd-zededa do you have a similar cluster upgrade test underway? If so we can merge the two rather than land both.

@codecov

codecov Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.38%. Comparing base (86a6aea) to head (4370107).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6673      +/-   ##
==========================================
+ Coverage   28.10%   28.38%   +0.27%     
==========================================
  Files         534      545      +11     
  Lines       97717    99456    +1739     
==========================================
+ Hits        27464    28231     +767     
- Misses      68019    68797     +778     
- Partials     2234     2428     +194     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark
eriknordmark marked this pull request as ready for review September 29, 2026 06:53
@rene rene added the testing New test or test framework changes label Oct 1, 2026
@rene

rene commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 3m 41s —— View job


Review complete

  • Read the changed file
  • Analyzed for bugs / quality / correctness
  • Posted review

I reviewed evetest/tests/cluster/upgrade_test.go. Overall this is a well-structured test; I found no correctness bugs in the upgrade loop or the readiness/freshness logic. A few notes:

Correctness — holds up:

  • waitForFreshNodesReady correctly avoids the stale-snapshot trap: it subscribes via WatchClusterInfo() after UpgradeEVE(...) has returned, and never consults the cached GetClusterInfo(), so a pre-reboot "Ready" report can't satisfy it. This matches the live-stream usage pattern in EdgeCluster.WaitUntilNodesAreReady.
  • Watching all three devices and accepting the first that reports everyone Ready correctly handles leadership moving when the bootstrap node (upgraded last) reboots — only the leader publishes cluster info.
  • Goroutine lifecycle is clean: buffered ready channel (no send-block), defer stop() per subscription, all goroutines exit on either success or ctx.Done(). Loop-variable capture is safe on Go 1.25.
  • allClusterNodesReady gates on healthy storage and every named node's READY condition being set; a node absent/without the condition during reboot correctly yields false.

Material gap (posted inline at the "Suite placement" doc):

  • The test is not registered in any RunTestSuite, so CI will never schedule it — it only runs via make evetest NAME=TestThreeNodesUpgrade. This confirms milan-zededa's earlier review point; either add a suite entry or move it under tests/upgrade.

Minor (not blocking, not posted inline):

  • allClusterNodesReady doesn't break after matching a node name / after finding the READY condition — harmless at N=3 nodes.
  • Per the PR body, the default 17.0.0-lts baseline doesn't form within the 30-min budget; that's a noted open investigation, not a code defect in this file.

I could not run go vet/go build on the package (command approval/network not available in this environment); the author reports a passing 53-min run on amd64 and Codecov shows the lines covered.
· branch cluster-upgrade-test

Comment thread evetest/tests/cluster/upgrade_test.go Outdated
// Suite placement
// ---------------
// - Standalone: its devices run a different EVE version than
// TestNodeClusterSuite's and cannot be reused.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirming milan-zededa's point with the concrete consequence: TestThreeNodesUpgrade is not referenced from testsuite_test.go's RunTestSuite nor any other suite, so as written it will never be scheduled by CI — it only runs when invoked explicitly via make evetest NAME=TestThreeNodesUpgrade. The "Standalone" rationale here (devices run a different EVE version and can't be reused by TestNodeClusterSuite) is valid for reuse, but it doesn't register the test for execution. Either add it to an existing/new suite entry point, or move it under tests/upgrade as suggested, so it actually gets exercised.

Note there is precedent for standalone-but-unrun cluster tests here (e.g. vmapp_test.go, pvcredownload_test.go are "not in TestNodeClusterSuite yet"), so this is an accepted interim state — but worth resolving before relying on this test as a regression guard.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eriknordmark Still waiting for this one to be addressed. Otherwise it would be good to go..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed: the test is registered in TestUpgradeSuite (see the thread on line 201).

Form a three-node cluster on an earlier EVE-k, deploy a container app,
and upgrade the nodes one at a time to the version under test,
bootstrap node last. After each upgrade a fresh cluster report must
show every node Ready, and at the end the app must still answer. The
test runs in TestUpgradeSuite and by default starts from the same
pre-17.0 master build as the suite's EVE-k upgrade variant, since
17.0-stable lacks a fix needed for reliable app deployment. The
initial version and repo are parameters; the target defaults to the
checked-out HEAD.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULraDivKMXrdxJxycz6g6h
@eriknordmark

Copy link
Copy Markdown
Contributor Author

Local run results, before the move to TestUpgradeSuite:

  • From 0.0.0-master-3f136150 to 0.0.0-port-scan-c903eb10 (evetest: port scans; fix the exposure they found on EVE and EVE-k #6665): passed in 53 min. Formation took 13.5 min, then 7.5, 8 and 14 min per node until all were Ready again. That run also checked each node's App-Tracker.
  • From 17.0.0-lts: the cluster did not become Ready within 30 min, before any upgrade started. The cause is not known.

The new default baseline, 0.0.0-master-66475696, has not been run with this test yet.

// which is needed for reliable app deployment.
// For the time being, we will use EVE version from the master, at the commit just
// before the big rewrite of kube-init from shell scripts to Go.
const initialEVEVersionForKubevirt = "0.0.0-master-66475696"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw. we can now use:

git tag --contains 3f95383ccf76b7803736122065967de220450402
17.0.1-rc1
17.0.1-rc2
17.0.1-rc3

But this can be changed later, when 17.0.1 is finally released.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW We refer to the same master branch hash in another cluster test. Let's fix both when 17.0.1 is out.

@milan-zededa

Copy link
Copy Markdown
Contributor

@eriknordmark Once #6833 is merged, you could backport this to 17.0 (I think we should have this test there as well and find out why it is failing).

@eriknordmark

Copy link
Copy Markdown
Contributor Author

No need to wait for eden for this evetest PR.

@eriknordmark
eriknordmark merged commit 850c08d into lf-edge:master Oct 19, 2026
46 of 54 checks passed
@eriknordmark
eriknordmark deleted the cluster-upgrade-test branch October 27, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants