Skip to content

evetest: don't reuse devices from a cluster with a tie-breaker - #7085

Merged
rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-tiebreaker-no-reuse
Nov 13, 2026
Merged

rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-tiebreaker-no-reuse

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

EVE does not support changing or removing the tie-breaker of a formed
cluster. Its setup (node labels, cordon, replica counts, nodeSelectors)
stays in place after a later config drops or moves the tie-breaker, and
after the tie-breaker leaves the cluster. When evetest reuses such
devices for the next test, that test inherits the old tie-breaker setup
whatever tie-breaker it asks for. Either the old tie-breaker keeps its
labels, or a node that rejoins gets no label. In both cases
longhorn-manager never runs on that node, so volumes cannot be created
or attached there.

This is behind recurring nightly failures:

  • TestClusterVMApp, after TestTieBreakerCluster and
    TestClusterToSingleConversion (failed in each of the last 5 nightlies)
  • TestClusterPVCGCDeletesOnlyOrphans, after TestDNIDandBackupDNID

The next test's tie-breaker is only known once it builds its cluster
config, after its devices are set up. So evetest now does not reuse
devices whose last cluster config named a tie-breaker at all, and the
affected tests form their clusters on fresh devices.

How to test and validate this PR

Run the cluster test suite, or just a test pair from above in sequence
(e.g. a suite with TestDNIDandBackupDNID followed by
TestClusterPVCGCDeletesOnlyOrphans). The evetest log shows
Device "..." was in a cluster with a tie-breaker; not reusing devices
before the second test, which then passes on fresh devices.

Changelog notes

No user-facing changes.

PR Backports

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them. Test-framework-only change, no documentation needed; it
    will be validated by the nightly evetest runs.

🤖 Generated with Claude Code

EVE does not support changing or removing the tie-breaker of a formed
cluster. Its setup (node labels, cordon, replica counts, nodeSelectors)
stays in place after a later config drops or moves the tie-breaker, and
after the tie-breaker leaves the cluster. A test that reuses such
devices inherits it whatever tie-breaker it asks for. The tie-breaker's
labels stay on it, or a node that rejoins gets no label, and in both
cases longhorn-manager never runs on that node, so volumes cannot be
created or attached there.

This fails TestClusterVMApp after TestTieBreakerCluster and
TestClusterToSingleConversion, and TestClusterPVCGCDeletesOnlyOrphans
after TestDNIDandBackupDNID.

The next test's tie-breaker is only known once it builds its cluster
config, after its devices are set up, so devices whose last cluster
config named a tie-breaker are not reused at all. Each affected
test then forms its cluster on fresh devices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa milan-zededa added stable Should be backported to stable release(s) testing New test or test framework changes labels Nov 11, 2026
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.57%. Comparing base (d5a49ec) to head (5d9017f).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7085      +/-   ##
==========================================
+ Coverage   29.08%   29.57%   +0.48%     
==========================================
  Files         550      561      +11     
  Lines       99453   101048    +1595     
==========================================
+ Hits        28926    29880     +954     
- Misses      68243    68927     +684     
+ Partials     2284     2241      -43     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene
rene merged commit c29ccf0 into lf-edge:master Nov 13, 2026
44 of 48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Evetest: Claude chose to skip 25 test/suite name(s) as unrelated to or redundant for this PR. Which ones and why: run summary.

Comment /evetest all to run the full suite, or /evetest <name> <name> ... to run only specific tests/suites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s) testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants