Repository navigation
evetest: fix pause-on-failure hang and bound live-investigation watcher - #6978
Merged
Merged
Conversation
With EVETEST_PAUSE_ON_FAILURE, t.Errorf/Fatalf block until the test is resumed. Several places called them while holding devicesM or netModelM, so the Status RPC (devicesM) and SSH-based CLI commands (netModelM) hung for the whole pause and nothing could observe the failure or resume the test. Report failures only after releasing the lock in checkRebootCounts (all mismatches in a single error), teardownDevices and UpdateNetworkModel. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
evetest status had no timeout, so one wedged Status RPC stalled the watcher silently until the job hit its 6h limit. Time out status, continue and the Claude investigation, and force-resume the test after repeated status timeouts so a paused run cannot hang indefinitely. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
milan-zededa
requested review from
eriknordmark,
rene,
rucoder,
shjala and
yash-zededa
as code owners
October 30, 2026 07:59
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6978 +/- ##
==========================================
+ Coverage 28.73% 29.03% +0.29%
==========================================
Files 544 555 +11
Lines 98600 100347 +1747
==========================================
+ Hits 28334 29131 +797
- Misses 67979 68733 +754
- Partials 2287 2483 +196 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
rene
approved these changes
Oct 30, 2026
Contributor
|
evetest, no Eden Tests needed. |
Merged
6 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With
EVETEST_PAUSE_ON_FAILURE, a failing test pauses with its devices stilllive so that the CI live-investigation watcher can inspect them through the
evetestCLI and then resume the test. In a nightly run ofTestNodeClusterSuitethis never happened: the test failed in the teardown reboot-count check, paused,
and the job sat there until the 6h limit.
Root cause:
checkRebootCountscalledt.Errorfwhile holdingdevicesM.Errorfblocks until the test is resumed, and theStatusRPC also takesdevicesM, soevetest statushung forever. The watcher called it with notimeout, so it never saw the failure, never investigated and never resumed.
This PR:
checkRebootCounts(all mismatches in a single error),
teardownDevicesandUpdateNetworkModel.These are the places where a realistic, environmental failure was raised under
devicesM/netModelM. Sites that fail only on test programming errors(unknown device, not onboarded, ...) are left as is.
live-investigation-watcher.sh:evetest statusand
evetest continueget a 10s timeout, and a single Claude investigation iscapped at 30 minutes. A status timeout is no longer mistaken for "run
finished", and after 3 consecutive timeouts the watcher force-resumes the test,
so a paused run cannot hang indefinitely.
How to test and validate this PR
Run any evetest suite (or a single test) with
EVETEST_PAUSE_ON_FAILURE=trueand make it fail at teardown, e.g. by inducing an unexpected device reboot so
that
checkRebootCountsreports a mismatch:evetest statusmust return promptly and show the failure text instead ofhanging.
evetest continuemust resume the test.new failing test: ...and append tolive-investigation.md.To check the watcher fallback on its own, make
evetest statushang (e.g. holddevicesMin a debugger) and verify that after about 45s the watcher logsevetest status is stuck; force-resuming the test.No automated test covers this; the pause/investigation flow only runs in the
CI workflows.
Changelog notes
None. Test framework and CI tooling only; no user-facing changes.
PR Backports
evetest changes are not backported individually. All evetest changes done on
master will be backported together once they are confirmed working.
Checklist
And the last but not least:
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
Device boxes are unchecked: this only touches the evetest harness and CI
scripts, nothing runs on EVE devices.
🤖 Generated with Claude Code