Skip to content

evetest: fix pause-on-failure hang and bound live-investigation watcher - #6978

Merged
rene merged 2 commits into
lf-edge:masterfrom
milan-zededa:evetest-pause-status-hang
Oct 30, 2026
Merged

rene merged 2 commits into
lf-edge:masterfrom
milan-zededa:evetest-pause-status-hang

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

With EVETEST_PAUSE_ON_FAILURE, a failing test pauses with its devices still
live so that the CI live-investigation watcher can inspect them through the
evetest CLI and then resume the test. In a nightly run of TestNodeClusterSuite
this never happened: the test failed in the teardown reboot-count check, paused,
and the job sat there until the 6h limit.

Root cause: checkRebootCounts called t.Errorf while holding devicesM.
Errorf blocks until the test is resumed, and the Status RPC also takes
devicesM, so evetest status hung forever. The watcher called it with no
timeout, so it never saw the failure, never investigated and never resumed.

This PR:

  • Reports failures only after releasing the harness mutex in checkRebootCounts
    (all mismatches in a single error), teardownDevices and UpdateNetworkModel.
    These are the places where a realistic, environmental failure was raised under
    devicesM / netModelM. Sites that fail only on test programming errors
    (unknown device, not onboarded, ...) are left as is.
  • Bounds every blocking call in live-investigation-watcher.sh: evetest status
    and evetest continue get a 10s timeout, and a single Claude investigation is
    capped at 30 minutes. A status timeout is no longer mistaken for "run
    finished", and after 3 consecutive timeouts the watcher force-resumes the test,
    so a paused run cannot hang indefinitely.

How to test and validate this PR

Run any evetest suite (or a single test) with EVETEST_PAUSE_ON_FAILURE=true
and make it fail at teardown, e.g. by inducing an unexpected device reboot so
that checkRebootCounts reports a mismatch:

  • evetest status must return promptly and show the failure text instead of
    hanging.
  • evetest continue must resume the test.
  • The live-investigation watcher must log new failing test: ... and append to
    live-investigation.md.

To check the watcher fallback on its own, make evetest status hang (e.g. hold
devicesM in a debugger) and verify that after about 45s the watcher logs
evetest status is stuck; force-resuming the test.

No automated test covers this; the pause/investigation flow only runs in the
CI workflows.

Changelog notes

None. Test framework and CI tooling only; no user-facing changes.

PR Backports

evetest changes are not backported individually. All evetest changes done on
master will be backported together once they are confirmed working.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

    Device boxes are unchecked: this only touches the evetest harness and CI
    scripts, nothing runs on EVE devices.

🤖 Generated with Claude Code

milan-zededa and others added 2 commits October 30, 2026 08:53
With EVETEST_PAUSE_ON_FAILURE, t.Errorf/Fatalf block until the test is
resumed. Several places called them while holding devicesM or netModelM,
so the Status RPC (devicesM) and SSH-based CLI commands (netModelM) hung
for the whole pause and nothing could observe the failure or resume the
test.

Report failures only after releasing the lock in checkRebootCounts (all
mismatches in a single error), teardownDevices and UpdateNetworkModel.

Signed-off-by: Milan Lenco <milan@zededa.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
evetest status had no timeout, so one wedged Status RPC stalled the
watcher silently until the job hit its 6h limit. Time out status,
continue and the Claude investigation, and force-resume the test after
repeated status timeouts so a paused run cannot hang indefinitely.

Signed-off-by: Milan Lenco <milan@zededa.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@milan-zededa milan-zededa added the testing New test or test framework changes label Oct 30, 2026
@github-actions
github-actions Bot requested a review from europaul October 30, 2026 07:59
@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.03%. Comparing base (3508904) to head (71ffc6b).
⚠️ Report is 22 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6978      +/-   ##
==========================================
+ Coverage   28.73%   29.03%   +0.29%     
==========================================
  Files         544      555      +11     
  Lines       98600   100347    +1747     
==========================================
+ Hits        28334    29131     +797     
- Misses      67979    68733     +754     
- Partials     2287     2483     +196     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene

rene commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

evetest, no Eden Tests needed.

@rene
rene merged commit c1afd1c into lf-edge:master Oct 30, 2026
45 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants