Repository navigation
evetest: bound pauses and keep the live-investigation watcher alive - #6989
Conversation
…tartup The watcher gave up if the evetest gRPC server was not reachable within 10 minutes of its launch, but the evetest image build that precedes it can take longer. A test failing after that paused forever, as nothing was left to investigate and resume it. Wait for as long as the shell running the evetest step is alive instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
A test paused at a checkpoint or on failure waits until someone runs 'evetest continue'. When nobody does, e.g. in CI, the run hangs until the job times out. EVETEST_PAUSE_LIMIT=<seconds> makes the harness resume on its own after that long, as if 'evetest continue' was issued. The default of 0 keeps waiting indefinitely. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Add a pause-limit input to the suite action, passed as EVETEST_PAUSE_LIMIT whenever pause-on-failure is enabled, so a paused test cannot hold the runner when the live investigation does not resume it. Cap one investigation below that limit so its report is written before the forced resume. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
af7f284 to
c768eb5
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6989 +/- ##
==========================================
+ Coverage 28.78% 29.07% +0.28%
==========================================
Files 544 555 +11
Lines 98795 100542 +1747
==========================================
+ Hits 28438 29229 +791
- Misses 68069 68830 +761
- Partials 2288 2483 +195 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
evetest: all suites skipped for this PRThis PR only touches evetest's own CI/pause-resume plumbing — not EVE device
The PR's own test plan validates this with dedicated Go unit tests
The nightly job re-runs the full suite matrix against the target branch An engineer can comment |
Description
A test paused on failure (
EVETEST_PAUSE_ON_FAILURE, used by the CI liveinvestigation) can stay paused forever. Observed on a PR run: the
live-investigation watcher gave up after 10 minutes because the evetest image
build before the gRPC server comes up took longer (~14 minutes). A test
failing afterwards paused with nothing left to resume it, and the job hung
until someone ran
evetest continueby hand.This PR contains:
gRPC server for as long as the shell running the evetest step is alive.
EVETEST_PAUSE_LIMIT=<seconds>option: the harness resumes a pause ata checkpoint or on failure on its own after that long, as if
evetest continuewas run. Default0waits indefinitely, as before.pause-limitinput (default 1200s), passed asEVETEST_PAUSE_LIMITwheneveranalyze-failuresis enabled. It applies toboth the PR and the nightly workflows. One Claude investigation is capped at
18 minutes, below that limit, so the report is written before a forced
resume.
PR dependencies
None.
How to test and validate this PR
go test ./evetest/(covers thelimit timer and that a stale expiry does not release a newer pause).
EVETEST_PAUSE_ON_FAILURE=true EVETEST_PAUSE_LIMIT=30and confirm the log showsPaused on failure,then
Pause limit of 30s reached on failure, resumingand the test ends.a long image build.
Changelog notes
None. CI and test tooling only.
PR Backports
The evetest CI workflow files will be backported as a whole once confirmed to
work reliably.
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
Unchecked items do not apply: CI and test tooling only, no device impact. The
change has not yet been exercised in a CI run.
🤖 Generated with Claude Code