Skip to content

evetest: bound pauses and keep the live-investigation watcher alive - #6989

Merged
milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-pause-limit
Nov 1, 2026
Merged

milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-pause-limit

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

A test paused on failure (EVETEST_PAUSE_ON_FAILURE, used by the CI live
investigation) can stay paused forever. Observed on a PR run: the
live-investigation watcher gave up after 10 minutes because the evetest image
build before the gRPC server comes up took longer (~14 minutes). A test
failing afterwards paused with nothing left to resume it, and the job hung
until someone ran evetest continue by hand.

This PR contains:

  • the watcher no longer gives up on a fixed clock. It keeps waiting for the
    gRPC server for as long as the shell running the evetest step is alive.
  • new EVETEST_PAUSE_LIMIT=<seconds> option: the harness resumes a pause at
    a checkpoint or on failure on its own after that long, as if
    evetest continue was run. Default 0 waits indefinitely, as before.
  • the suite action gets a pause-limit input (default 1200s), passed as
    EVETEST_PAUSE_LIMIT whenever analyze-failures is enabled. It applies to
    both the PR and the nightly workflows. One Claude investigation is capped at
    18 minutes, below that limit, so the report is written before a forced
    resume.

PR dependencies

None.

How to test and validate this PR

  • Unit tests for the pause release logic: go test ./evetest/ (covers the
    limit timer and that a stale expiry does not release a newer pause).
  • Manually: run a failing test with EVETEST_PAUSE_ON_FAILURE=true EVETEST_PAUSE_LIMIT=30 and confirm the log shows Paused on failure,
    then Pause limit of 30s reached on failure, resuming and the test ends.
  • CI: a suite failing on a PR run no longer hangs the job; the watcher survives
    a long image build.

Changelog notes

None. CI and test tooling only.

PR Backports

The evetest CI workflow files will be backported as a whole once confirmed to
work reliably.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

Unchecked items do not apply: CI and test tooling only, no device impact. The
change has not yet been exercised in a CI run.

🤖 Generated with Claude Code

@github-actions
github-actions Bot requested a review from europaul October 30, 2026 16:05
@milan-zededa milan-zededa added the testing New test or test framework changes label Oct 30, 2026
milan-zededa and others added 3 commits October 31, 2026 21:02
…tartup

The watcher gave up if the evetest gRPC server was not reachable within
10 minutes of its launch, but the evetest image build that precedes it
can take longer. A test failing after that paused forever, as nothing
was left to investigate and resume it. Wait for as long as the shell
running the evetest step is alive instead.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
A test paused at a checkpoint or on failure waits until someone runs
'evetest continue'. When nobody does, e.g. in CI, the run hangs until
the job times out. EVETEST_PAUSE_LIMIT=<seconds> makes the harness
resume on its own after that long, as if 'evetest continue' was issued.
The default of 0 keeps waiting indefinitely.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Add a pause-limit input to the suite action, passed as EVETEST_PAUSE_LIMIT
whenever pause-on-failure is enabled, so a paused test cannot hold the
runner when the live investigation does not resume it. Cap one
investigation below that limit so its report is written before the forced
resume.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.07%. Comparing base (203e89f) to head (c768eb5).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6989      +/-   ##
==========================================
+ Coverage   28.78%   29.07%   +0.28%     
==========================================
  Files         544      555      +11     
  Lines       98795   100542    +1747     
==========================================
+ Hits        28438    29229     +791     
- Misses      68069    68830     +761     
- Partials     2288     2483     +195     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

evetest: all suites skipped for this PR

This PR only touches evetest's own CI/pause-resume plumbing — not EVE device
behavior:

  • evetest/harness.go, evetest/grpcserver.go, evetest/testing.go: a new
    EVETEST_PAUSE_LIMIT that bounds how long a test stays paused at a
    checkpoint or on failure. The new code paths (pauseLimitTimer,
    releasePause) only run inside the existing shouldPause branch, which is
    off by default and unrelated to any device/app/network/storage behavior
    under test.
  • .github/actions/evetest-run-suite/live-investigation-watcher.sh and
    action.yml: the background live-investigation watcher now tracks the
    owning shell's PID instead of a fixed 10-minute startup deadline, and wires
    through the new pause-limit input.
  • No pkg/ or device-facing code is touched.

The PR's own test plan validates this with dedicated Go unit tests
(go test ./evetest/, see evetest/pauselimit_test.go), which already cover
the pause-limit timer and stale-expiry handling. None of the functional
suites below exercise checkpoint/failure pausing or the CI watcher script, so
they provide no additional regression coverage for this change:

  • TestApplicationConnectivitySuite
  • TestAppsSuite
  • TestBootstrapSuite
  • TestControllerFaultsSuite
  • TestDatastoreSuite
  • TestDeviceConnectivitySuite
  • TestDiagSuite
  • TestHardwareSuite
  • TestKonvertSuite
  • TestKubeRegistrationSuite
  • TestLPSSuite
  • TestNodeClusterResilienceSuite
  • TestNodeClusterSuite
  • TestPcibackErrorSuite
  • TestPortScanSuite
  • TestSecuritySuite
  • TestStorageSuite
  • TestTelemetrySuite
  • TestUpgradeSuite
  • TestKvmToKRepartitionAppVolume
  • TestKvmToKRepartitionVolmig
  • TestPVCUploadReport
  • TestVaultKeyKeptOnTPMError

The nightly job re-runs the full suite matrix against the target branch
regardless, so this skip doesn't reduce long-term coverage.

An engineer can comment /evetest all to force the full suite to run
instead, or /evetest <name> <name> ... to run only specific tests/suites.

@milan-zededa
milan-zededa merged commit 9c4eefa into lf-edge:master Nov 1, 2026
75 of 78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants