Skip to content

evetest: integrate PR and nightly CI workflows with AI-assisted test selection and triage - #6680

Merged
milan-zededa merged 5 commits into
lf-edge:masterfrom
milan-zededa:evetest-ci-pr
Oct 28, 2026
Merged

milan-zededa merged 5 commits into
lf-edge:masterfrom
milan-zededa:evetest-ci-pr

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds evetest (the self-hosted, EVE-device-level test framework already in
evetest/) as an actual GitHub Actions CI pipeline, with two workflows and
AI-assisted test selection/triage in between:

  • evetest-pr.yml — runs on a pull request once the PR build has
    succeeded and either the PR is approved (whoever its author is) or a
    CODEOWNER has requested a run with an /evetest comment. Merely opening a
    PR starts nothing. By default Claude is given the PR diff and the full
    list of evetest suites/tests up front and decides which suites are safe to
    skip for that specific PR (unrelated to the change, or redundant with
    another kept test). A CODEOWNER controls what runs with a PR comment:

    • /evetest: run now, with Claude deciding what to skip
    • /evetest all: run every suite
    • /evetest <name> <name> ...: run only the named suites, tests or
      variants (.github/scripts/evetest_select.sh turns the names into the
      suite matrix and an EVETEST_SKIP list; an unknown name fails the run)

    The comment is picked up by rerun-ci.yml, which re-runs the PR Gate. The
    gate (pr-gate.yml) reads the latest /evetest comment from a CODEOWNER
    (CODEOWNERS taken from the PR's base branch, never from the PR's own tree),
    decides whether evetest may start, and hands the run mode and arguments to
    evetest-pr.yml in a separate evetest-run-context artifact. Eden's
    run-context.json and its approval requirement are unchanged.
    Each remaining suite runs as its own matrix
    job on a self-hosted runner; on failure, the device/SDN VMs are kept alive
    (EVETEST_PAUSE_ON_FAILURE) and Claude investigates live over evetest's own
    gRPC control plane (status, logs, collect-info, SSH into EVE/SDN) rather
    than post-mortem from collected artifacts, then posts a root-cause writeup
    as a PR comment.

  • evetest-nightly.yml — runs every suite (no PR-based filtering) against
    the tip of master/an LTS branch on a schedule, as the full regression net
    that catches anything a PR run filtered out. Merges every suite's JUnit
    results into an Allure report (pass/fail trend and per-test flakiness
    history over time) and appends failures to a NIGHTLY-LEDGER.md, so a
    recurring failure is recognized as "known since " instead of being
    re-investigated from scratch on every run. The live-investigation mechanism
    is shared with the PR workflow. Optionally builds a COVER=y image and
    merges Go coverage from every suite into one HTML report, published
    alongside the Allure report and ledger on GitHub Pages.

Both workflows execute suites via a shared evetest-run-suite composite
action (checkout/image loading, running the suite, formatting/uploading
results, the pause-on-failure live investigation) and are fanned out across
self-hosted runners labelled evetest-runner, one suite per runner at a
time (strategy.fail-fast: false), so PR/nightly throughput scales with the
number of registered runners.

Validated end-to-end on a fork (see "How to test" below) before opening this
PR against lf-edge/eve.

How to test and validate this PR

This PR only adds CI wiring (workflows + a composite action); evetest
itself is pre-existing and already unit-tested (evetest/skip_test.go for
the new EVETEST_SKIP logic added here, go test ./evetest/...).

To validate the workflow integration itself, at least one self-hosted runner
with the evetest-runner label (and an evetest device broker it can reach) is
required; the intent is that infra maintainers validate on real hardware
before/while this is enabled for lf-edge/eve:

  1. Register a self-hosted runner with the evetest-runner label.
  2. Open a PR against a branch with this workflow present: confirm
    evetest-pr does not start while the PR is unapproved, and triggers after
    the PR build succeeds and the PR is approved, that Claude posts a
    skip-decision PR comment (or none, if it found nothing safe to skip), and
    that each non-skipped suite reports its own Evetest / <suite> commit
    status. On an unapproved PR, confirm a CODEOWNER's /evetest comment
    starts the Claude-decided run, /evetest all runs the full suite, and
    /evetest <name> ... runs only the named tests/suites (and a comment from
    a non-CODEOWNER does nothing).
  3. Force a suite to fail (e.g. temporarily break a test) and confirm
    EVETEST_PAUSE_ON_FAILURE keeps the device alive and a root-cause
    analysis comment is posted.
  4. Manually dispatch evetest-nightly.yml (optionally with
    collect-coverage) and confirm the Allure report, NIGHTLY-LEDGER.md and
    (if requested) the coverage report are published to the gh-pages branch.

This was exercised end-to-end on a personal fork before opening this PR:

  • Example PR run: github.com/milan-zededa/eve/pull/46
  • Example nightly run: github.com/milan-zededa/eve/actions/runs/35042303584
  • Published reports: milan-zededa.github.io/eve/test/master/ (Allure),
    milan-zededa.github.io/eve/test/master/coverage/ (coverage), and the
    ledger at github.com/milan-zededa/eve/blob/gh-pages/test/master/NIGHTLY-LEDGER.md

Changelog notes

No user-facing changes (CI/tooling only).

PR Backports

These add new GitHub Actions workflows/self-hosted-runner wiring specific to
this repo's own CI, not application code, so none of the four backports can
be a straight cherry-pick — each will need adjusting to whatever CI files
already exist on that branch (e.g. pr-gate.yml's current shape there) at
backport time.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I
    didn't check them: this PR adds CI/infrastructure workflows, not
    device-facing code, so "tested on amd64/arm64 device" doesn't apply in
    the usual sense — see "How to test and validate this PR" instead for
    how this was actually exercised (a fork run + the manual validation
    steps needed on self-hosted runners before enabling it here). A
    documentation writeup can follow as a separate PR once the workflows
    are confirmed stable on lf-edge/eve's own runners.

@milan-zededa milan-zededa added stable Should be backported to stable release(s) testing New test or test framework changes labels Sep 30, 2026
@codecov

codecov Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.77%. Comparing base (8d07232) to head (ecf1be4).
⚠️ Report is 10 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6680      +/-   ##
==========================================
+ Coverage   28.48%   28.77%   +0.29%     
==========================================
  Files         540      551      +11     
  Lines       98258   100005    +1747     
==========================================
+ Hits        27985    28774     +789     
- Misses      68003    68766     +763     
- Partials     2270     2465     +195     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@milan-zededa

Copy link
Copy Markdown
Contributor Author

@claude

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @milan-zededa's task in 8m 28s —— View job


Review complete

I reviewed the CI wiring (workflows + composite action + shell scripts) and the evetest Go changes. The Go changes (skip.go, harness.go, testsuite.go, list-tests, constants) are clean and well-tested. Findings below were posted as inline comments.

  • Read the PR diff
  • Review workflows & composite action (shell/YAML)
  • Review evetest Go changes (skip logic, harness)
  • Post inline review comments

Inline findings

  1. go install ...@latest unpinned (evetest-run-suite/action.yml:181, :307) — reproducibility + supply-chain risk; a format change in gotestfmt/gotestsum would silently break result parsing.
  2. shift-headings.sh rewrites # lines inside fenced code blocks — corrupts shell/log snippets in Claude's analysis when posted to the PR/ledger.
  3. evetest-pr.yml concurrency fallback to workflow_run.id — for fork PRs pull_requests[] is empty, so cancel-in-progress never cancels a prior commit's suites, wasting one-per-runner self-hosted capacity.
  4. merge-and-report-coverage.sh adds a dir to CLEAN_DIRS before the covmeta.* check — a meta-less dir reaches covdata merge, which discards all input on the first unparsable pod, defeating the quarantine logic.

Cross-cutting note (no change required, by design)

The PR-flow evetest/discover-suites jobs check out and execute fork-controlled code (make, Dockerfiles, the full suite) on the self-hosted evetest-runner fleet, within a workflow_run context where repo secrets (ANTHROPIC_API_KEY, GITHUB_TOKEN) are reachable. This is gated behind build-success + approval/CODEOWNER /evetest, and the gate correctly reads CODEOWNERS from the base branch (pr-gate.yml) with a double-check at both rerun-ci.yml and pr-gate.yml — a solid design. Worth keeping in mind that approval is the only barrier before arbitrary fork code runs with those secrets present in the environment; ensure approvals are treated accordingly. (Minor: rerun-ci.yml's CODEOWNER match is case-sensitive while pr-gate.yml's is case-insensitive — harmless but inconsistent.)
· branch evetest-ci-pr

Comment thread .github/workflows/evetest-pr.yml Resolved
@milan-zededa
milan-zededa force-pushed the evetest-ci-pr branch 2 times, most recently from 34f456e to 0fb1486 Compare October 9, 2026 15:27
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@claude

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @milan-zededa's task in 3m 14s —— View job


Re-review after latest fixes

I re-read the full diff (git diff origin/master...HEAD) with a focus on the four earlier findings and any new regressions.

  • Read current PR diff
  • Verify the four earlier findings are resolved
  • Review for new/remaining issues
  • Post findings

Earlier findings — all resolved

  1. Unpinned go install …@latest — now gotestfmt@v2.5.0 and gotestsum@v1.13.0 (evetest-run-suite/action.yml). ✅
  2. shift-headings.sh rewriting # lines inside code fences — now tracks ```/~~~ fences in a shared awk fragment, skipping fenced lines both when detecting the shallowest heading and when shifting. Fence-toggle logic is correct. ✅
  3. Concurrency fallback to workflow_run.id for fork PRs — now keyed on head_repository.full_name:head_branch, matching pr-gate.yml, so a new commit//evetest on a fork PR cancels the prior run. ✅
  4. merge-and-report-coverage.sh appending to CLEAN_DIRS before the covmeta.* check — the append now happens only after the [ -n "$META" ] guard, so a meta-less dir never reaches covdata merge. ✅

Additionally, the cross-cutting note from last time about case-sensitivity inconsistency between rerun-ci.yml and pr-gate.yml is now fixed — rerun-ci.yml lowercases both the CODEOWNERS list (tolower(...)) and the comment author (${COMMENT_USER,,}), consistent with pr-gate.yml.

New review

No new blocking issues found. Notes:

  • The security posture is unchanged and as designed: the PR-flow discover-suites/evetest jobs check out and run fork-controlled code (Docker image build, the suites) with repo secrets reachable, gated behind build-success + approval/CODEOWNER /evetest, with CODEOWNERS read from the base branch. Approval remains the sole barrier before fork code runs with those secrets — worth continuing to treat approvals accordingly.
  • Minor/non-blocking: in the Claude-decided skip path (evetest-pr.yml "Determine EVETEST_SKIP"), skip-decision.json contents are written straight to $GITHUB_OUTPUT via echo "skip=$SKIP" without the character validation that the /evetest <name> (select) path applies. A multi-line value would break the output; in practice the names are single-line and later only split on commas by the Go skip logic, so this is low risk — just inconsistent with the validated select path.

The evetest Go changes (skip.go/skip_test.go, harness.go parent-name matching, testsuite.go, list-tests -suites-only, constants) are clean and well-covered by skip_test.go.

LGTM — the four findings are correctly addressed.
· branch evetest-ci-pr

@milan-zededa
milan-zededa force-pushed the evetest-ci-pr branch 3 times, most recently from 6d0b97f to fb7fd59 Compare October 12, 2026 15:10

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kicking off tests

@milan-zededa

Copy link
Copy Markdown
Contributor Author

@rene btw. while this is ready for review, I'm keeping this as draft until we have runners prepared (didn't have time to work on that yet), so it does not get merged before we are ready.

@github-actions
github-actions Bot requested a review from rene October 13, 2026 13:49
@milan-zededa
milan-zededa force-pushed the evetest-ci-pr branch 2 times, most recently from 945927d to 16d37d7 Compare October 13, 2026 13:52
@milan-zededa
milan-zededa force-pushed the evetest-ci-pr branch 2 times, most recently from 1c31eae to c6f9048 Compare October 27, 2026 09:17
milan-zededa and others added 3 commits October 27, 2026 16:09
Adds evetest-pr.yml (per-PR run, against a freshly built-but-
unpublished PR image, gated on its build having succeeded for that
exact commit AND either the PR being approved or a CODEOWNER having
requested a run with an "/evetest" comment -- checked live by the PR
Gate, since a new build, a new review or a new comment can come first)
and evetest-nightly.yml (full run against a branch's tip, on a schedule/
manual dispatch, against whatever publish.yml already published for
that commit -- the full-coverage safety net for evetest-pr.yml's own
PR-time test skipping).

Shared per-suite execution lives in the evetest-run-suite composite
action: checkout/image loading, running the suite, formatting/
uploading results, and (when a suite fails) an EVETEST_PAUSE_ON_FAILURE
live investigation -- live-investigation-watcher.sh polls the suite
over evetest's own gRPC control plane and has Claude inspect the
still-running device/SDN directly (status, eve ssh/logs, collect-info,
sdn ssh) rather than post-mortem from collected artifacts, which often
can't find a root cause. Findings land in one claude-analysis.md per
suite, posted as a PR comment and prepended into a nightly failure
ledger (published alongside an Allure report on GitHub Pages) so a
recurring failure is recognized as such instead of re-investigated
from scratch every time.

On a PR, Claude is also asked up front which suites/tests can be
skipped as clearly unrelated or redundant with another already-kept
test, given the diff and each suite's own doc comments -- skipped
suites still get their own (immediately-resolved) status check rather
than never appearing at all.

A CODEOWNER controls what runs with a PR comment (rerun-ci.yml re-runs
the PR Gate, which reads the comment and tells evetest-pr.yml):
  /evetest               - run now, with Claude deciding what to skip
  /evetest all           - run every suite
  /evetest <name> ...    - run only the named suites/tests/variants
                           (select-tests.sh turns names into the
                           matrix and an EVETEST_SKIP list)

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
New collect-coverage workflow_dispatch input on evetest-nightly.yml:
when set, builds a COVER=y EVE image on the builder fleet (COVER=y
changes pillar's own package build, so it's a separate image, not a
toggle on the one publish.yml already published) and has every suite
collect coverage (EVETEST_COLLECT_COVERAGE), merging it all into one
overall per-package and total summary alongside each suite's own.

The line-by-line HTML report (source + coverage highlighting) is
published to gh-pages next to the Allure report; the raw merged
covdata is also uploaded as its own artifact for combining with other
coverage locally (see "make coverage-merge"). Left off by default,
since it rebuilds the image instead of pulling the published one.

Signed-off-by: Milan Lenco <milan@zededa.com>
Add trigger-evetest-nightly.sh, which dispatches evetest-nightly.yml via
the GitHub API so it can be driven by any external scheduler with
Internet access, instead of relying on GitHub's best-effort schedule
trigger.

The target repository and branch are mandatory arguments, and the token
is read from the GH_TOKEN environment variable rather than stored in the
script.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Add --registration-token, so a runner can be installed by someone who
was handed a short-lived registration token (from the GitHub UI or API)
instead of a PAT with repository Administration permission. --gh-pat is
now required only when no registration token is given.

With --destroy, --gh-pat becomes optional: without it the VM and its
cloud-init snippet are removed, but the runner stays registered on
GitHub and has to be removed there manually.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa marked this pull request as ready for review October 28, 2026 11:24
@milan-zededa

Copy link
Copy Markdown
Contributor Author

We now have a few evetest runners available, so we can merge this and start trying out evetest workflows.

Name the GitHub runner exactly as given by --name, and the VM after it
by default, instead of prefixing both with "gh-runner-". A new optional
--vm-name sets the Proxmox VM name separately; pass it again with
--destroy. The cloud-init snippet file keeps its prefix so it cannot
clash with other snippets.

Without the prefix, a VM name could match the evetest broker VM or its
per-test "ev-*" VMs, which --destroy would then remove, so reject those
names.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa merged commit e93c1b1 into lf-edge:master Oct 28, 2026
38 of 47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s) testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants