Repository navigation
evetest: integrate PR and nightly CI workflows with AI-assisted test selection and triage - #6680
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6680 +/- ##
==========================================
+ Coverage 28.48% 28.77% +0.29%
==========================================
Files 540 551 +11
Lines 98258 100005 +1747
==========================================
+ Hits 27985 28774 +789
- Misses 68003 68766 +763
- Partials 2270 2465 +195 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
d25d014 to
485293f
Compare
485293f to
886a917
Compare
1af1012 to
2457a6f
Compare
|
Claude finished @milan-zededa's task in 8m 28s —— View job Review completeI reviewed the CI wiring (workflows + composite action + shell scripts) and the
Inline findings
Cross-cutting note (no change required, by design)The PR-flow |
34f456e to
0fb1486
Compare
|
Claude finished @milan-zededa's task in 3m 14s —— View job Re-review after latest fixesI re-read the full diff (
Earlier findings — all resolved
Additionally, the cross-cutting note from last time about case-sensitivity inconsistency between New reviewNo new blocking issues found. Notes:
The LGTM — the four findings are correctly addressed. |
6d0b97f to
fb7fd59
Compare
|
@rene btw. while this is ready for review, I'm keeping this as draft until we have runners prepared (didn't have time to work on that yet), so it does not get merged before we are ready. |
fb7fd59 to
2b9c444
Compare
945927d to
16d37d7
Compare
16d37d7 to
dfb1f6c
Compare
1c31eae to
c6f9048
Compare
Adds evetest-pr.yml (per-PR run, against a freshly built-but-
unpublished PR image, gated on its build having succeeded for that
exact commit AND either the PR being approved or a CODEOWNER having
requested a run with an "/evetest" comment -- checked live by the PR
Gate, since a new build, a new review or a new comment can come first)
and evetest-nightly.yml (full run against a branch's tip, on a schedule/
manual dispatch, against whatever publish.yml already published for
that commit -- the full-coverage safety net for evetest-pr.yml's own
PR-time test skipping).
Shared per-suite execution lives in the evetest-run-suite composite
action: checkout/image loading, running the suite, formatting/
uploading results, and (when a suite fails) an EVETEST_PAUSE_ON_FAILURE
live investigation -- live-investigation-watcher.sh polls the suite
over evetest's own gRPC control plane and has Claude inspect the
still-running device/SDN directly (status, eve ssh/logs, collect-info,
sdn ssh) rather than post-mortem from collected artifacts, which often
can't find a root cause. Findings land in one claude-analysis.md per
suite, posted as a PR comment and prepended into a nightly failure
ledger (published alongside an Allure report on GitHub Pages) so a
recurring failure is recognized as such instead of re-investigated
from scratch every time.
On a PR, Claude is also asked up front which suites/tests can be
skipped as clearly unrelated or redundant with another already-kept
test, given the diff and each suite's own doc comments -- skipped
suites still get their own (immediately-resolved) status check rather
than never appearing at all.
A CODEOWNER controls what runs with a PR comment (rerun-ci.yml re-runs
the PR Gate, which reads the comment and tells evetest-pr.yml):
/evetest - run now, with Claude deciding what to skip
/evetest all - run every suite
/evetest <name> ... - run only the named suites/tests/variants
(select-tests.sh turns names into the
matrix and an EVETEST_SKIP list)
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
New collect-coverage workflow_dispatch input on evetest-nightly.yml: when set, builds a COVER=y EVE image on the builder fleet (COVER=y changes pillar's own package build, so it's a separate image, not a toggle on the one publish.yml already published) and has every suite collect coverage (EVETEST_COLLECT_COVERAGE), merging it all into one overall per-package and total summary alongside each suite's own. The line-by-line HTML report (source + coverage highlighting) is published to gh-pages next to the Allure report; the raw merged covdata is also uploaded as its own artifact for combining with other coverage locally (see "make coverage-merge"). Left off by default, since it rebuilds the image instead of pulling the published one. Signed-off-by: Milan Lenco <milan@zededa.com>
Add trigger-evetest-nightly.sh, which dispatches evetest-nightly.yml via the GitHub API so it can be driven by any external scheduler with Internet access, instead of relying on GitHub's best-effort schedule trigger. The target repository and branch are mandatory arguments, and the token is read from the GH_TOKEN environment variable rather than stored in the script. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
c6f9048 to
27c3711
Compare
Add --registration-token, so a runner can be installed by someone who was handed a short-lived registration token (from the GitHub UI or API) instead of a PAT with repository Administration permission. --gh-pat is now required only when no registration token is given. With --destroy, --gh-pat becomes optional: without it the VM and its cloud-init snippet are removed, but the runner stays registered on GitHub and has to be removed there manually. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
|
We now have a few evetest runners available, so we can merge this and start trying out evetest workflows. |
Name the GitHub runner exactly as given by --name, and the VM after it by default, instead of prefixing both with "gh-runner-". A new optional --vm-name sets the Proxmox VM name separately; pass it again with --destroy. The cloud-init snippet file keeps its prefix so it cannot clash with other snippets. Without the prefix, a VM name could match the evetest broker VM or its per-test "ev-*" VMs, which --destroy would then remove, so reject those names. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
a1eb08c to
ecf1be4
Compare
Description
Adds
evetest(the self-hosted, EVE-device-level test framework already inevetest/) as an actual GitHub Actions CI pipeline, with two workflows andAI-assisted test selection/triage in between:
evetest-pr.yml— runs on a pull request once the PR build hassucceeded and either the PR is approved (whoever its author is) or a
CODEOWNER has requested a run with an
/evetestcomment. Merely opening aPR starts nothing. By default Claude is given the PR diff and the full
list of evetest suites/tests up front and decides which suites are safe to
skip for that specific PR (unrelated to the change, or redundant with
another kept test). A CODEOWNER controls what runs with a PR comment:
/evetest: run now, with Claude deciding what to skip/evetest all: run every suite/evetest <name> <name> ...: run only the named suites, tests orvariants (
.github/scripts/evetest_select.shturns the names into thesuite matrix and an
EVETEST_SKIPlist; an unknown name fails the run)The comment is picked up by
rerun-ci.yml, which re-runs the PR Gate. Thegate (
pr-gate.yml) reads the latest/evetestcomment from a CODEOWNER(CODEOWNERS taken from the PR's base branch, never from the PR's own tree),
decides whether evetest may start, and hands the run mode and arguments to
evetest-pr.ymlin a separateevetest-run-contextartifact. Eden'srun-context.jsonand its approval requirement are unchanged.Each remaining suite runs as its own matrix
job on a self-hosted runner; on failure, the device/SDN VMs are kept alive
(
EVETEST_PAUSE_ON_FAILURE) and Claude investigates live over evetest's owngRPC control plane (status, logs,
collect-info, SSH into EVE/SDN) ratherthan post-mortem from collected artifacts, then posts a root-cause writeup
as a PR comment.
evetest-nightly.yml— runs every suite (no PR-based filtering) againstthe tip of master/an LTS branch on a schedule, as the full regression net
that catches anything a PR run filtered out. Merges every suite's JUnit
results into an Allure report (pass/fail trend and per-test flakiness
history over time) and appends failures to a
NIGHTLY-LEDGER.md, so arecurring failure is recognized as "known since " instead of being
re-investigated from scratch on every run. The live-investigation mechanism
is shared with the PR workflow. Optionally builds a
COVER=yimage andmerges Go coverage from every suite into one HTML report, published
alongside the Allure report and ledger on GitHub Pages.
Both workflows execute suites via a shared
evetest-run-suitecompositeaction (checkout/image loading, running the suite, formatting/uploading
results, the pause-on-failure live investigation) and are fanned out across
self-hosted runners labelled
evetest-runner, one suite per runner at atime (
strategy.fail-fast: false), so PR/nightly throughput scales with thenumber of registered runners.
Validated end-to-end on a fork (see "How to test" below) before opening this
PR against
lf-edge/eve.How to test and validate this PR
This PR only adds CI wiring (workflows + a composite action);
evetestitself is pre-existing and already unit-tested (
evetest/skip_test.goforthe new
EVETEST_SKIPlogic added here,go test ./evetest/...).To validate the workflow integration itself, at least one self-hosted runner
with the
evetest-runnerlabel (and an evetest device broker it can reach) isrequired; the intent is that infra maintainers validate on real hardware
before/while this is enabled for
lf-edge/eve:evetest-runnerlabel.evetest-prdoes not start while the PR is unapproved, and triggers afterthe PR build succeeds and the PR is approved, that Claude posts a
skip-decision PR comment (or none, if it found nothing safe to skip), and
that each non-skipped suite reports its own
Evetest / <suite>commitstatus. On an unapproved PR, confirm a CODEOWNER's
/evetestcommentstarts the Claude-decided run,
/evetest allruns the full suite, and/evetest <name> ...runs only the named tests/suites (and a comment froma non-CODEOWNER does nothing).
EVETEST_PAUSE_ON_FAILUREkeeps the device alive and a root-causeanalysis comment is posted.
evetest-nightly.yml(optionally withcollect-coverage) and confirm the Allure report,NIGHTLY-LEDGER.mdand(if requested) the coverage report are published to the
gh-pagesbranch.This was exercised end-to-end on a personal fork before opening this PR:
github.com/milan-zededa/eve/pull/46github.com/milan-zededa/eve/actions/runs/35042303584milan-zededa.github.io/eve/test/master/(Allure),milan-zededa.github.io/eve/test/master/coverage/(coverage), and theledger at
github.com/milan-zededa/eve/blob/gh-pages/test/master/NIGHTLY-LEDGER.mdChangelog notes
No user-facing changes (CI/tooling only).
PR Backports
These add new GitHub Actions workflows/self-hosted-runner wiring specific to
this repo's own CI, not application code, so none of the four backports can
be a straight cherry-pick — each will need adjusting to whatever CI files
already exist on that branch (e.g.
pr-gate.yml's current shape there) atbackport time.
Checklist
didn't check them: this PR adds CI/infrastructure workflows, not
device-facing code, so "tested on amd64/arm64 device" doesn't apply in
the usual sense — see "How to test and validate this PR" instead for
how this was actually exercised (a fork run + the manual validation
steps needed on self-hosted runners before enabling it here). A
documentation writeup can follow as a separate PR once the workflows
are confirmed stable on
lf-edge/eve's own runners.