Repository navigation
[17.0-stable] evetest: add the evetest CI workflows and the fixes made to them on master - #7111
Conversation
Adds evetest-pr.yml (per-PR run, against a freshly built-but-
unpublished PR image, gated on its build having succeeded for that
exact commit AND either the PR being approved or a CODEOWNER having
requested a run with an "/evetest" comment -- checked live by the PR
Gate, since a new build, a new review or a new comment can come first)
and evetest-nightly.yml (full run against a branch's tip, on a schedule/
manual dispatch, against whatever publish.yml already published for
that commit -- the full-coverage safety net for evetest-pr.yml's own
PR-time test skipping).
Shared per-suite execution lives in the evetest-run-suite composite
action: checkout/image loading, running the suite, formatting/
uploading results, and (when a suite fails) an EVETEST_PAUSE_ON_FAILURE
live investigation -- live-investigation-watcher.sh polls the suite
over evetest's own gRPC control plane and has Claude inspect the
still-running device/SDN directly (status, eve ssh/logs, collect-info,
sdn ssh) rather than post-mortem from collected artifacts, which often
can't find a root cause. Findings land in one claude-analysis.md per
suite, posted as a PR comment and prepended into a nightly failure
ledger (published alongside an Allure report on GitHub Pages) so a
recurring failure is recognized as such instead of re-investigated
from scratch every time.
On a PR, Claude is also asked up front which suites/tests can be
skipped as clearly unrelated or redundant with another already-kept
test, given the diff and each suite's own doc comments -- skipped
suites still get their own (immediately-resolved) status check rather
than never appearing at all.
A CODEOWNER controls what runs with a PR comment (rerun-ci.yml re-runs
the PR Gate, which reads the comment and tells evetest-pr.yml):
/evetest - run now, with Claude deciding what to skip
/evetest all - run every suite
/evetest <name> ... - run only the named suites/tests/variants
(select-tests.sh turns names into the
matrix and an EVETEST_SKIP list)
Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
(cherry picked from commit 259455f)
New collect-coverage workflow_dispatch input on evetest-nightly.yml: when set, builds a COVER=y EVE image on the builder fleet (COVER=y changes pillar's own package build, so it's a separate image, not a toggle on the one publish.yml already published) and has every suite collect coverage (EVETEST_COLLECT_COVERAGE), merging it all into one overall per-package and total summary alongside each suite's own. The line-by-line HTML report (source + coverage highlighting) is published to gh-pages next to the Allure report; the raw merged covdata is also uploaded as its own artifact for combining with other coverage locally (see "make coverage-merge"). Left off by default, since it rebuilds the image instead of pulling the published one. Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 63c7ff3)
Add trigger-evetest-nightly.sh, which dispatches evetest-nightly.yml via the GitHub API so it can be driven by any external scheduler with Internet access, instead of relying on GitHub's best-effort schedule trigger. The target repository and branch are mandatory arguments, and the token is read from the GH_TOKEN environment variable rather than stored in the script. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 6fc9331)
Add --registration-token, so a runner can be installed by someone who was handed a short-lived registration token (from the GitHub UI or API) instead of a PAT with repository Administration permission. --gh-pat is now required only when no registration token is given. With --destroy, --gh-pat becomes optional: without it the VM and its cloud-init snippet are removed, but the runner stays registered on GitHub and has to be removed there manually. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit edca7d1)
Name the GitHub runner exactly as given by --name, and the VM after it by default, instead of prefixing both with "gh-runner-". A new optional --vm-name sets the Proxmox VM name separately; pass it again with --destroy. The cloud-init snippet file keeps its prefix so it cannot clash with other snippets. Without the prefix, a VM name could match the evetest broker VM or its per-test "ev-*" VMs, which --destroy would then remove, so reject those names. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit e93c1b1)
A remote broker pulls the SDN image itself, so an SDN image built from an unmerged change was unavailable to it and every test failed. When SetupDevices can neither find nor pull the SDN image, the broker still fails the request with the pull error, now as a FailedPrecondition status carrying an ErrorInfo detail with a dedicated reason. A harness that recognizes it uploads its local copy through the new PushSDNContainerImage RPC and repeats the request, as it already does for the EVE container image. An older harness just fails with the pull error. The broker-side receive logic and the harness-side upload loop are shared between the two image types. make evetest now ensures the local SDN image also with a remote broker. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit fbf37c7)
Allow the nightly trigger to set the collect-coverage workflow input, so a cron can enable coverage collection on selected days only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit a273c3e)
The concurrency group of evetest-pr was built from the head_branch of the triggering PR Gate run. That is the PR branch for a review-triggered gate but the default branch for a workflow_run-triggered one, so the two triggers of a single PR landed in different groups (and both ran Claude to decide the skips and then the whole suite), while all workflow_run-triggered gates of unrelated PRs shared one group and cancelled each other. Drop the group and instead decide in discover-suites, before Claude is asked anything, using the PR number from the gate's run-context: - a trigger for a commit that another active run already tests is ignored; - a trigger for a new head commit of the PR cancels the active run of the older commit; - a trigger for a commit that is no longer the PR head is ignored. Active runs are recognised by an evetest-active-<pr>-<sha> artifact that each run uploads once it has passed the check; unlike a per-commit status it can be found after a force-push. A cancelled or timed out run leaves the suites that never got a runner with their "Queued" status, so add a cleanup job that turns the statuses of this run that are still pending into errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 389299a)
The "Report suite statuses" step posts one commit status per suite in a quick burst. A single 5xx from the GitHub status API aborted the whole discover-suites job and, with it, the evetest run. Retry with backoff on 5xx, 429 and 403 before giving up. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 91e37ce)
Claude used to post a new PR comment on every evetest run that skipped tests, so reruns piled up stale comments. Have the workflow post the summary instead: a marker-tagged comment is updated in place for the latest run, and removed when the latest run skips nothing (e.g. after "/evetest all"). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 33d2fb5)
Every failed run posted a new analysis comment, so reruns piled up stale ones. Tag the comment with a per-suite marker and, at the end of each run (unless cancelled), collapse the suite's previous comment: as resolved if the run passed, as outdated if it failed again. A new comment is posted only when the run failed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 6f6cd80)
With EVETEST_PAUSE_ON_FAILURE, t.Errorf/Fatalf block until the test is resumed. Several places called them while holding devicesM or netModelM, so the Status RPC (devicesM) and SSH-based CLI commands (netModelM) hung for the whole pause and nothing could observe the failure or resume the test. Report failures only after releasing the lock in checkRebootCounts (all mismatches in a single error), teardownDevices and UpdateNetworkModel. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> (cherry picked from commit bea59cb)
evetest status had no timeout, so one wedged Status RPC stalled the watcher silently until the job hit its 6h limit. Time out status, continue and the Claude investigation, and force-resume the test after repeated status timeouts so a paused run cannot hang indefinitely. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> (cherry picked from commit c1afd1c)
Status functions are only allowed in if: conditions, so the runner rejected the composite action at load time. Derive the minimize classifier from job.status in the shell instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit a12ab60)
Clicking a failed test in the nightly Allure report showed only gotestsum's generic "Failed" message plus the raw log trace; the real failure message and Claude's conclusion ended up only in the nightly ledger and the job's step summary. Inject both into the JUnit XML before it is uploaded for the report: the new inject-junit-analysis.py rewrites each failing testcase's failure message with the failure message and conclusion that live-investigation-watcher.sh recorded in live-investigation.md, so Allure's test overview page displays them. The suite-level testcase fails whenever any of its subtests do, so it gets every recorded analysis, each prefixed with the subtest's name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 2240189)
…tartup The watcher gave up if the evetest gRPC server was not reachable within 10 minutes of its launch, but the evetest image build that precedes it can take longer. A test failing after that paused forever, as nothing was left to investigate and resume it. Wait for as long as the shell running the evetest step is alive instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 215f8f4)
A test paused at a checkpoint or on failure waits until someone runs 'evetest continue'. When nobody does, e.g. in CI, the run hangs until the job times out. EVETEST_PAUSE_LIMIT=<seconds> makes the harness resume on its own after that long, as if 'evetest continue' was issued. The default of 0 keeps waiting indefinitely. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit d322dac)
Add a pause-limit input to the suite action, passed as EVETEST_PAUSE_LIMIT whenever pause-on-failure is enabled, so a paused test cannot hold the runner when the live investigation does not resume it. Cap one investigation below that limit so its report is written before the forced resume. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 9c4eefa)
For PRs from forks, evetest-pr checks out the PR's head repository, so the git remote in the workspace points at the fork. Without GH_REPO, gh infers the repository from that remote, and both `gh pr comment` (posting Claude's failure analysis) and `gh pr diff` (in live-investigation-watcher.sh) look up the PR number in the fork and fail with "Could not resolve to a PullRequest". The analysis comment step has continue-on-error, so it failed silently and no analysis was posted; the watcher likewise ran Claude without the PR diff. Set GH_REPO to github.repository for both steps, as the step that collapses previous analysis comments already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit f4b5c6a)
`gh pr diff` fails with HTTP 406 for a PR touching more than 300 files, which a vendor bump easily exceeds (e.g. PR lf-edge#5644). discover-suites then failed before any suite was scheduled, so evetest never ran for such PRs even though "Ready for evetest" was reported. Check out the PR head from the upstream repo with actions/checkout, blobless and with the full history of every branch (about 20 MB), and diff it against its merge-base with the base branch. Vendored code is left out of the diff; instead, the vendored modules added, removed or bumped are read from vendor/modules.txt and listed in pr-context.txt. If fetching or diffing fails, Claude is not asked for a skip decision and the run falls back to the full suite instead of failing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit e6f1a74)
The suite jobs run the evetest-run-suite action and its scripts with secrets (Anthropic key, GitHub token) from the PR checkout, which may be a fork. After checking out the PR, replace that directory with the base branch's copy so the PR cannot alter the code that handles them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit ccbf879)
The "Run evetest" step exports ANTHROPIC_API_KEY and GH_TOKEN so the live-investigation watcher can use them, but the same step then runs `make -C evetest evetest`, i.e. PR-controlled code, with the same environment. Unset both once the watcher has been started; it keeps its own copy and make and everything it spawns no longer inherit them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 5307aff)
The live-investigation watcher still fetched the PR diff with `gh pr diff`, which the API refuses for PRs touching more than 300 files (e.g. vendor bumps). Claude was then told no diff was supplied. Move the git-based diff computation, introduced for the test-skipping step, into .github/actions/evetest-run-suite/pr-diff.sh and use it from both the workflow and the watcher. Vendored code stays excluded and is summarised in pr-vendored.txt. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 0d9fde1)
PR Gate used the latest /evetest comment of a CODEOWNER regardless of its age, so every later push of the PR re-triggered evetest from the same stale comment, and the comment stayed visible forever. PR Gate now ignores minimized comments and passes the node ID of the comment it acted on to evetest-pr. Once that workflow starts a run (or finds one already testing the same commit), it reacts to the comment with a rocket and collapses it as resolved. A further run requires a new /evetest comment, or a PR approval. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit e97d6a4)
The Proxmox broker crashed with "panic: send on closed channel" inside go-proxmox's TermWebSocket after a VM's serial console connection broke. The function's close callback closes its result channels one second after signalling its goroutines to stop. If the connection is already broken, the writer goroutine fails to send the websocket close frame and blocks reporting that on the unbuffered error channel, which nobody reads any more. The channel is then closed under it and the panic happens in a library goroutine, so it cannot be recovered and takes down the broker together with every session it serves. The latest go-proxmox release has the same code. Talk the termproxy protocol directly with gorilla/websocket instead, reusing only the library's authenticated API session. There are no channels: Read and Write go straight to the websocket, writes are serialized, every write has a deadline, a failed keepalive closes the console so a silently dead connection cannot block Read forever, and Close only uses methods gorilla allows concurrently with everything else. The tests run the client against a fake termproxy server, including a connection that breaks underneath an active console. That is the scenario that made the old code panic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 9b323d1)
PVE answers a lookup of a VM that does not exist with an HTTP 500, the same status it uses for any other server-side failure, such as failing to fork an API worker on a host that is out of memory. Two code paths misread such failures: - When the create call failed, SetupDevice checked whether the VM existed anyway and quietly assumed nothing was created if that lookup failed. On a host out of memory, PVE had already written a placeholder config holding "lock: create" before failing to fork the create worker, and that VM was leaked. Even when found, it could not have been deleted, since a plain delete refuses a locked VM. - Teardown treated any lookup error other than a transport error as proof that the VM was gone, and leaked the VM on a 500. Add removeVM, which treats only PVE's "Configuration file ... does not exist" as a missing VM, stops the VM if it is running (paused included) and deletes it, overriding any lock left on it. Teardown and both failed-setup paths use it. On a failed create the cleanup runs before vmCreateMutex is released: if nothing was created, the VMID is free again, and removing it after another device was given that ID would delete that device's VM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 723baf2)
When the Proxmox broker crashed, every session it served lost its teardown, and nothing removed the leftovers afterwards: the VMs kept running and holding host memory, and their xconnect VNets and uploaded firmware volumes stayed behind. One crash left 22 VMs, 27 VNets and 28 firmware volumes on the host. Client sessions do not survive a broker restart, so at startup, before serving any session, remove everything the provider creates: VMs named with the "ev-" prefix, xconnect VNets of the evetest SDN zone, and "ev-" volumes in the import storage. The uplink VNet is installer-owned and stays. Also remove the nameless "lock: create" placeholder VMs that PVE leaves behind when it fails to start a create worker, unless a task is still running for them, so that a VM being created from the PVE UI is spared. This relies on running a single broker per Proxmox host, as the broker installer sets up. The sweep is best-effort: failures are logged and the broker starts anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 123986c)
The broker service had Restart=no, so a single crash took the broker down until someone restarted it by hand, and every test run in the meantime failed to connect. Restart it on failure instead. The broker now removes what the crashed run left behind at startup, so a restarted broker comes back clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 4c739f3)
The qemu provider starts device VMs without KVM async page faults (-kvm-asyncpf,-kvm-asyncpf-int): on a host that overcommits memory, a guest task faulting on a page the host swapped out can otherwise wait forever for KVM's "page ready" wakeup. Proxmox VMs were still created with plain "cpu: host", and a nightly run hit exactly that hang while the Proxmox host was swapping: an app's qemu inside EVE sat in kvm_async_pf_task_wait until the test timed out. PVE's "cpu" option only accepts a fixed set of flags, so disable the two properties as -global defaults of the host CPU model instead. On PVE's QEMU 11.0 this leaves the flags of PVE's own -cpu option in effect (checked over QMP: kvm-asyncpf and kvm-asyncpf-int read false, kvm-pv-eoi from -cpu still true). Async page faults are x86-only, so arm64 VMs are left alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit ef2b234)
waitTask waited with go-proxmox's Task.Wait, which returns nil as soon
as a task stops running, whether it succeeded or failed. A VM start
that failed was therefore reported as successful. In the nightly runs
this happened to seven SDN VMs whose QEMU exited at start ("start
failed: QEMU exited with code 1"): the broker logged that the SDN
device was powered on, then dialed its gRPC tunnel at an IP nobody
held for five minutes until the test failed with "no route to host".
The same blind spot hid failed VM creates, uploads, deletes and SDN
applies. Task.Wait also gave up at the first task status poll that
failed, so one lost response (e.g. a TLS handshake timeout) aborted a
power-off that may well have succeeded.
Poll tasks with our own loop instead: a task that PVE reports as
failed is an error carrying its exit status, up to five failed status
polls in a row are tolerated, and the overall deadline stays.
Reporting failures exposes cases that only worked because failures
were ignored, e.g. starting a VM that already runs. Power operations
now go through setVMPowerState, which reads the VM first and skips the
operation when the VM already is in the target state, and retries the
operation otherwise. The same check makes a retry after an attempt
whose response was lost count as done instead of repeating it. A
failed graceful shutdown is not retried: it means the guest ignored the
ACPI request. VM lookups by device name are retried too, except for a
VM that does not exist; until now a single truncated API response
("unexpected end of JSON input") failed a power-on outright.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit 4b73a6b)
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 17.0-stable #7111 +/- ##
===============================================
+ Coverage 23.55% 24.12% +0.57%
===============================================
Files 517 527 +10
Lines 95834 97411 +1577
===============================================
+ Hits 22574 23504 +930
- Misses 71343 72035 +692
+ Partials 1917 1872 -45 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Note on the Yetus failure: this is a known deterministic false positive, not a real issue. The perlcritic plugin counts perlcritic's own "source OK" success line (emitted by --verbose 1 for a clean file) as a new violation — the branch scan excludes evetest/deploy/proxmox/evetest-oom-hook.pl (0 lines) while the patch scan scans it (1 line) → "1 new" → -1. A re-run will not clear it. The identical failure occurred on #7042, which was merged. A fix (perlcritic override plugin with --quiet in .yetus/plugins.d/) has been verified end-to-end, including a negative control confirming real violations are still caught. |
Description
Add the evetest CI workflows to
17.0-stable, together with the fixes that weremade to the workflows and to evetest itself on
masterafter the evetest copyof #6833 was taken. With this, evetest runs in the
17.0-stableCI: on PRs(
evetest-pr) and nightly (evetest-nightly).All commits are cherry-picks (
git cherry-pick -x) of themastercommits.Backport of #6680, #6968, #6970, #6975, #6978, #6988, #6989, #7002, #7003,
#7012, #7020, #7021, #7022, #7024, #7030, #7042, #7043, #7045, #7079, #7084,
#7085, #7097, #7099, #7103
Only the commits of these PRs that apply to
17.0-stableare included:fix of this PR is already in the evetest copy of [17.0-stable] evetest: sync with master and adapt the tests to EVE 17.0 #6833)
hardening of
evetest-prlong suite, staging of EVE images once per host
Deviations from the original commits:
not exist in
17.0-stable: the konvert tests (4184477d1d) and the vaultkey mode tests (
9b2b6fefd0). The konvert part of27a85cc06cis also leftout.
0e464f71dd(evetest: fix recurring test and harness failures #7024) is adapted: it does not depend on the "fresh nodes only"wait, which is not in
17.0-stable.7eedcf86ba(evetest: cover what only eden tests covered so far #7084) callsRequestReboot(true), because17.0-stablehasthe single-argument form of that function.
9b323d148d(evetest: Proxmox broker reliability fixes #7020) has ago.modconflict resolved by keeping theeve-apiand pillar versions of17.0-stableand addinggorilla/websocket.images of
17.0-stable.Not included
because it depends on [17.0-stable] pkg/kube: port cluster-init.sh to Go daemon #7058 (
[17.0-stable] pkg/kube: port cluster-init.sh to Go daemon) being merged first. It will be backported separately afterwards.The witness part will be skipped.
TestMemoryPressureEvents(Move EVE to cgroup v2 #6772, cgroup v2), native Kubernetes NI attachment(Support attaching native Kubernetes workloads to EVE network instances #6697) and the 2-node HA test (2-Node HA, part 1: witness as the third etcd voting member #6933) are not part of
17.0-stable.PR dependencies
None. #7070 will follow once #7058 is merged (see above).
How to test and validate this PR
go vet ./...inevetest/passes.evetest-pr), and bya manual dispatch of
evetest-nightlyon17.0-stableafter the merge.Changelog notes
None. Test and CI changes only; no user-facing changes.
Checklist
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.
No device testing is needed: the changes affect only the evetest framework and
CI workflows.
🤖 Generated with Claude Code