Skip to content

evetest: protect the broker from OOM kills on Proxmox hosts and split a long suite - #7042

Merged
milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-broker-proxmox-fixes
Nov 8, 2026
Merged

milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-broker-proxmox-fixes

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

Three changes to the evetest CI setup, after a nightly run in which the Proxmox host OOM-killed the broker VM and most suites failed:

  1. OOM-killer priority by role. A new host hookscript, evetest-oom-hook.pl, sets oom_score_adj on a Proxmox VM's QEMU process at start. Under memory pressure the kernel now kills the test VMs created by the broker first (+500), then the GitHub runner VMs (-500), and the broker VM last (-900). The role is taken from the VM's tags. The broker and runner installers set the tag and attach the hookscript. A VM has only one hookscript, so for test VMs the existing evetest-hook.pl runs the new script.
  2. Runner label documentation. Document registering full-size runners with only evetest-runner, so that they stay free for heavy suites, when a host has many light runners and few full-size ones.
  3. Split TestApplicationConnectivitySuite (about 2 hours, tagged heavy only for that reason, although every test uses a single device) into four topical suites, none tagged heavy: TestLocalNetworkInstanceSuite, TestSwitchNetworkInstanceSuite, TestAppRoutingSuite and TestAppNICsSuite. They can now also run on light runners, and each takes about 20-40 minutes. All 25 test cases are kept.

How to test and validate this PR

  • OOM priorities: install or re-run the broker installer on a Proxmox host, start some test VMs, and check oom_score_adj of the QEMU processes (/proc/<pid>/oom_score_adj for the pids in /var/run/qemu-server/<vmid>.pid): -900 for the broker VM, -500 for runner VMs, +500 for test VMs. journalctl -t evetest-oom-hook shows each start. This was verified by hand on two Proxmox hosts. The installers themselves were not re-run end to end.
  • Suite split: GOWORK=off go fmt/vet/build ./tests/... in evetest/ pass, and make -C evetest list-test-suites lists the four new suites without the heavy tag. The new suites have not been run yet; the next nightly will run them.

Changelog notes

None. No user-facing changes (test infrastructure only).

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

The amd64/arm64 device boxes do not apply: this changes test infrastructure
(Proxmox host scripts and evetest suites), not EVE itself.

🤖 Generated with Claude Code

@milan-zededa
milan-zededa requested a review from rucoder as a code owner November 5, 2026 10:45
@milan-zededa milan-zededa added stable Should be backported to stable release(s) testing New test or test framework changes labels Nov 5, 2026
@milan-zededa
milan-zededa force-pushed the evetest-broker-proxmox-fixes branch from d7abe71 to 52cf728 Compare November 5, 2026 10:46
@milan-zededa
milan-zededa force-pushed the evetest-broker-proxmox-fixes branch from 52cf728 to 9c9d9ce Compare November 5, 2026 10:57
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.59%. Comparing base (d87216a) to head (b68f953).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7042      +/-   ##
==========================================
+ Coverage   29.07%   29.59%   +0.51%     
==========================================
  Files         548      559      +11     
  Lines       99033   100628    +1595     
==========================================
+ Hits        28793    29780     +987     
- Misses      67969    68617     +648     
+ Partials     2271     2231      -40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@milan-zededa

milan-zededa commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Notes on the two failing checks:

  • Apache Yetus, perlcritic -1: this looks like a false positive. The only perlcritic output for the patch is evetest/deploy/proxmox/evetest-oom-hook.pl source OK (see patch-perlcritic-result.txt in the yetus-scan artifact), which is what perlcritic prints for a file without violations, and the plugin counts it as one new issue because the file is new. Running perlcritic 1.140 on the file at its default severity (5) also reports source OK and nothing else.
  • Check Docker Hashes Consistency: not caused by this PR, which changes only files under evetest/. The same check fails on master (push 7cea040d89): pkg/debug/Dockerfile uses lfedge/eve-recovertpm:f39bf74c..., but c1a132f0... is built in this repo.

@milan-zededa
milan-zededa requested a review from rene November 5, 2026 14:40

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@milan-zededa you have docker hash issues to fix....

@milan-zededa

milan-zededa commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

@milan-zededa you have docker hash issues to fix....

@rene That looks like issue on master, this PR makes changes only inside evetest.
I did put a comment about this, see above.

milan-zededa and others added 3 commits November 5, 2026 19:32
When a Proxmox host runs out of RAM the kernel killed the largest qemu
process, which once was the broker VM and failed every test that was
using it.

Add evetest-oom-hook.pl, a host hookscript that sets oom_score_adj on
guest post-start by role: the broker VM is best protected (-900), runner
VMs less (-500), and VMs created by the broker for tests are killed
first (+500). The role is taken from the VM's tags. The broker and
runner installers set the tag and attach the hookscript directly. A VM
has only one hookscript, so evetest-hook.pl, which the broker attaches
to test VMs, runs it for them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
When a host has many light runners but very few full-size ones, the
full-size runners can be registered with only the evetest-runner label,
so that they never take light suites and stay free for heavy ones.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The suite ran about two hours and was tagged heavy only because of that
runtime, although every test uses a single device. Split it by topic
into TestLocalNetworkInstanceSuite, TestSwitchNetworkInstanceSuite,
TestAppRoutingSuite and TestAppNICsSuite, none of them tagged heavy, so
that they can also run on the light runners and finish in 20-40 minutes
each. Each test keeps its own device requirements, so none depends on
its former neighbours.

Update the suite references in the tests' doc comments.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa force-pushed the evetest-broker-proxmox-fixes branch from 9c9d9ce to b68f953 Compare November 5, 2026 18:32
@milan-zededa

Copy link
Copy Markdown
Contributor Author

OK, it seems that the docker hash inconsistency was resolved on master (I just rebased this PR)

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Evetest: Claude chose to skip 19 test/suite name(s) as unrelated to or redundant for this PR. Which ones and why: run summary.

Comment /evetest all to run the full suite, or /evetest <name> <name> ... to run only specific tests/suites.

@milan-zededa
milan-zededa merged commit 8d1b767 into lf-edge:master Nov 8, 2026
77 of 82 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s) testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants