Repository navigation
evetest: protect the broker from OOM kills on Proxmox hosts and split a long suite - #7042
Conversation
d7abe71 to
52cf728
Compare
52cf728 to
9c9d9ce
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7042 +/- ##
==========================================
+ Coverage 29.07% 29.59% +0.51%
==========================================
Files 548 559 +11
Lines 99033 100628 +1595
==========================================
+ Hits 28793 29780 +987
- Misses 67969 68617 +648
+ Partials 2271 2231 -40 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Notes on the two failing checks:
|
rene
left a comment
There was a problem hiding this comment.
@milan-zededa you have docker hash issues to fix....
@rene That looks like issue on master, this PR makes changes only inside evetest. |
When a Proxmox host runs out of RAM the kernel killed the largest qemu process, which once was the broker VM and failed every test that was using it. Add evetest-oom-hook.pl, a host hookscript that sets oom_score_adj on guest post-start by role: the broker VM is best protected (-900), runner VMs less (-500), and VMs created by the broker for tests are killed first (+500). The role is taken from the VM's tags. The broker and runner installers set the tag and attach the hookscript directly. A VM has only one hookscript, so evetest-hook.pl, which the broker attaches to test VMs, runs it for them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
When a host has many light runners but very few full-size ones, the full-size runners can be registered with only the evetest-runner label, so that they never take light suites and stay free for heavy ones. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
The suite ran about two hours and was tagged heavy only because of that runtime, although every test uses a single device. Split it by topic into TestLocalNetworkInstanceSuite, TestSwitchNetworkInstanceSuite, TestAppRoutingSuite and TestAppNICsSuite, none of them tagged heavy, so that they can also run on the light runners and finish in 20-40 minutes each. Each test keeps its own device requirements, so none depends on its former neighbours. Update the suite references in the tests' doc comments. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
9c9d9ce to
b68f953
Compare
|
OK, it seems that the docker hash inconsistency was resolved on master (I just rebased this PR) |
|
Evetest: Claude chose to skip 19 test/suite name(s) as unrelated to or redundant for this PR. Which ones and why: run summary. Comment |
Description
Three changes to the evetest CI setup, after a nightly run in which the Proxmox host OOM-killed the broker VM and most suites failed:
evetest-oom-hook.pl, setsoom_score_adjon a Proxmox VM's QEMU process at start. Under memory pressure the kernel now kills the test VMs created by the broker first (+500), then the GitHub runner VMs (-500), and the broker VM last (-900). The role is taken from the VM's tags. The broker and runner installers set the tag and attach the hookscript. A VM has only one hookscript, so for test VMs the existingevetest-hook.plruns the new script.evetest-runner, so that they stay free for heavy suites, when a host has many light runners and few full-size ones.TestApplicationConnectivitySuite(about 2 hours, tagged heavy only for that reason, although every test uses a single device) into four topical suites, none tagged heavy:TestLocalNetworkInstanceSuite,TestSwitchNetworkInstanceSuite,TestAppRoutingSuiteandTestAppNICsSuite. They can now also run on light runners, and each takes about 20-40 minutes. All 25 test cases are kept.How to test and validate this PR
oom_score_adjof the QEMU processes (/proc/<pid>/oom_score_adjfor the pids in/var/run/qemu-server/<vmid>.pid): -900 for the broker VM, -500 for runner VMs, +500 for test VMs.journalctl -t evetest-oom-hookshows each start. This was verified by hand on two Proxmox hosts. The installers themselves were not re-run end to end.GOWORK=off go fmt/vet/build ./tests/...inevetest/pass, andmake -C evetest list-test-suiteslists the four new suites without the heavy tag. The new suites have not been run yet; the next nightly will run them.Changelog notes
None. No user-facing changes (test infrastructure only).
PR Backports
Checklist
And the last but not least:
check them.
The amd64/arm64 device boxes do not apply: this changes test infrastructure
(Proxmox host scripts and evetest suites), not EVE itself.
🤖 Generated with Claude Code