Repository navigation
pillar: fix flaky timing and network unit tests - #6953
Conversation
TestSingleEthInterface and TestSingleEthInterfaceWithIPv6 failed intermittently on loaded CI runners, with DNS servers still reported after the interface lost its addresses. One simulated interface update publishes address, DNS and route events, and the first of them wakes the test. The reconciler's watcher picks randomly between a queued event and the pause request from Reconcile, so Reconcile could run before the DNS change was processed and skip the DNS state. Wait until the watcher has received every published event before reconciling. The mock network monitor reports the number of events still queued for its subscribers for this purpose. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0110937 to
e52be4f
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6953 +/- ##
==========================================
+ Coverage 28.47% 28.75% +0.28%
==========================================
Files 540 551 +11
Lines 98258 100011 +1753
==========================================
+ Hits 27978 28758 +780
- Misses 68012 68789 +777
- Partials 2268 2464 +196 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
TestManageStatFileSizeTargetSizesAvgThreshold asserted that rewriting a 6.8 MB stat file takes under 5 s. Under -race on a loaded CI runner it sometimes does not, failing the test although the trimmed file is correct. The test checks the result, which is what it is for; log the duration and drop the bound. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestWork required each sleeping job to finish within 100 ms of its nominal duration, which a loaded CI runner under -race does not guarantee. Run TestWork and TestLength under testing/synctest so the measured durations are exact. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test sleeps for twice the stats window and expects the slice to have filled at one sample per millisecond, which fails when real sleeps run long under -race on a loaded runner. Run it under testing/synctest, record every sample, and wait for the monitor to exit before the bubble ends. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe latency is measured in real time around the mock's simulated RTT, so a slow scheduler only inflates it, and the fixed headroom the tests allowed was exceeded on loaded CI runners. A probe that outlasts UserProbeTimeout fails instead of reporting latency, so use that as the upper bound. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the newer "zedagent" DPC fails and the manager falls back to "manual", the better-DPC timer re-tests "zedagent" and briefly makes it current again, so the one-shot check of the final DPC list failed whenever that retest landed in between. Push the timer out of the test's window and wait for the whole expected list to hold at once. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first global config adds the lastresort DPC, and the PNAC tests waited only for some verification to finish. When lastresort was verified first, the PNAC event was handled before the test's DPC was applied and the reacquire was canceled, so the retry never came. Wait for the test's DPC by key, and disable the periodic connectivity test, which blocks the main loop and delays the timed retries. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The send and verify tests reached google.com and postman-echo.com, so a slow resolver or a CI runner without Internet access failed them; a 5 s DNS timeout did so on CI. Serve them from an httptest TLS server bound to the port's IPv4 address instead. The URL is an IP literal, so these tests no longer exercise DNS resolution. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Claude finished @rene's task in 1m 53s —— View job Code Review
Verdict: correct, no blocking issues. I traced the fix against the actual watcher logic to confirm it closes the race rather than just narrowing it:
Soundness checks that hold:
Minor (non-blocking, no change requested): LGTM as a test-only flake fix. |
evetest skip decisionThis PR only touches Since every evetest suite/test exercises real device/e2e behavior built from production binaries, none of them provide coverage for the area this PR actually changes (determinism of pillar's Go unit tests). Skipping the entire evetest suite:
An engineer can comment |
|
@milan-zededa I added seven commits on top of the one you approved (e52be4f), one per timing-sensitive test that has been failing intermittently in the Publish coverage job: agentlog, worker, watcher, portprober, dpcmanager (fallback-to-manual and the PNAC tests) and controllerconn. The controllerconn send tests now use a local HTTPS server instead of google.com, so they no longer exercise DNS resolution. Could you take a look? |
|
Skipping all evetest suites for this PR:
Reason: every changed file in this PR is a pillar An engineer can comment |
Description
Pillar unit tests that depend on timing or on Internet access fail
intermittently on loaded CI runners under
-race. The Publish workflow'scoveragejob, which runsmake teston every push, failed 18 of 63 runs on2026-10-25..28 on a rotating set of these tests, and the same tests turn
Go Testsred on unrelated PRs. All changes are test-only, one commit percause:
TestSingleEthInterface*: one simulated interface updatepublishes address, DNS and route events, and
Reconcilecould pause thewatcher before the DNS event was processed (e.g.
https://github.com/lf-edge/eve/actions/runs/40497409949,
linux_test.go:342).The mock network monitor now reports how many events are still queued, and
the tests wait for zero before reconciling.
TestWork/TestLengthand watcherTestGoroutinesMonitorUpdateParamsKeepStatsDecrease: run on a fake clock(
testing/synctest), so measured sleeps and sample counts are exact.UserProbeTimeout, the only upperbound that holds when the scheduler is slow.
TestFallsBackToWorkingManual…: the better-DPC timer re-testedthe failed DPC and briefly made it current while the final state was read.
TestPNACDHCPReacquire*: the PNAC event could be handled whilethe lastresort DPC was current, and applying the test's DPC then canceled the
reacquire. The tests now wait for their own DPC.
default-route interface's IPv4 address instead of google.com and
postman-echo.com. These tests no longer exercise DNS resolution.
How to test and validate this PR
From
pkg/pillar, each of these must pass:The controllerconn send tests need a default route but no Internet access.
For the dpcreconciler race, temporarily add
time.Sleep(20 * time.Millisecond)at the end of the
case event := <-netEvents:branch ofwatcher()inpkg/pillar/dpcreconciler/linux.go, then rungo test -count=20 -run TestSingleEthInterface ./dpcreconciler/. Without thefix, the DNS assertions fail on nearly every iteration; with it, all pass.
Changelog notes
None.
PR Backports
The flaky tests exist on all LTS branches, where they cause the same CI
failures.
testing/synctestneeds Go 1.25, so on 16.0, 14.5 and 13.4 (Go1.24/1.23) the worker and watcher commits are left out.
Checklist
And the last but not least:
check them.
Test-only change: no documentation or device testing applies.
🤖 Generated with Claude Code