Skip to content

pillar: fix flaky timing and network unit tests - #6953

Merged
rene merged 8 commits into
lf-edge:masterfrom
eriknordmark:dpcreconciler-dns-race
Oct 30, 2026
Merged

rene merged 8 commits into
lf-edge:masterfrom
eriknordmark:dpcreconciler-dns-race

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Description

Pillar unit tests that depend on timing or on Internet access fail
intermittently on loaded CI runners under -race. The Publish workflow's
coverage job, which runs make test on every push, failed 18 of 63 runs on
2026-10-25..28 on a rotating set of these tests, and the same tests turn
Go Tests red on unrelated PRs. All changes are test-only, one commit per
cause:

  • dpcreconciler TestSingleEthInterface*: one simulated interface update
    publishes address, DNS and route events, and Reconcile could pause the
    watcher before the DNS event was processed (e.g.
    https://github.com/lf-edge/eve/actions/runs/40497409949, linux_test.go:342).
    The mock network monitor now reports how many events are still queued, and
    the tests wait for zero before reconciling.
  • agentlog: drop the 5 s wall-clock bound on the stat-file rewrite.
  • worker TestWork/TestLength and watcher
    TestGoroutinesMonitorUpdateParamsKeepStatsDecrease: run on a fake clock
    (testing/synctest), so measured sleeps and sample counts are exact.
  • portprober: bound measured latency by UserProbeTimeout, the only upper
    bound that holds when the scheduler is slow.
  • dpcmanager TestFallsBackToWorkingManual…: the better-DPC timer re-tested
    the failed DPC and briefly made it current while the final state was read.
  • dpcmanager TestPNACDHCPReacquire*: the PNAC event could be handled while
    the lastresort DPC was current, and applying the test's DPC then canceled the
    reacquire. The tests now wait for their own DPC.
  • controllerconn send/verify tests: use a local HTTPS server on the
    default-route interface's IPv4 address instead of google.com and
    postman-echo.com. These tests no longer exercise DNS resolution.

How to test and validate this PR

From pkg/pillar, each of these must pass:

go test -race -count=20 -run 'TestWork$|TestLength$' ./worker/
go test -race -count=20 -run TestGoroutinesMonitorUpdateParamsKeepStatsDecrease ./cmd/watcher/
go test -race -count=3 -run 'TestPNAC|TestFallsBackToWorkingManual' ./dpcmanager/
go test -race ./agentlog/ ./portprober/ ./dpcmanager/ ./controllerconn/ ./dpcreconciler/... ./netmonitor/...

The controllerconn send tests need a default route but no Internet access.

For the dpcreconciler race, temporarily add time.Sleep(20 * time.Millisecond)
at the end of the case event := <-netEvents: branch of watcher() in
pkg/pillar/dpcreconciler/linux.go, then run
go test -count=20 -run TestSingleEthInterface ./dpcreconciler/. Without the
fix, the DNS assertions fail on nearly every iteration; with it, all pass.

Changelog notes

None.

PR Backports

The flaky tests exist on all LTS branches, where they cause the same CI
failures. testing/synctest needs Go 1.25, so on 16.0, 14.5 and 13.4 (Go
1.24/1.23) the worker and watcher commits are left out.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

Test-only change: no documentation or device testing applies.

🤖 Generated with Claude Code

@eriknordmark eriknordmark added the stable Should be backported to stable release(s) label Oct 27, 2026
TestSingleEthInterface and TestSingleEthInterfaceWithIPv6 failed
intermittently on loaded CI runners, with DNS servers still reported
after the interface lost its addresses. One simulated interface update
publishes address, DNS and route events, and the first of them wakes
the test. The reconciler's watcher picks randomly between a queued
event and the pause request from Reconcile, so Reconcile could run
before the DNS change was processed and skip the DNS state.

Wait until the watcher has received every published event before
reconciling. The mock network monitor reports the number of events
still queued for its subscribers for this purpose.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the dpcreconciler-dns-race branch from 0110937 to e52be4f Compare October 28, 2026 18:43
@github-actions
github-actions Bot requested a review from milan-zededa October 28, 2026 18:44
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.75%. Comparing base (3a27e25) to head (c3e3ee8).
⚠️ Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/netmonitor/mock.go 0.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6953      +/-   ##
==========================================
+ Coverage   28.47%   28.75%   +0.28%     
==========================================
  Files         540      551      +11     
  Lines       98258   100011    +1753     
==========================================
+ Hits        27978    28758     +780     
- Misses      68012    68789     +777     
- Partials     2268     2464     +196     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

eriknordmark and others added 7 commits October 28, 2026 22:37
TestManageStatFileSizeTargetSizesAvgThreshold asserted that rewriting a
6.8 MB stat file takes under 5 s. Under -race on a loaded CI runner it
sometimes does not, failing the test although the trimmed file is
correct. The test checks the result, which is what it is for; log the
duration and drop the bound.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestWork required each sleeping job to finish within 100 ms of its
nominal duration, which a loaded CI runner under -race does not
guarantee. Run TestWork and TestLength under testing/synctest so the
measured durations are exact.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test sleeps for twice the stats window and expects the slice to
have filled at one sample per millisecond, which fails when real
sleeps run long under -race on a loaded runner. Run it under
testing/synctest, record every sample, and wait for the monitor to
exit before the bubble ends.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe latency is measured in real time around the mock's simulated
RTT, so a slow scheduler only inflates it, and the fixed headroom the
tests allowed was exceeded on loaded CI runners. A probe that outlasts
UserProbeTimeout fails instead of reporting latency, so use that as
the upper bound.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the newer "zedagent" DPC fails and the manager falls back to
"manual", the better-DPC timer re-tests "zedagent" and briefly makes
it current again, so the one-shot check of the final DPC list failed
whenever that retest landed in between. Push the timer out of the
test's window and wait for the whole expected list to hold at once.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first global config adds the lastresort DPC, and the PNAC tests
waited only for some verification to finish. When lastresort was
verified first, the PNAC event was handled before the test's DPC was
applied and the reacquire was canceled, so the retry never came.
Wait for the test's DPC by key, and disable the periodic connectivity
test, which blocks the main loop and delays the timed retries.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The send and verify tests reached google.com and postman-echo.com, so
a slow resolver or a CI runner without Internet access failed them; a
5 s DNS timeout did so on CI. Serve them from an httptest TLS server
bound to the port's IPv4 address instead. The URL is an IP literal,
so these tests no longer exercise DNS resolution.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rene

rene commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 1m 53s —— View job


Code Review

  • Read the diff (linux_test.go, netmonitor/mock.go)
  • Verified the synchronization reasoning against the watcher/pause code

Verdict: correct, no blocking issues.

I traced the fix against the actual watcher logic to confirm it closes the race rather than just narrowing it:

  • The watcher holds r.Lock() for the whole lifetime of its loop (linux.go:292), releasing it only inside the pause branch (linux.go:345). A pause from Reconcile is therefore only honored at the next select iteration, after the current event's switch/addPendingReconcile has fully run.
  • pauseWatcher() sends watcherCtrlPause then blocks on r.Lock() (linux.go:278-280), so Reconcile cannot proceed until the watcher reaches that unlock.
  • Consequently, once an event leaves the channel (<-netEvents, linux.go:299), it is guaranteed to be processed before Reconcile can observe state. So gating on UndeliveredEvents() == 0 (channel drained) is sufficient — the "event received but not yet processed" window is covered by the lock. This matches the PR's claim.

Soundness checks that hold:

  • publishEvent runs synchronously under m.Lock() (mock.go:409), and each test block publishes all events before calling waitForNetEvents, so the queue count decreases monotonically to zero — no transient-zero false positive.
  • UndeliveredEvents() takes m.Lock(), consistent with the publisher; the concurrent len() on a channel being drained by the watcher is goroutine-safe, so -race is clean.

Minor (non-blocking, no change requested): UndeliveredEvents() sums queued events across all subscribers. That's fine for these single-subscriber tests, but if a future test registered a second subscriber that didn't drain, this helper would block until its Eventually timeout. Worth a one-line note in the doc comment if more subscribers ever appear, but not needed now.

LGTM as a test-only flake fix.
· branch dpcreconciler-dns-race

@github-actions

Copy link
Copy Markdown
Contributor

evetest skip decision

This PR only touches *_test.go files (plus netmonitor/mock.go, a test-only mock) under pkg/pillar: dpcreconciler, agentlog, worker, cmd/watcher, controllerconn, and dpcmanager. It fixes flaky/racy assertions in pillar's own unit tests and doesn't change any production code path.

Since every evetest suite/test exercises real device/e2e behavior built from production binaries, none of them provide coverage for the area this PR actually changes (determinism of pillar's Go unit tests). Skipping the entire evetest suite:

  • TestApplicationConnectivitySuite
  • TestAppsSuite
  • TestBootstrapSuite
  • TestControllerFaultsSuite
  • TestDatastoreSuite
  • TestDeviceConnectivitySuite
  • TestDiagSuite
  • TestHardwareSuite
  • TestKonvertSuite
  • TestKubeRegistrationSuite
  • TestLPSSuite
  • TestNodeClusterResilienceSuite
  • TestNodeClusterSuite
  • TestPcibackErrorSuite
  • TestPortScanSuite
  • TestSecuritySuite
  • TestStorageSuite
  • TestTelemetrySuite
  • TestUpgradeSuite
  • TestPVCUploadReport
  • TestVaultKeyKeptOnTPMError
  • TestKvmToKRepartitionAppVolume
  • TestKvmToKRepartitionVolmig

An engineer can comment /evetest all to force the full suite to run instead, or /evetest <name> <name> ... to run only specific tests/suites.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

@milan-zededa I added seven commits on top of the one you approved (e52be4f), one per timing-sensitive test that has been failing intermittently in the Publish coverage job: agentlog, worker, watcher, portprober, dpcmanager (fallback-to-manual and the PNAC tests) and controllerconn. The controllerconn send tests now use a local HTTPS server instead of google.com, so they no longer exercise DNS resolution. Could you take a look?

@eriknordmark eriknordmark changed the title dpcreconciler: fix race in interface update tests pillar: fix flaky timing and network unit tests Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Skipping all evetest suites for this PR:

  • TestApplicationConnectivitySuite
  • TestAppsSuite
  • TestBootstrapSuite
  • TestControllerFaultsSuite
  • TestDatastoreSuite
  • TestDeviceConnectivitySuite
  • TestDiagSuite
  • TestHardwareSuite
  • TestKonvertSuite
  • TestKubeRegistrationSuite
  • TestLPSSuite
  • TestNodeClusterResilienceSuite
  • TestNodeClusterSuite
  • TestPcibackErrorSuite
  • TestPortScanSuite
  • TestSecuritySuite
  • TestStorageSuite
  • TestTelemetrySuite
  • TestUpgradeSuite

Reason: every changed file in this PR is a pillar _test.go file (or a test-only mock, netmonitor/mock.go) in dpcreconciler, agentlog, worker, watcher, portprober, dpcmanager, and controllerconn. The change only stabilizes flaky assertions in pillar's own Go unit tests (timing bounds, fake clocks, a local HTTPS test server instead of real DNS) and makes zero changes to production/runtime code. Since evetest suites exercise actual device/runtime behavior on a built EVE image, none of them can observe any difference from this PR, so all are unrelated.

An engineer can comment /evetest all to force the full suite to run instead, or /evetest <name> <name> ... to run only specific tests/suites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants