Skip to content

evetest: push SDN image to a remote broker and fix a skipped-subtest deadlock - #6968

Merged
rene merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-sdn-push
Oct 29, 2026
Merged

rene merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-sdn-push

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR addresses issues found while testing the evetest CI, so it is necessary
to get it in for the evetest CI to work properly. It contains two fixes and a small addition to the nightly trigger script.

Push the SDN image from the evetest harness to a remote broker.
The CI uses a remote broker, which pulls the SDN image itself. When a PR changes
the SDN (e.g. #6929), the new SDN image is not published, so the broker cannot get it and all
tests fail. The same is solved for EVE images by pushing them from the harness to
the broker; this PR does the same for the SDN image:

  • SetupDevices still fails the request when the broker has the SDN image neither
    locally nor can pull it, but the error is now a FailedPrecondition status with
    an ErrorInfo detail (reason MISSING_SDN_CONTAINER_IMAGE). Nothing has been
    provisioned at that point.
  • A harness recognizing the error uploads its local SDN image using the new
    PushSDNContainerImage RPC and repeats SetupDevices. An older harness just
    fails with the pull error, as before.
  • The broker-side receive logic and the harness-side upload loop are shared
    between the EVE and SDN images.
  • make evetest now ensures the local SDN image also with a remote broker
    (still skipped when EVETEST_SDN_VERSION is set explicitly).

Fix a deadlock when a suite subtest is skipped by EVETEST_SKIP.
Init takes testM before checking EVETEST_SKIP for a suite subtest, but the
skip path called t.Skipf without releasing it. Skipf ends the goroutine, leaving
testM held, so the next testM.Lock() in RunTestSuite blocked forever and
the run hung. testM is now unlocked before skipping, like on every other exit
from that branch.

Allow enabling coverage in the nightly trigger script.
evetest/deploy/proxmox/trigger-evetest-nightly.sh gets a --collect-coverage
flag, which sets the collect-coverage input of the evetest-nightly workflow
(off by default). This lets the cron driving the nightly run enable coverage
collection on selected days only, e.g. on Sundays.

How to test and validate this PR

  • Run make evetest NAME=<test> against a remote broker (EVETEST_BROKER_ADDRESS)
    with an SDN image that was built locally and is not published (e.g. after changing
    anything under evetest/sdn/). The harness logs that the broker is missing the SDN
    container image, pushes it, and the test proceeds. Before this change, it failed
    with a pull error.
  • Run a suite with EVETEST_SKIP matching one of its subtests. The subtest is
    skipped and the remaining subtests run, instead of the run hanging.
  • Both fixes were validated with the evetest CI brokers upgraded to a build of this
    PR. go vet and go test pass for evetest and evetest/broker, and
    TestIsMissingSDNImageErr covers the detection of the broker error.

Changelog notes

None. Only the evetest test framework and its CI are changed, with no user-facing
changes.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

    The change only affects the evetest framework and the broker, not EVE itself,
    so there is no EVE device to test it on.

🤖 Generated with Claude Code

milan-zededa and others added 2 commits October 29, 2026 11:08
A remote broker pulls the SDN image itself, so an SDN image built from
an unmerged change was unavailable to it and every test failed.

When SetupDevices can neither find nor pull the SDN image, the broker
still fails the request with the pull error, now as a FailedPrecondition
status carrying an ErrorInfo detail with a dedicated reason. A harness
that recognizes it uploads its local copy through the new
PushSDNContainerImage RPC and repeats the request, as it already does
for the EVE container image. An older harness just fails with the pull
error. The broker-side receive logic and the harness-side upload loop
are shared between the two image types.

make evetest now ensures the local SDN image also with a remote broker.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Init takes testM before checking EVETEST_SKIP for a suite subtest, but
the skip path called t.Skipf without releasing it. Skipf ends the
goroutine via runtime.Goexit, leaving testM held, so the next
testM.Lock() in RunTestSuite blocked forever and the run hung.

Unlock testM before skipping, like every other exit from this branch.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa milan-zededa added stable Should be backported to stable release(s) testing New test or test framework changes labels Oct 29, 2026
@milan-zededa
milan-zededa requested a review from rucoder as a code owner October 29, 2026 10:50
@milan-zededa
milan-zededa requested a review from rene October 29, 2026 10:52
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.77%. Comparing base (02ad7ce) to head (3bc8b92).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6968      +/-   ##
==========================================
+ Coverage   28.48%   28.77%   +0.28%     
==========================================
  Files         540      551      +11     
  Lines       98258   100005    +1747     
==========================================
+ Hits        27990    28772     +782     
- Misses      67998    68769     +771     
- Partials     2270     2464     +194     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Allow the nightly trigger to set the collect-coverage workflow input,
so a cron can enable coverage collection on selected days only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@rene
rene merged commit a273c3e into lf-edge:master Oct 29, 2026
39 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s) testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants