Repository navigation
evetest: fix stale PR image on brokers and stage EVE images once per host - #7043
Merged
Merged
Conversation
github-actions
Bot
requested review from
eriknordmark,
europaul,
rene,
rucoder,
shjala and
yash-zededa
November 5, 2026 13:42
Contributor
Author
Contributor
|
Claude finished @milan-zededa's task in 4m 25s —— View job Code reviewI reviewed the broker claim/upload/pull logic, the CLI commands, the setup/edgedevice changes, and the CI workflow/action wiring (generated
Findings
Everything else looks solid:
No blocking issues from my side; the inline point is the only thing worth a second look. |
milan-zededa
force-pushed
the
evetest-pr-image
branch
from
November 5, 2026 14:08
44681dd to
c4832b3
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7043 +/- ##
==========================================
+ Coverage 29.09% 29.59% +0.50%
==========================================
Files 548 559 +11
Lines 99033 100628 +1595
==========================================
+ Hits 28811 29782 +971
- Misses 67949 68620 +671
+ Partials 2273 2226 -47 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
milan-zededa
force-pushed
the
evetest-pr-image
branch
2 times, most recently
from
November 5, 2026 19:04
616ee3b to
65a34b8
Compare
milan-zededa
marked this pull request as ready for review
November 5, 2026 19:06
milan-zededa
force-pushed
the
evetest-pr-image
branch
from
November 5, 2026 20:17
65a34b8 to
f27402c
Compare
milan-zededa
marked this pull request as draft
November 5, 2026 20:19
milan-zededa
force-pushed
the
evetest-pr-image
branch
from
November 6, 2026 10:19
f27402c to
d87b6f7
Compare
milan-zededa
marked this pull request as ready for review
November 6, 2026 10:26
The exported image was named after the PR number only, so a consumer that caches images by name (the evetest broker) kept testing the first build of a PR after later pushes. Also tag the image with its unique build version; the evetest-run-suite loader derives the version to test from that tag. The PR-number tag stays for Eden. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Every suite job of a PR (or coverage nightly) run downloaded both EVE
image artifacts, about 2.4 GB, and loaded them into its own Docker, so a
run fetched them once per suite from the same two hosts.
Let the jobs share one upload per broker instead. A job asks the broker
for the right to upload an image ("evetest broker image claim-upload"):
it gets a ticket, or learns that the broker already has the image or that
another job is uploading it, in which case it waits and takes over if that
job fails. Only the claim winner downloads the artifact and uploads it
("image upload"); a claim lapses after 60 minutes if the upload does not
start and after 10 minutes without upload progress, and is released when the
upload ends or on request ("image release-claim").
The upload names the image by the ticket alone.
A test that needs the image in the runner's Docker (an EVE upgrade
target) loads it back from the broker with the new PullEVEContainerImage
when it cannot pull it.
The build publishes its image version as an artifact so that the jobs
know which image to ask for before downloading anything; evetest-pr.yml
accepts it only in the form of that PR's versions. The coverage nightly
takes the version from its build job outputs. The download-image-artifacts
path of evetest-run-suite is removed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The job summary heading was emitted as an HTML <h2> directly followed by Claude's markdown report, so the report's first line (its title) was swallowed into the HTML block and shown as literal text. Its sub-headings also rendered at the same level as the section heading. Write the section heading as markdown followed by a blank line, and run the report through shift-headings.sh so its headings nest below it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
milan-zededa
force-pushed
the
evetest-pr-image
branch
from
November 8, 2026 15:37
2bcce76 to
8d369d3
Compare
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two problems in how EVE images reach the evetest brokers in PR (and coverage
nightly) runs:
The broker could test a stale PR image. PR images were exported as
evebuild/pr:<PR>-<hv>-<arch>, i.e. tagged by the PR number only, and thebroker decides whether it needs an upload by checking for that name. After a
push that changes EVE code, the broker kept building devices from the PR's
earlier build (it keeps unused images for 7 days).
build-evenow also tagsthe image with its unique build version
(
evebuild/pr:0.0.0-pr<PR>-<commit>-<hv>-<arch>), and the suites ask forthat name. The PR-number tag stays in the same tar for Eden, which cuts the
tag at the first
-and cannot use the versioned one (TODO in the code todrop it together with Eden).
Every suite job downloaded both EVE flavors. Each of the ~19 suite jobs
fetched the kvm (0.7 GB) and k (1.7 GB) image artifacts at the same moment and
loaded them into its own Docker, even for suites that use one flavor. The
downloads took 10 to 33 minutes per job, often longer than the tests, and a
failed download failed the whole suite before any test ran.
Now the jobs share one upload per broker:
evetest broker image claim-uploadasks the broker for the right toupload an image: it exits 0 with a ticket if this job has to provide it, 10
if the broker already has it. While another job is uploading it, it waits
and takes over if that job fails. A claim lapses after 60 minutes if the
upload does not start (time to fetch the image) and after 10 minutes
without upload progress once it runs.
download-artifactandevetest broker image upload --ticket T <tar>. The ticket identifies theimage, and the broker checks that the archive carries that tag.
evetest broker image release-claimfrees a claim after a failed download.ClaimEVEImageUpload,ReleaseEVEImageUploadClaim(no client session needed), a
claim_ticketonPushEVEContainerImage, andPullEVEContainerImagefor tests that need the image in the runner's ownDocker (an EVE upgrade target), used when
docker pullcannot get it.which image to ask for before downloading anything.
evetest-pr.ymlaccepts it only in the form of that PR's versions. The coverage nightly takes
the version from its build jobs' outputs.
does not wait for that holder: ticketed uploads skip the concurrent-upload
wait, and the takeover cancels the stalled upload.
download-image-artifactspath ofevetest-run-suiteis removed.How to test and validate this PR
Automated:
go test ./brokerinevetest/covers the claim lease (exclusivity,expiry and takeover, renewal, release) and, with a Docker daemon, the whole
claim, ticketed upload (wrong ticket, archive without the claimed tag, retry
after a failed upload) and
PullEVEContainerImageflow.Manual (the brokers are upgraded, so these can be run now):
evetest-pron a PR that changes EVE code, push a second commit, andcheck in the suite logs that the second run asks the brokers for the new
version (
0.0.0-pr<PR>-<new commit>) and that devices report it."Granted the claim of the image ... it has to be uploaded" per flavor and the
others finish
claim-uploadwith "the broker already has the image".collect-coverageenabled and check the same.TestEVEUpgradefrom a PR run and check that it gets the target imagefrom the broker.
actionlintand yetus, so the checks above still need a run against theupgraded brokers.
Changelog notes
None. CI and test tooling only, no user-facing changes.
PR Backports
Checklist
check them.
🤖 Generated with Claude Code