Skip to content

evetest: fix stale PR image on brokers and stage EVE images once per host - #7043

Merged
milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-pr-image
Nov 9, 2026
Merged

milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-pr-image

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

Two problems in how EVE images reach the evetest brokers in PR (and coverage
nightly) runs:

  1. The broker could test a stale PR image. PR images were exported as
    evebuild/pr:<PR>-<hv>-<arch>, i.e. tagged by the PR number only, and the
    broker decides whether it needs an upload by checking for that name. After a
    push that changes EVE code, the broker kept building devices from the PR's
    earlier build (it keeps unused images for 7 days). build-eve now also tags
    the image with its unique build version
    (evebuild/pr:0.0.0-pr<PR>-<commit>-<hv>-<arch>), and the suites ask for
    that name. The PR-number tag stays in the same tar for Eden, which cuts the
    tag at the first - and cannot use the versioned one (TODO in the code to
    drop it together with Eden).

  2. Every suite job downloaded both EVE flavors. Each of the ~19 suite jobs
    fetched the kvm (0.7 GB) and k (1.7 GB) image artifacts at the same moment and
    loaded them into its own Docker, even for suites that use one flavor. The
    downloads took 10 to 33 minutes per job, often longer than the tests, and a
    failed download failed the whole suite before any test ran.

    Now the jobs share one upload per broker:

    • evetest broker image claim-upload asks the broker for the right to
      upload an image: it exits 0 with a ticket if this job has to provide it, 10
      if the broker already has it. While another job is uploading it, it waits
      and takes over if that job fails. A claim lapses after 60 minutes if the
      upload does not start (time to fetch the image) and after 10 minutes
      without upload progress once it runs.
    • Only the claim winner runs download-artifact and
      evetest broker image upload --ticket T <tar>. The ticket identifies the
      image, and the broker checks that the archive carries that tag.
    • evetest broker image release-claim frees a claim after a failed download.
    • New broker RPCs: ClaimEVEImageUpload, ReleaseEVEImageUploadClaim
      (no client session needed), a claim_ticket on PushEVEContainerImage, and
      PullEVEContainerImage for tests that need the image in the runner's own
      Docker (an EVE upgrade target), used when docker pull cannot get it.
    • The build publishes its image version as an artifact so that the jobs know
      which image to ask for before downloading anything. evetest-pr.yml
      accepts it only in the form of that PR's versions. The coverage nightly takes
      the version from its build jobs' outputs.
    • A client that takes a claim over from a holder that stalled past its lease
      does not wait for that holder: ticketed uploads skip the concurrent-upload
      wait, and the takeover cancels the stalled upload.
    • The download-image-artifacts path of evetest-run-suite is removed.

How to test and validate this PR

Automated: go test ./broker in evetest/ covers the claim lease (exclusivity,
expiry and takeover, renewal, release) and, with a Docker daemon, the whole
claim, ticketed upload (wrong ticket, archive without the claimed tag, retry
after a failed upload) and PullEVEContainerImage flow.

Manual (the brokers are upgraded, so these can be run now):

  • Run evetest-pr on a PR that changes EVE code, push a second commit, and
    check in the suite logs that the second run asks the brokers for the new
    version (0.0.0-pr<PR>-<new commit>) and that devices report it.
  • In the same run, check that on each broker host exactly one job logs
    "Granted the claim of the image ... it has to be uploaded" per flavor and the
    others finish claim-upload with "the broker already has the image".
  • Run the nightly with collect-coverage enabled and check the same.
  • Run TestEVEUpgrade from a PR run and check that it gets the target image
    from the broker.
  • Not run in CI yet: the workflow changes have only been checked with
    actionlint and yetus, so the checks above still need a run against the
    upgraded brokers.

Changelog notes

None. CI and test tooling only, no user-facing changes.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

@milan-zededa milan-zededa added stable Should be backported to stable release(s) testing New test or test framework changes labels Nov 5, 2026
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@claude

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @milan-zededa's task in 4m 25s —— View job


Code review

I reviewed the broker claim/upload/pull logic, the CLI commands, the setup/edgedevice changes, and the CI workflow/action wiring (generated *.pb.go and vendor files skipped).

  • Read changed files
  • Review broker claim/upload logic
  • Review CLI commands
  • Review workflow/action changes
  • Post findings

Findings

  1. One correctness concern, posted inline on evetest/broker/broker.go: a claim-ticketed upload still passes through receiveDockerImage's separate imageUploads dedup, which can defeat the lease takeover in the hang case it's designed for (the taker waits on the stalled holder's upload instead of uploading its own bytes, then fails the HaveDockerImage check). Narrow timing window, flagged as plausible.

Everything else looks solid:

  • The lease state machine (imageclaim.go) is clean and well-tested (exclusivity, expiry/takeover, renew-after-lapse, pruning, imageOf).
  • Ticket-based uploads correctly need no session, renew per chunk, validate the claimed tag is actually present, and release on both success and failure.
  • Superseded tickets are rejected (imageOf miss / renew mismatch), and a late release by the previous holder can't drop a new claim.
  • CLI exit-code contract (0/10/11/1), Unimplemented→"update the broker", and stdout-only ticket are consistent with the action's case $rc handling.
  • Version plumbing is coherent: build-eve emits the versioned tag + version output/artifact, evetest-pr.yml validates it against ^0\.0\.0-pr<PR>-…$, and the version is hv-independent so reusing the kvm build's output for the k flavor in the nightly is sound.
  • pullEVEImage registry→broker fallback and the pipe-based streaming in pullEVEImageFromBroker handle NotFound and EOF correctly.

No blocking issues from my side; the inline point is the only thing worth a second look.
· branch evetest-pr-image

Comment thread evetest/broker/broker.go Resolved
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.59%. Comparing base (8d1b767) to head (8d369d3).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7043      +/-   ##
==========================================
+ Coverage   29.09%   29.59%   +0.50%     
==========================================
  Files         548      559      +11     
  Lines       99033   100628    +1595     
==========================================
+ Hits        28811    29782     +971     
- Misses      67949    68620     +671     
+ Partials     2273     2226      -47     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@milan-zededa
milan-zededa force-pushed the evetest-pr-image branch 2 times, most recently from 616ee3b to 65a34b8 Compare November 5, 2026 19:04
@milan-zededa
milan-zededa marked this pull request as ready for review November 5, 2026 19:06
@milan-zededa
milan-zededa marked this pull request as draft November 5, 2026 20:19
@milan-zededa
milan-zededa marked this pull request as ready for review November 6, 2026 10:26
milan-zededa and others added 3 commits November 8, 2026 16:37
The exported image was named after the PR number only, so a consumer
that caches images by name (the evetest broker) kept testing the first
build of a PR after later pushes. Also tag the image with its unique
build version; the evetest-run-suite loader derives the version to test
from that tag. The PR-number tag stays for Eden.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Every suite job of a PR (or coverage nightly) run downloaded both EVE
image artifacts, about 2.4 GB, and loaded them into its own Docker, so a
run fetched them once per suite from the same two hosts.

Let the jobs share one upload per broker instead. A job asks the broker
for the right to upload an image ("evetest broker image claim-upload"):
it gets a ticket, or learns that the broker already has the image or that
another job is uploading it, in which case it waits and takes over if that
job fails. Only the claim winner downloads the artifact and uploads it
("image upload"); a claim lapses after 60 minutes if the upload does not
start and after 10 minutes without upload progress, and is released when the
upload ends or on request ("image release-claim").
The upload names the image by the ticket alone.

A test that needs the image in the runner's Docker (an EVE upgrade
target) loads it back from the broker with the new PullEVEContainerImage
when it cannot pull it.

The build publishes its image version as an artifact so that the jobs
know which image to ask for before downloading anything; evetest-pr.yml
accepts it only in the form of that PR's versions. The coverage nightly
takes the version from its build job outputs. The download-image-artifacts
path of evetest-run-suite is removed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The job summary heading was emitted as an HTML <h2> directly followed
by Claude's markdown report, so the report's first line (its title)
was swallowed into the HTML block and shown as literal text. Its
sub-headings also rendered at the same level as the section heading.

Write the section heading as markdown followed by a blank line, and
run the report through shift-headings.sh so its headings nest below it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@milan-zededa
milan-zededa merged commit 9e884c7 into lf-edge:master Nov 9, 2026
60 of 83 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s) testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants