Skip to content

CI performance: get-deps: depend only on reachable Dockerfile stages - #6957

Merged
rene merged 2 commits into
lf-edge:masterfrom
eriknordmark:get-deps-reachable
Oct 29, 2026
Merged

rene merged 2 commits into
lf-edge:masterfrom
eriknordmark:get-deps-reachable

Conversation

@eriknordmark

Copy link
Copy Markdown
Contributor

Description

A native build of pkg/pillar, pkg/qemu or pkg/bsp-imx made Make build
eve-cross-compilers first, although buildkit never builds the Dockerfile
stage that uses it. When that image is missing from Docker Hub for the host
arch, Go Tests and Fuzz Tests compile the gcc cross toolchains for hours and
Fuzz Tests is cancelled at the 6 h limit (#6942, #6953).

get-deps now follows the FROM, COPY --from and RUN --mount from=
references from the last stage, for the target arch on the build host's arch,
and lists only the images those stages use. When a stage selection does not
resolve (riscv64 pillar and qemu have no riscv64 stage), it keeps every stage
and says so. The deps file is per host and target arch,
pkg-deps-$(HOSTARCH)-$(ZARCH).mk, and also covers the host arch for the
packages in PKGS_HOSTARCH.

The first commit fixes the Dockerfile scanner, which took a comment ending in
\ as a continuation and dropped the next instruction. buildkit does not. In
the tree that hid pkg/qemu's COPY --from=uefi-build; no FROM is affected,
so check-docker-hashes-consistency output is unchanged.

Generated deps change only by removals, each matching its Dockerfile:
cross-compilers from pillar, qemu and bsp-imx on native builds, and alpine from
optee-client (amd64, riscv64) and bpftrace (riscv64), whose final stage is
scratch there. arm64 on amd64 is unchanged.

How to test and validate this PR

(cd tools/dockerfile-from-checker && go test ./dockerfilefrom/)
(cd tools/get-deps && go mod vendor && go test .)
make -n pkg/pillar | grep -c 'pkg build.*pkg/cross-compilers'            # 0
make -n ZARCH=arm64 pkg/pillar | grep -c 'pkg build.*pkg/cross-compilers' # 1

On master both make -n lines print 1. In CI, Go Tests and Fuzz Tests should
not build eve-cross-compilers even when its tag is not on Docker Hub.

Changelog notes

None.

PR Backports

  • 17.0-stable: No, CI build time only.
  • 16.0-stable: No, CI build time only.
  • 14.5-stable: No, CI build time only.
  • 13.4-stable: No, CI build time only.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them. Build-tooling change only: no documentation or device testing
    applies.

🤖 Generated with Claude Code

eriknordmark and others added 2 commits October 28, 2026 14:52
The Dockerfile scanner treated a comment line ending in the escape
character as the start of a continuation, so it folded the next
instruction into the comment and dropped it. buildkit strips comment
lines before it looks for a continuation, so that instruction is
built. In the tree this hides pkg/qemu's COPY --from=uefi-build; no
FROM is affected, so check-docker-hashes-consistency reports the same.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A native build of pillar, qemu or bsp-imx made Make build
eve-cross-compilers first, although buildkit never builds the stage
that uses it. When that image is not on Docker Hub for the host arch,
Go Tests and Fuzz Tests compile the gcc cross toolchains for hours.

get-deps now follows the FROM, COPY --from and RUN --mount from=
references from the last stage, for the target arch on the build
host's arch, and lists only the images those stages use. When a
stage selection does not resolve, as for riscv64 pillar, it keeps
every stage. The deps file is per host and target arch, and covers
the host arch too for packages built for both.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eriknordmark eriknordmark changed the title get-deps: depend only on reachable Dockerfile stages CI performance: get-deps: depend only on reachable Dockerfile stages Oct 28, 2026
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.76%. Comparing base (02ad7ce) to head (e7fccde).
⚠️ Report is 11 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6957      +/-   ##
==========================================
+ Coverage   28.48%   28.76%   +0.28%     
==========================================
  Files         540      551      +11     
  Lines       98258   100005    +1747     
==========================================
+ Hits        27990    28769     +779     
- Misses      67998    68774     +776     
- Partials     2270     2462     +192     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark

Copy link
Copy Markdown
Contributor Author

Fork A/B test: on eriknordmark/eve, both branches carry two test-only commits, one that moves Go Tests and Fuzz Tests to ubuntu-24.04 and one that changes pkg/cross-compilers so its tag is not on Docker Hub.

master this PR
Fuzz Tests eve-cross-compilers … amd64 not found, will build, then bootstrap.sh "riscv64"; cancelled after 25 min (run) success in 10 min, no cross-compilers build (run)
Go Tests same build; cancelled 22 min into bootstrap.sh (run) no cross-compilers build, 2919 pillar tests pass (run)

The PR side's Go Tests run is red because, after the pillar tests, make test's pkg/debug build got a 403 from ezix.org for the lshw tarball on the hosted runner.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rene
rene merged commit 68969db into lf-edge:master Oct 29, 2026
47 of 51 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Skipping the full evetest suite for this PR — it only changes CI/build tooling (Makefile dependency-graph generation in tools/get-deps and the Dockerfile FROM/COPY --from/RUN --mount=from= scanner in tools/dockerfile-from-checker, plus a stale comment fix) so native pkg builds stop pulling in unreachable Dockerfile stages (e.g. eve-cross-compilers). No pillar or device-runtime code is touched, so none of these suites exercise the changed code path:

  • TestApplicationConnectivitySuite
  • TestAppsSuite
  • TestBootstrapSuite
  • TestControllerFaultsSuite
  • TestDatastoreSuite
  • TestDeviceConnectivitySuite
  • TestDiagSuite
  • TestHardwareSuite
  • TestKonvertSuite
  • TestKubeRegistrationSuite
  • TestLPSSuite
  • TestNodeClusterResilienceSuite
  • TestNodeClusterSuite
  • TestPcibackErrorSuite
  • TestPortScanSuite
  • TestSecuritySuite
  • TestStorageSuite
  • TestTelemetrySuite
  • TestUpgradeSuite

The PR's own test plan instead validates the change via the dockerfilefrom and get-deps Go unit tests.

An engineer can comment /evetest all to force the full suite to run instead, or /evetest <name> <name> ... to run only specific tests/suites.

@eriknordmark
eriknordmark deleted the get-deps-reachable branch October 31, 2026 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants