Repository navigation
ci: stop re-downloading cosign/syft on guests that already have them - #572
Merged
Merged
Conversation
`docker-build` called `sign-and-attest` without `install-tools`, so the input defaulted to `true` and the action ran `sigstore/cosign-installer` + `anchore/sbom-action/download-syft` unconditionally — on `pcbuild` guests whose rootfs already ships cosign v2.4.1 and syft v1.16.0 at exactly the pinned versions. `install-attest-tools` is the action that consults that preinstall (exact version match on PATH, falling through to a download on mismatch). `sign-and-attest` has no such guard. Measured in release run 41086127339, both jobs on guests that had both tools: build-platforms install-attest-tools "preinstalled: true" 0.45 s docker-build sign-and-attest downloads anyway 83.7 s 74 of those 83.7 s were a stall with no output, on the fake-IP egress the fleet design doc §5.1 records as intermittently hanging. The same path cost 7m11s in run 39402578614 and hard-failed run 40007865716 attempt 2 (`failed to install syft`), discarding a release. Nothing surfaces it: the consuming steps are continue-on-error. So add an `install-attest-tools` step to `docker-build` — the same step `build-platforms` and `docker-finalize` already have — and pass `install-tools: 'false'` to `sign-and-attest`. Same fix in `branch-preview.yaml`, whose two legs are both `heavy_go` and were paying it twice per preview build. On the hosted legs (caddy, kubectl) neither binary is present, both guards miss, and this downloads as before at ~4 s — a strict improvement on all four legs, not a fleet-only trade. This is item 2 of forge's `provisioning/firecracker-buildvm/OWED-rootfs-attest-tools.md`: the guest bake landed 2026-10-24 and was verified live, but `api` never consulted it. Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
smecsia
requested review from
Cre-eD,
Laboratory and
universe-ops
as code owners
November 8, 2026 16:52
Semgrep Scan ResultsRepository:
Scanned at 2026-11-08 16:53 UTC |
Security Scan ResultsRepository:
Scanned at 2026-11-08 16:54 UTC |
baasforger
approved these changes
Nov 8, 2026
universe-ops
approved these changes
Nov 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
docker-buildcallssign-and-attestwithout passinginstall-tools, so the input defaults to'true'and the action runssigstore/cosign-installer+anchore/sbom-action/download-syftunconditionally — onpcbuildguests whose rootfs already ships cosign v2.4.1 and syft v1.16.0 at exactly the pinned versions.install-attest-toolsis the action that consults that preinstall (exact version match on PATH; a mismatch falls through and downloads, so a wrong preinstall costs one download, never a wrong tool).sign-and-attesthas no such guard.Measured
Release run 41086127339, both jobs on
pcbuildguests that had both tools:Build sc for linux/amd64install-attest-toolscosign preinstalled at v2.4.1: true; syft at v1.16.0: trueDocker build and push github-actionssign-and-attest74 of those 83.7 s were a stall with no output before
cosign-installerprinted its first line — the fake-IP egress that the fleet design doc §5.1 records as intermittently hanging. The same path cost 7 min 11 s in run 39402578614 and hard-failed run 40007865716 attempt 2 (failed to install syft), discarding an entire release. Nothing surfaces it, because the consuming steps arecontinue-on-error.Full sub-step breakdown of that 273 s
Sign + atteststep, from the raw job log:The change
Add an
install-attest-toolsstep todocker-build— the same stepbuild-platforms(push.yaml:294) anddocker-finalize(push.yaml:1334) already have — and passinstall-tools: 'false'tosign-and-attest.Same fix in
branch-preview.yaml, whose twodocker-buildlegs are bothheavy_go: trueand so were paying the download twice on every preview build.On the hosted legs (
caddy,kubectl) neither binary is present, both guards miss, and this downloads exactly as before at ~4 s. Strict improvement on all four legs, not a fleet-only trade.Expected effect
github-actionsleg:Sign + attest273 s → ~190 scloud-helpers-awsleg: 196 s → ~115 scaddy/kubectl: unchanged (~41 s / ~37 s)Roughly −165 s per release and −165 s per preview build, plus removing two exposures per run to the egress path that has already discarded one release outright.
Risk
If the new
install-attest-toolsstep flakes (it iscontinue-on-error, matchingbuild-platforms), the sign/SBOM/attest sub-steps now fail their own version checks and the release gate reports it, rather than silently falling back to another minute of egress. That is the intended posture: a flake should be visible. The preview workflow has no gate, so there it surfaces through the existing warn-only aggregator.No change to the attestation chain, the cosign identity, the pinned tool versions, or any
runs-on.Provenance
Item 2 of
forge/provisioning/firecracker-buildvm/OWED-rootfs-attest-tools.md. The guest bake landed 2026-10-24 (actions#50 +bake-attest-tools-into-rootfs.sh) and was verified live, butapinever consulted it — so the fleet has been carrying the tools for two weeks whiledocker-builddownloaded them anyway. Re-verified in the base rootfs today:GitVersion: v2.4.1,Version: 1.16.0.Not in this PR
The 162 s CycloneDX SBOM step is now the largest line item in
Sign + attest—syft scan registry:…pulls the whole 1.36 GB image back down from Docker Hub through the same egress, immediately after pushing it. Worth its own change; measuring first.