Skip to content

[14.5-stable] Update ca-certificates to 20261020-r0 to stop trusting the compromised eMudhra CAs. - #6913

Merged
rene merged 2 commits into
lf-edge:14.5-stablefrom
shjala:backport/14.5-ca-certificates-20261020
Nov 4, 2026
Merged

rene merged 2 commits into
lf-edge:14.5-stablefrom
shjala:backport/14.5-ca-certificates-20261020

Conversation

@shjala

@shjala shjala commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Description

Backport of #6891 to 14.5-stable.

Update the CA packages to 20261020-r0 so the system trust store no longer trusts the emSign roots removed after the eMudhra intermediate-CA compromise. Both commits were cherry-picked from master with -x:

  • 2a6d5b0313f8e337e100488fe71885ac23a5186a
  • 4f93991f4bd186894815708541427cc3b14f8b11

This branch uses Alpine 3.16. Its repository still provides ca-certificates 20240226-r0, and its cache builder deliberately preserves existing APKs. A tag-only backport of the master change therefore would not update the certificates.

The first commit is adapted to build the checksum-pinned 20261020 CA source using the Alpine 3.16 package recipe and the branch's existing OpenSSL development package. Both ca-certificates and ca-certificates-bundle are replaced in each populated cache before reindexing and signing, including packages fetched as dependencies. The existing Alpine and Go versions are preserved.

The second commit updates downstream Alpine references and regenerates dependent tags from this stable tree. This branch's hash checker includes cross-compilers, so its image references are updated too. Existing package selection and branch-specific hash-check exclusions are preserved; packages absent on this branch are not introduced.

New Alpine image tag: lfedge/eve-alpine:80c5dbe1085d0650030be8a110eaec66ef532672.

How to test and validate this PR

Completed locally:

  • Fixed the CI-reported tabs and Dockerfile RUN cd lint issues within the original first commit, preserving generated heredoc contents. Regenerated dependent image references in the original second commit (including Dockerfile.spec). The full local Yetus suite now returns +1 overall (codespell, blanks, hadolint, shellcheck); hash consistency also passes after these changes.

  • Rebased the original two commits onto current 14.5-stable (aeed12555), preserving both -x trailers and the base branch’s GitHub cross-compiler download fix. Resolved image-reference conflicts and regenerated the cross-compiler and OP-TEE hashes; no merge commit added. Hash consistency, shell syntax, and whitespace checks passed again after rebasing.

  • make -o pkg-deps.mk check-docker-hashes-consistency passes with this branch's checker. -o skips dependency-file regeneration, not the hash check.

  • The newer master checker also passes with this branch's configured exclusions, including references resolved through Dockerfile ARG.

  • git diff upstream/14.5-stable HEAD --check and sh -n on the modified/new shell files pass.

  • Verified that both master -x references are retained and changes outside pkg/alpine are image-tag substitutions only.

  • Built the shared package recipe and exercised the cache replacement script in an amd64 Alpine 3.16 container. Both CA packages install as 20261020-r0; c_rehash links against OpenSSL 1.1 and update-ca-certificates --fresh succeeds.

  • Verified removal of the affected emSign roots and AAA Certificate Services from the generated CA configuration. The regenerated certificate bundle is byte-for-byte identical to the bundle in the updated master Alpine image (b4d0b9fe886ec5283e91da1ade3a47ed537bb47a); SHA-256: 5961c0a3c65459931c023e0075c09733f09aea4ca439cb85bc4a08bb4f3f568b.

Still required:

  1. Build the full branch-specific Alpine/EVE images. In the new Alpine image, run apk info -v | grep '^ca-certificates' and verify both packages are 20261020-r0.
  2. Run update-ca-certificates --fresh and inspect /etc/ca-certificates.conf and /etc/ssl/certs/ca-certificates.crt; the affected emSign G3/C3/C1 roots and AAA Certificate Services must be absent. The unrelated emSign G1 root is not part of this removal.
  3. Repeat the compromised-intermediate verification from Update ca-certificates to 20261020-r0 to stop trusting the compromised eMudhra CAs. #6891 against the resulting bundle.
  4. Test controller onboarding and public HTTPS datastore downloads on amd64 and arm64 devices.

The local container test covers package building, installation, and cache replacement, not a full EVE image build. Hardware tests and other architecture builds have not been performed locally.

Changelog notes

Update the system CA certificate bundle to 20261020 and remove trust in the emSign roots associated with compromised eMudhra intermediate certificate authorities.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs:

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't check them.

The build adaptation is documented in code comments and this description; no separate user documentation is added. Device checkboxes remain unchecked because validation was performed in an amd64 container, not on EVE hardware.

@shjala
shjala requested review from rene and rucoder as code owners October 22, 2026 16:44
@shjala shjala added stable Should be backported to stable release(s) security Provides a security fix labels Oct 22, 2026
@shjala
shjala requested a review from milan-zededa as a code owner October 22, 2026 16:44
@shjala shjala added the next-14.5.x-rc PR must be present in the next 14.5.x patch release label Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @shjala's task in 2m 42s —— View job


Review: ca-certificates 20261020 backport to 14.5-stable

  • Read PR context and diff vs origin/14.5-stable
  • Review pkg/alpine/Dockerfile, build-cache.sh, and new ca-certificates/* recipe
  • Verify multi-stage/riscv64 behavior and cache-replacement logic
  • Confirm the 50 image-tag bumps are tag-only
  • Post review

Assessment

This is a careful, well-scoped backport. The core problem — Alpine 3.16's repo only ships ca-certificates 20240226-r0, and build-cache.sh deliberately never overwrites existing APKs — is solved correctly by building a checksum-pinned 20261020 package locally and swapping it into each populated cache before reindexing/signing.

I verified:

  • pkg/alpine/Dockerfile — the abuild step and the mirror-building RUN loop both execute in the cache-build stage (cache-${TARGETARCH}), so the CA package is rebuilt for all three arches including riscv64 (riscv64's prebuilt base runs the same step). No arch silently keeps the old bundle.
  • build-cache.sh — the replacement loop handles ca-certificates and ca-certificates-bundle independently; the globs (${p}-[0-9]*.apk) correctly avoid matching the -doc subpackage and avoid cross-matching bundle during the ca-certificates iteration. With set -e, a missing arch APK would fail the build loudly rather than ship stale certs — good fail-fast behavior given this is a security fix.
  • APKBUILD / .trigger / .post-deinstall — faithful to the Alpine 3.16 aports recipe; install=/triggers= scripts are correctly shipped from startdir (no sha512sums entry needed, matching upstream), and only the source= tarball is checksum-pinned. The v3.22 distfiles path with pkgver=20261020 is the intentional cross-version source pin.
  • Image-tag bumps — spot-checked pillar, cross-compilers, eve/Dockerfile.in; all are eve-alpine/dependent-image tag substitutions only, no behavioral changes.

No correctness, security, or performance issues found. The dependent-tag regeneration and hash-check consistency are the main things that still need the full CI build to confirm (as the PR description already calls out under "Still required").
· branch backport/14.5-ca-certificates-20261020

Comment thread pkg/alpine/ca-certificates/APKBUILD Outdated
source="https://distfiles.alpinelinux.org/distfiles/v3.22/ca-certificates-$pkgver.tar.bz2"

build() {
make

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shjala , could you please fix these Yetus issue....

@eriknordmark

Copy link
Copy Markdown
Contributor

Go Tests fails here because the bpftrace-compiler tests run past their 1h timeout on the self-hosted runners. #6936 raised the timeout to 5h on 14.5-stable. Re-running doesn't help, because the job tests the PR head. Please rebase onto the current 14.5-stable.

@eriknordmark

Copy link
Copy Markdown
Contributor

@shjala some conflicts to fix

@shjala
shjala force-pushed the backport/14.5-ca-certificates-20261020 branch from 1c81c22 to cb089ec Compare November 3, 2026 09:09
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @shjala's task in 4m 35s —— View job


Review: ca-certificates 20261020 backport to 14.5-stable

  • Read PR context and diff vs origin/14.5-stable
  • Review new ca-certificates/* recipe, build-cache.sh, and Dockerfile changes
  • Verify the 50 image-tag bumps are tag-only
  • Post review

Assessment

The core approach is sound: Alpine 3.16 only ships ca-certificates 20240226-r0 and build-cache.sh never overwrites existing APKs, so building a checksum-pinned 20261020 package and swapping it into each populated cache before reindex/sign is the right fix. The cache-replacement globs (${p}-[0-9]*.apk) correctly avoid matching -bundle/-doc during the ca-certificates iteration, each package is guarded independently, and set -e makes a missing-arch APK fail loudly rather than ship stale certs. The tag bumps spot-check as pure eve-alpine/dependent-image substitutions.

I found two lint items that will likely fail the in-progress Apache Yetus job (posted as inline comments):

  1. pkg/alpine/Dockerfile:33 — new RUN cd … && abuild … trips hadolint DL3003; it's the only RUN cd in the file and .hadolint.yaml doesn't ignore DL3003. Add # hadolint ignore=DL3003 or use WORKDIR.
  2. pkg/alpine/ca-certificates/APKBUILD — corroborates the earlier reviewer note about Yetus. shellcheck will warn SC2164 (cd without || exit inside the conf-generating subshell) and SC2154 for abuild-provided vars; suggested fixes inline.

No correctness, security, or performance issues in the recipe or cache logic itself. The remaining validation (full branch image build, bundle verification, onboarding/datastore tests) is the hardware/CI work already tracked in the PR description, plus the pending rebase requested by the maintainer.
· branch backport/14.5-ca-certificates-20261020

@github-actions
github-actions Bot requested a review from rene November 3, 2026 09:10
Comment thread pkg/alpine/Dockerfile Outdated
# Alpine 3.16 no longer receives CA bundle updates. Rebuild the current
# certificate data against this branch's libraries instead of upgrading Alpine.
COPY ca-certificates /tmp/ca-backport/main/ca-certificates/
RUN cd /tmp/ca-backport/main/ca-certificates && abuild -F -P /tmp/ca-packages -r

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This RUN cd … && … will trip hadolint DL3003 ("Use WORKDIR to switch to a directory"). The repo's .hadolint.yaml only ignores DL3059 and DL3002, and this is the only RUN cd in the file — every other directory switch here uses WORKDIR or a per-line # hadolint ignore=. So Apache Yetus/hadolint will likely fail on this line.

Either add an ignore comment (matching the existing DL3018/DL3059 pattern in this file):

Suggested change
RUN cd /tmp/ca-backport/main/ca-certificates && abuild -F -P /tmp/ca-packages -r
# hadolint ignore=DL3003
RUN cd /tmp/ca-backport/main/ca-certificates && abuild -F -P /tmp/ca-packages -r

or switch to WORKDIR /tmp/ca-backport/main/ca-certificates before the RUN (the next step already resets WORKDIR /tmp/mirrors).

Comment thread pkg/alpine/ca-certificates/APKBUILD Outdated
echo "# Automatically generated by $pkgname-$pkgver-r$pkgrel"
echo "# $(date -u)"
echo "# Do not edit."
cd "$pkgdir"/usr/share/ca-certificates

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the Yetus issue flagged earlier on this file: shellcheck (which Yetus runs) will warn here on SC2164 — cd without failure handling. Since this cd is inside a subshell that redirects into ca-certificates.conf, a failed cd would silently generate the conf from the wrong directory, so the fix is also correct on its own merits:

Suggested change
cd "$pkgdir"/usr/share/ca-certificates
cd "$pkgdir"/usr/share/ca-certificates || exit 1

Note that shellcheck will additionally emit SC2154 for the abuild-provided vars ($pkgdir, $pkgname, $subpkgdir, …), which it can't see are assigned. If those are part of the Yetus failures, add a # shellcheck disable=SC2154 (and shell=sh) directive near the top of the file rather than touching each use.

shjala and others added 2 commits November 3, 2026 10:17
Eight eMudhra intermediate CAs were found with recoverable P-384 keys
and revoked. They chain to emSign ECC Root CA - G3, which our current
ca-certificates bundle (20260909) still trusts. EVE does not check
revocation, so a TLS server certificate issued under one of them would
be accepted.

Alpine 3.22 ships ca-certificates 20261020-r0, which drops that root.
Rebuild eve-alpine on top of the current cache so it picks up the new
bundle. build-cache.sh refetches every listed package, so other
packages updated upstream since the last build come along too.

Backport to Alpine 3.16: its repository still ships ca-certificates
20240226, and its cache builder does not refresh existing packages.
Build the checksum-pinned 20261020 source with the Alpine 3.16 recipe
and the branch's OpenSSL, then replace both CA packages in every
populated mirror before reindexing. Preserve the Alpine and Go versions.

Signed-off-by: Shahriyar Jalayeri <shahriyar@posteo.de>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2a6d5b0)
Point every package at the refreshed eve-alpine so EVE images ship
ca-certificates 20261020-r0, which no longer trusts emSign ECC Root
CA - G3 and the compromised eMudhra intermediates that chain to it.

Packages built from eve-alpine get a new tag when eve-alpine changes,
and so do the packages that pin them. Point every package that pins
one of them (optee-os, bpftrace, dom0-ztools, fscrypt, gpt-tools,
recovertpm, storage-resizer, uefi, debug) at its new tag.

Backport adaptation: regenerate image tags from this stable branch.
Its hash-consistency check includes cross-compilers, so refresh that
image and its consumers too. Preserve the branch-specific package set
and the existing hash-check exclusions.

Signed-off-by: Shahriyar Jalayeri <shahriyar@posteo.de>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4f93991)
@shjala
shjala force-pushed the backport/14.5-ca-certificates-20261020 branch from cb089ec to 03ea036 Compare November 3, 2026 09:19
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @shjala's task in 3m 58s —— View job


Review: ca-certificates 20261020 backport to 14.5-stable

  • Read PR context and diff vs origin/14.5-stable
  • Review new ca-certificates/* recipe, Dockerfile, and build-cache.sh
  • Verify abuild output layout vs. the cache-replacement copy
  • Confirm the ~50 image-tag bumps are tag-only
  • Post review

Assessment

The overall approach is correct: Alpine 3.16 only ships ca-certificates 20240226-r0 and build-cache.sh never overwrites existing APKs, so building a checksum-pinned 20261020 package locally (abuild) and swapping it into each populated cache before reindex/sign is the right fix. The abuild step and the mirror loop both run in the cache-build stage, so /tmp/ca-packages is available to build-cache.sh for all arches. The earlier hadolint DL3003 issue is resolved (now uses WORKDIR). The tag bumps spot-check as pure eve-alpine/dependent-image substitutions.

One likely build break (inline on build-cache.sh:45): ca-certificates-bundle is a noarch subpackage, so abuild writes it to /tmp/ca-packages/main/noarch/, but the copy loop reads from /tmp/ca-packages/main/$(apk --print-arch)/. For the bundle iteration the cp glob matches nothing; with the preceding rm -f and set -e, any cache that contains ca-certificates-bundle would delete the old bundle and then abort the build. The arch-specific ca-certificates package itself is unaffected. Fix suggested inline (copy from main/*/). This path is worth confirming against the full image build already listed under "Still required" — the local container test may not have exercised the bundle copy.

Minor (lint, already flagged by @rene / earlier bot comments, still open): the APKBUILD package() still has a bare cd "$pkgdir"/usr/share/ca-certificates inside the conf-generating subshell. If Apache Yetus runs shellcheck on this recipe it will emit SC2164 (and SC2154 for abuild-provided vars). A cd … || exit 1 plus a top-of-file # shellcheck disable=SC2154 / shell=sh directive would clear it. Not a correctness issue on its own.

No other correctness, security, or performance concerns in the recipe or cache logic. Remaining validation (full branch image build, bundle verification, onboarding/datastore tests on amd64/arm64) is the hardware/CI work already tracked in the PR description.
· branch backport/14.5-ca-certificates-20261020

Comment thread pkg/alpine/build-cache.sh
for p in ca-certificates ca-certificates-bundle; do
if ls "$CACHE/${p}"-[0-9]*.apk >/dev/null 2>&1; then
rm -f "$CACHE/${p}"-[0-9]*.apk
cp /tmp/ca-packages/main/"$(apk --print-arch)"/"${p}"-[0-9]*.apk "$CACHE/"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likely build break: the ca-certificates-bundle copy looks in the wrong directory.

ca-certificates-bundle is declared as a noarch subpackage (subpackages="$pkgname-doc $pkgname-bundle::noarch" in the APKBUILD). abuild -P /tmp/ca-packages writes packages to $REPODEST/$repo/$arch, and noarch subpackages go to the noarch arch directory — so the bundle lands in /tmp/ca-packages/main/noarch/, not /tmp/ca-packages/main/$(apk --print-arch)/.

For the ca-certificates-bundle iteration, the cp glob therefore matches nothing. Combined with the preceding rm -f and the script's set -e, any populated cache that contains ca-certificates-bundle will have the old bundle deleted and then fail the cp → the whole cache build aborts. (The arch-specific ca-certificates package itself is fine, since it goes to main/$arch/.)

Suggest copying from whichever arch dir abuild used so both the arch package and the noarch bundle resolve:

Suggested change
cp /tmp/ca-packages/main/"$(apk --print-arch)"/"${p}"-[0-9]*.apk "$CACHE/"
cp /tmp/ca-packages/main/*/"${p}"-[0-9]*.apk "$CACHE/"

Worth confirming against a real full image build (per the "Still required" items) — the local container test may not have gone through this exact build-cache.sh path for the bundle.

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

evetest: skipped suites for this PR

This PR is a backport that bumps the eve-alpine base image (and every package
that pins it) to a cache rebuilt with ca-certificates/ca-certificates-bundle
20261020-r0, so the trust store no longer trusts the emSign roots tied to the
revoked eMudhra intermediate CAs. All other file changes are mechanical image
tag updates (pkg/*/Dockerfile) that follow from that rebuild — no runtime
pillar/Go code or feature behavior changes.

The only behavior that actually changes is which root CAs the system trust
store accepts for generic outbound TLS. TestDatastoreSuite is kept because it
is the dedicated suite for datastore download/certificate-trust plumbing
(including TestAWSDatastore/TestAzureDatastore, which validate TLS against
real external endpoints using the default system trust store).

Skipped as unrelated to that change (each suite's own purpose has nothing to
do with the OS CA trust store):

  • TestApplicationConnectivitySuite — network-instance/app networking plumbing (NI, ACLs, VLANs), no TLS/cert involvement.
  • TestAppsSuite — app lifecycle (restart, purge, VNC, user-data, OCI layer download bounding), unrelated to CA trust.
  • TestBootstrapSuite — onboarding/provisioning mechanisms (static IP, proxy, VLAN, network boot); the device trusts the controller via the pinned onboarding certificate, not the OS CA bundle.
  • TestControllerFaultsSuite — controller fault-injection/retry behavior (proxy drops/rejects requests), not certificate trust.
  • TestDeviceConnectivitySuite — DHCP/static IP/DNS/port-failover/cellular/wifi connectivity, no TLS/cert aspect.
  • TestDiagSuite — diagnostic output.
  • TestHardwareSuite — USB flash drive hotplug.
  • TestLPSSuite — local profile server / radio silence / local info reporting.
  • TestPcibackErrorSuite — PCI passthrough error reporting.
  • TestPortScanSuite — external port scan.
  • TestSecuritySuite — AppArmor, vTPM, and controller signing/encryption certificate rotation; these exercise EVE's own application-level ciphers derived from the controller's certs, not the OS ca-certificates trust store.
  • TestStorageSuite — volumes/disk/ZFS layout.
  • TestTelemetrySuite — device info/metrics/logs reporting content.
  • TestUpgradeSuite — EVE image upgrade flow; onboarding/download path is pinned/covered more directly by TestDatastoreSuite.

An engineer can comment /evetest all to force the full suite to run instead,
or /evetest <name> <name> ... to run only specific tests/suites.

@rene
rene merged commit fd33e31 into lf-edge:14.5-stable Nov 4, 2026
78 of 88 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

next-14.5.x-rc PR must be present in the next 14.5.x patch release security Provides a security fix stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants