Skip to content

debug: use Alpine OpenSSH password-authentication fix - #7046

Closed
shjala wants to merge 1 commit into
lf-edge:masterfrom
shjala:upgrade/openssh-10.6p2
Closed

shjala wants to merge 1 commit into
lf-edge:masterfrom
shjala:upgrade/openssh-10.6p2

Conversation

@shjala

@shjala shjala commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Description

Use Alpine’s fixed OpenSSH packages on master.

Install Alpine 3.22 OpenSSH 10.0_p1-r11, which includes the upstream musl password-authentication bypass fix. The upstream version number stays 10.0p1 because Alpine backported the fix in package revision r11: Alpine patch.

Use official signed APKs pinned by version and SHA-256 for amd64, arm64 and riscv64, without rebuilding the shared Alpine package cache. Architecture-specific ADD --checksum stages fetch the APKs; installation runs offline, matching LinuxKit’s network-disabled build steps. Alpine's package database supplies the SBOM version. Update the installer debug-image pin.

Password and keyboard-interactive authentication remain disabled. The isolated SSH test deliberately enables password authentication for disposable accounts to check the bypass, public-key login and SFTP. Test accounts and keys are not shipped.

How to test and validate this PR

  • Local amd64 full debug test-stage build with --network=none passed, including SSH regression, public-key login, SFTP, production authentication settings and collect-info tests.
  • make check-docker-hashes-consistency and SPDX checks passed. Independently compared the installer pin against linuxkit pkg show-tag pkg/debug.
  • No new Hadolint findings compared with the target branch.
  • Build with docker build --network=none --target test pkg/debug.
  • On a device, verify authorized SSH/EdgeView access and SFTP. Check apk info -v reports openssh-server-10.0_p1-r11; ssh -V alone does not show Alpine's patch revision.
  • Physical-device, arm64 and riscv64 execution are not claimed; CI must validate the pushed revision.

Changelog notes

Fix the OpenSSH musl password-authentication bypass using Alpine's patched packages. SSH access remains public-key only.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation (build rationale and test instructions)
  • I've tested my PR on amd64 device (local container tests only)
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or provided validation limitations

@shjala shjala added stable Should be backported to stable release(s) security Provides a security fix labels Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @shjala's task in 4m 34s —— View job


Code Review: Upgrade OpenSSH to 10.6p2

  • Reviewed Dockerfile multi-stage changes and runtime-image isolation
  • Reviewed the new tests/openssh.sh regression
  • Checked SBOM registration, privsep user, config include, and effective-config test
  • Posted inline findings

Summary

Solid, well-contained change. Key correctness checks pass review:

  • No test leakage into the runtime image. openssh.sh writes test accounts/keys//dev/null into /out, but it only runs in the openssh-test and test stages; the final scratch image copies --from=build, so disposable accounts, host keys and /ssh-test do not ship. Confirmed.
  • Privsep plumbing is coherent. The sshd user is added in the openssh stage and propagated into /out/etc/passwd (build stage cp, and explicitly in openssh-test). make install creates /var/empty, and the build-time sshd start in the test stage exercises this path, so runtime startup risk is low despite no on-device test.
  • Production config preserved. The test stage runs sshd -T against the final /out config and asserts PasswordAuthentication no / KbdInteractiveAuthentication no — a good regression guard for the pubkey-only posture.
  • The regression test is meaningful (reproduces the bypass against 10.6p1 per the description, and asserts rejection + pubkey + SFTP still work).

Inline findings (both low severity)

  1. Integrity now rests solely on the SHA-256 pin; the GPG signature check is manual-only (not reproduced in-build), and the version is now manually tracked rather than via apk upgrade — relevant for future CVE bumps and the planned backports.
  2. rm of ssh-keysign without -f is slightly fragile.

Cross-cutting considerations (non-blocking)

  • Build cost: OpenSSH is now compiled from source in every debug-package build, including emulated arm64/riscv64 under QEMU, which will add noticeable build time. Worth confirming CI budgets are fine.
  • Backports: the PR notes 17.0/16.0/14.5/13.4 backports are to follow; the manual version+hash pin means each branch must be bumped independently.

No blocking issues found.

Comment thread pkg/debug/Dockerfile Resolved
Comment thread pkg/debug/Dockerfile Resolved
@shjala
shjala force-pushed the upgrade/openssh-10.6p2 branch from d66fefe to 8553375 Compare November 5, 2026 16:33
@shjala
shjala requested review from rene and rucoder as code owners November 5, 2026 16:33
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @shjala's task in 2m 56s —— View job


Code Review: Upgrade OpenSSH to 10.6p2

  • Read changed files (debug Dockerfile, tests/openssh.sh, installer Dockerfile)
  • Analyzed build/runtime isolation, SBOM, privsep, config posture
  • Reviewed the regression test correctness
  • Checked runtime host-key generation path and test gating

Verdict

No blocking issues. The change is well-contained and the security-relevant invariants hold. The two findings from the earlier review pass are resolved on the current commit (rm -f for ssh-keysign is applied; the SHA-256 pin + manual version tracking tradeoff is acknowledged and acceptable since the pin is reproducible).

What I verified

  • No test leakage into the runtime image. openssh.sh writes disposable accounts/keys//ssh-test into /out, but it only runs in the openssh-test and test stages. The final scratch image copies --from=build, so none of those artifacts ship. Confirmed.
  • Runtime host keys are not a regression. install-nokeys intentionally ships no host keys; ssh-service.sh still runs ssh-keygen -A on startup when none exist, so sshd gets keys at boot as before.
  • Privsep is exercised at build time. The openssh.sh regression starts a real privsep sshd (UsePAM no, default privilege separation) and only proceeds once it accepts connections — this implicitly validates that /var/empty and the sshd user exist in /out, which is the main on-device startup risk given no device testing was done.
  • Production posture preserved. The test stage runs sshd -T against the final /out config and asserts PasswordAuthentication no / KbdInteractiveAuthentication no. The upstream sshd_config does not ship an Include for sshd_config.d, so adding it at line 1 is correct and non-duplicate, and first-match semantics make the pubkey-only drop-in authoritative.
  • Regression is meaningful and CI-reachable. It sends a wrong password then a 300-char password on the same connection (the *sshole reproduction path), asserting rejection plus working pubkey login and SFTP. make test in pkg/debug builds --target test, so this runs under the package's test flow.
  • SBOM registration works. OPENSSH_VERSION is set in the openssh stage and inherited by the build stage, so register-sbom-pkg.sh records openssh-10.6p2.

Non-blocking notes

  • Build cost: OpenSSH now compiles from source in every debug-package build, including emulated arm64/riscv64 under QEMU — worth confirming CI time budgets are comfortable.
  • Backports: because the version+hash is manually pinned (not apk upgrade-tracked), each of the 17.0/16.0/14.5/13.4 branches and any future CVE bump must update OPENSSH_VERSION + checksum independently.
  • Integrity in-build rests solely on the SHA-256 pin (GPG verification is a documented manual step, not reproduced in the build) — acceptable, just the single in-build control.

I can't formally approve or submit a GitHub review (not permitted for this app), but from a correctness/security standpoint this looks ready.
· branch upgrade/openssh-10.6p2

@codecov

codecov Bot commented Nov 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.59%. Comparing base (d87216a) to head (8553375).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7046      +/-   ##
==========================================
+ Coverage   29.07%   29.59%   +0.52%     
==========================================
  Files         548      559      +11     
  Lines       99033   100628    +1595     
==========================================
+ Hits        28793    29782     +989     
- Misses      67969    68619     +650     
+ Partials     2271     2227      -44     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene

rene commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@rene

rene commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Actually, PR #7029 should pick the new version already.... I think it will cover this issue....

@shjala
shjala force-pushed the upgrade/openssh-10.6p2 branch from 8553375 to 9beff29 Compare November 6, 2026 09:22
@shjala shjala changed the title debug: upgrade OpenSSH to 10.6p2 debug: use Alpine OpenSSH password-authentication fix Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @shjala's task in 3m 8s —— View job


Code Review: OpenSSH password-authentication fix

  • Read current changed files (pkg/debug/Dockerfile, tests/openssh.sh, openssh.sha256, installer pin)
  • Reviewed build/runtime isolation, SBOM, signature verification and config posture
  • Reviewed the regression test correctness
  • Posted inline findings

Context note

The implementation changed since the PR was opened (commit 9beff295): it no longer builds OpenSSH 10.6p2 from source — it now installs Alpine 3.22's backported openssh-10.0_p1-r11 APKs, SHA-256-pinned per arch (following the suggestion to use Alpine's fix). The earlier bot reviews and two inline comments in this thread (manual GPG, rm -f ssh-keysign) describe the removed from-source build and no longer apply.

What holds up well

  • In-build signature verification is now real. apk add (no --allow-untrusted) verifies the official Alpine package signatures against the keys in /out, so integrity no longer rests on a manual GPG step — an improvement over the from-source variant.
  • Reproducible pinning. Archives are fetched then checked with sha256sum -c, and the installed version is re-asserted with grep -Fx "openssh-server-${OPENSSH_VERSION}".
  • No test leakage into the runtime image. openssh.sh writes disposable accounts/keys//ssh-test into /out, but it only runs in the openssh-test/test stages; the final scratch image is COPY --from=build, so none of it ships.
  • Production posture preserved. The test stage runs sshd -T against the final /out config and asserts PasswordAuthentication no / KbdInteractiveAuthentication no; the pubkey-only drop-in remains authoritative.

Findings (both posted inline, non-blocking)

  1. Version/description mismatch — PR title, body, changelog note and the ssh -V validation step still say 10.6p2, but the code ships Alpine 10.0_p1-r11 (a lower base version with the fix backported). Update the PR metadata/commit so the release notes are accurate.
  2. Regression negative-control — the test was validated to reproduce the bypass against upstream 10.6p1, but the shipped version is 10.0_p1. Worth confirming (and noting in the commit) that it still reproduces against the unpatched Alpine 10.0_p1 (pre-r11), so it genuinely guards the shipped code path rather than passing trivially.

CI

All completed checks pass (Yetus, docker-hash consistency, SBOM coverage, commit-lint, security review); PR build, Go Tests, CodeQL, Semgrep, OSV-Scanner still in progress — the docker build that exercises the test stage runs in PR build, so that's the one to watch.

No blocking issues found.
· branch upgrade/openssh-10.6p2

Comment thread pkg/debug/Dockerfile
# Alpine 3.22 backported the upstream password-authentication fix in r11.
# Install its signed APKs without rebuilding the shared Alpine package cache.
# Pin every architecture's archives to keep builds reproducible.
ENV OPENSSH_VERSION="10.0_p1-r11"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation now ships Alpine's backported openssh-10.0_p1-r11, but the PR title ("Upgrade OpenSSH to 10.6p2"), description (from-source build + manual GPG verification), the changelog note ("Update OpenSSH to 10.6p2"), and the validation step ("confirm ssh -V reports 10.6p2") all still describe the previous from-source approach. Please update the PR body/changelog/commit so the release notes match what actually ships — otherwise the published changelog will be wrong, and the two earlier inline review comments on this file (manual GPG, rm -f ssh-keysign) are now stale and no longer apply.

Note this is also a lower OpenSSH base version (10.0 vs 10.6); that's acceptable since Alpine backports the security fix, but it should be called out explicitly given the "upgrade" framing.

One upside worth noting: unlike the from-source variant, apk add here verifies the official Alpine package signatures against the keys in /out at build time (no --allow-untrusted), so signature checking is now reproduced in-build rather than being a manual step.

echo wrong-password
exit 0
fi
awk 'BEGIN { for (i=0; i<300; i++) printf "A"; print "" }'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regression was validated to reproduce the bypass against unmodified upstream 10.6p1 (per the original PR notes), but the shipped package is now 10.0_p1. Please confirm this test still reproduces the bypass against the unpatched Alpine 10.0_p1 (i.e. pre-r11) and not only the fixed build — otherwise, if 10.0_p1's code path differs, the test could pass trivially and give a false sense of security without actually guarding the version you ship. A one-line note in the commit message stating the negative-control result against pre-r11 would make this a trustworthy regression guard.

@rene

rene commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@shjala , I confirmed that with this PR #7054 Alpine will pick the fixed version of openssh. For master I prefer to follow this approach (and not merge this PR), because it also gets other updates (that also fixes other CVEs). For stable branches, we need to discuss, maybe we can keep your approach.

Install Alpine 3.22 OpenSSH 10.0_p1-r11, which backports the upstream
musl password-authentication bypass fix. Use signed, SHA-256-pinned APKs
for amd64, arm64 and riscv64 without rebuilding the shared package cache.
Alpine package metadata supplies the installed version to the SBOM.

Keep public-key-only authentication. Add an isolated regression that
requires rejection of the bypass and checks public-key login, SFTP and
production authentication settings. Refresh the installer debug image pin.

Signed-off-by: Shahriyar Jalayeri <shahriyar@posteo.de>
@shjala
shjala force-pushed the upgrade/openssh-10.6p2 branch from 9beff29 to 558cb14 Compare November 6, 2026 15:16
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @shjala's task in 3m 4s —— View job


Code Review: OpenSSH password-authentication fix

  • Read changed files (pkg/debug/Dockerfile, tests/openssh.sh, installer pin)
  • Analyze build/runtime isolation, SBOM, signature verification, config posture
  • Review regression test correctness
  • Post inline findings

The current commit (558cb147) installs Alpine 3.22's backported openssh-10.0_p1-r11 signed APKs (SHA-256-pinned per arch) instead of the earlier from-source 10.6p2 build. The PR body/changelog now describe the APK approach, so the earlier version-mismatch finding is resolved.

Findings (both posted inline, non-blocking):

  1. Dual install of openssh — openssh-client/openssh-server remain in PKGS (base install via eve-alpine-deploy.sh) and the pinned r11 APKs are force-installed on top. Redundant source of truth, and fragile if the base cache ever moves ahead of r11. Dropping openssh from PKGS would make the pin authoritative. (inline)
  2. Regression negative-control — the test only asserts rejection for a locked (*) account, which any correct sshd does. Please confirm it actually reproduces the bypass against the shipped unpatched line (Alpine 10.0_p1 pre-r11), not just upstream 10.6p1, and record that in the commit. (inline)

What holds up:

  • No test leakage: openssh.sh writes test accounts/keys//ssh-test only in the openssh-test/test stages; the final scratch image is COPY --from=build, so none of it ships.
  • In-build signature verification is real (apk add without --allow-untrusted, keys from /out).
  • Production posture preserved: the test stage runs sshd -T and asserts PasswordAuthentication no / KbdInteractiveAuthentication no.
  • Version is re-asserted via grep -Fx "openssh-server-${OPENSSH_VERSION}", so a drift would fail the build.

Cross-cutting: Per @rene's comment, master is expected to pick up Alpine's fix via #7054 rather than merge this; the manual pin is effectively redundant there. The pin is more relevant for the stable backports (14.5/13.4 need source upgrades since their Alpine would downgrade). No blocking issues found.
· branch upgrade/openssh-10.6p2

Comment thread pkg/debug/Dockerfile
# Local APK filenames and ADD checksums pin every package version.
# hadolint ignore=DL3018
RUN set -eu; \
apk add --no-network --no-cache --root /out /tmp/openssh/*.apk; \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenSSH is now installed twice: openssh-client/openssh-server are still in PKGS (line 50, unchanged from master) so eve-alpine-deploy.sh installs whatever version the pinned eve-alpine cache resolves to, and then this step force-installs the pinned r11 APKs on top.

Consequences worth addressing:

  • Redundant / confusing source of truth. The effective version depends on both the base cache and the pin. To make the pinned APKs authoritative, drop openssh-client openssh-server from PKGS so there is a single install path.
  • Fragility if the base ever moves ahead of r11. If a future eve-alpine bump ships openssh > 10.0_p1-r11, this apk add of the older local files may not take effect, and the grep -Fx guard would then fail the build on a confusing basis rather than cleanly pinning.

Given maintainer feedback in the thread (rely on Alpine's own fix via #7054 for master, since Alpine now ships the backport), the manual pin is effectively redundant on master. If this lands on stable branches instead, removing openssh from PKGS here would make the pin the single, deterministic source.

Comment on lines +54 to +63
# Password authentication is deliberately ON here: test the upstream fix,
# independently of EVE's production publickey-only mitigation.
if DISPLAY=:0 SSH_ASKPASS_REQUIRE=force SSH_ASKPASS=/ssh-test/askpass \
chroot "$root" /usr/bin/ssh -F /dev/null -p 22222 \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-o PreferredAuthentications=password -o PubkeyAuthentication=no \
-o NumberOfPasswordPrompts=2 ssh-test@127.0.0.1 true; then
echo 'ERROR: password bypass succeeded' >&2
exit 1
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regression only asserts that password auth is rejected for the * (locked) account. A correctly-behaving sshd rejects that regardless of whether the bypass bug is present, so the test has value only if this exact sequence (salt-priming attempt + 300-char password on the same connection) genuinely reproduces the bypass against the unpatched code path you ship — i.e. Alpine 10.0_p1 pre-r11, not upstream 10.6p1 which the earlier from-source variant was validated against.

Please confirm the negative control: that this test fails (bypass succeeds) when run against 10.0_p1-r0..r10, and note that result in the commit message. Otherwise, if the exploit technique doesn't apply to 10.0_p1's code, the test passes trivially on both vulnerable and fixed builds and gives false assurance for the version actually shipped.

@rene

rene commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Confirmed we already have this version on master, so this PR is not needed (let's avoid introduce more complexity and pin versions manually on master):

~/eve$ docker run --rm -it `make alpine-show-tag` /bin/sh
Unable to find image 'lfedge/eve-alpine:61283a3a978a7cdc9073a1ba678717fc3ea1fe97' locally
61283a3a978a7cdc9073a1ba678717fc3ea1fe97: Pulling from lfedge/eve-alpine
4f1eab66c075: Pull complete 
Digest: sha256:402cead0919744d55a270f697d1b2e103f6819b74a2d5856088d2c71df8e4f9a
Status: Downloaded newer image for lfedge/eve-alpine:61283a3a978a7cdc9073a1ba678717fc3ea1fe97
/ # apk add openssh
(1/10) Installing openssh-keygen (10.0_p1-r11)
(2/10) Installing ncurses-terminfo-base (6.5_p20250503-r0)
(3/10) Installing libncursesw (6.5_p20250503-r0)
(4/10) Installing libedit (20250104.3.1-r1)
(5/10) Installing openssh-client-common (10.0_p1-r11)
(6/10) Installing openssh-client-default (10.0_p1-r11)
(7/10) Installing openssh-sftp-server (10.0_p1-r11)
(8/10) Installing openssh-server-common (10.0_p1-r11)
(9/10) Installing openssh-server (10.0_p1-r11)
(10/10) Installing openssh (10.0_p1-r11)
Executing busybox-1.37.0-r20.trigger
OK: 14 MiB in 26 packages
/ # 

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is not needed on master.

@shjala shjala closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

security Provides a security fix stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants