Skip to content

evetest: port scans; fix the exposure they found on EVE and EVE-k - #6665

Merged
eriknordmark merged 5 commits into
lf-edge:masterfrom
eriknordmark:port-scan
Sep 29, 2026
Merged

eriknordmark merged 5 commits into
lf-edge:masterfrom
eriknordmark:port-scan

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds evetest coverage that scans every TCP and UDP port an EVE device exposes, using nmap from the evetest container. It covers EVE-kvm, single-node EVE-k, and every node of a three-node EVE-k cluster. On the cluster it scans both the management address and the cluster IP.

  • Any open UDP port fails.
  • SSH must offer publickey authentication only. nmap's ssh-auth-methods checks this without sending credentials.
  • Any other open TCP port must:
    • accept only TLS 1.2 or later, with ciphers that ssl-enum-ciphers grades A;
    • refuse plaintext HTTP;
    • reject a request carrying no credentials, either with 401 or by demanding a client certificate.

The harness image gains nmap and nmap-scripts, hence the evetest/VERSION bump.

The other three commits fix what the scans found:

  • sshd offers only publickey login. No account has a password, so nothing that is allowed changes. Until now, though, sshd offered password and keyboard-interactive login to anyone who connected.
  • The EVE-k cluster status server (12346/tcp) is authenticated. It served plain HTTP to anyone on the cluster network, including the App-Tracker's per-application dumps. On the cluster IP it now accepts only TLS 1.3 and only requests carrying a bearer token. Every node derives the token and the server certificate from the cluster join token, so kube-init on a joining node and zedkube on its peers authenticate each other with no key distribution. The cluster IP serves only what other nodes need, /status and /app/. The operator-facing App-Tracker, including /cluster-app/, moves to plain HTTP on 127.0.0.1:12346, which edgeview reaches via tcp/127.0.0.1:12346.
  • k3s's servicelb and Traefik stay disabled. kube-init wrote the local-storage drop-in with a bare disable: key, which replaced the base config's servicelb and traefik entries, so every EVE-k cluster exposed Traefik on 80/443 of every node address. The disable+ fix had gone into a copy under cfg-manifests that kube-init never reads; that copy is removed. A unit test applies k3s's merge rule to the drop-ins kube-init renders and fails if either component is re-enabled.

Mixed-version clusters: a node that joins for the first time needs its bootstrap node to run the same side of the 12346 change. Nodes that restart, including after an upgrade, do not query /status.

The last commit adds a Network Exposure section to docs/EVE-K.md: the ports EVE-k admits on the cluster interface and how each is protected.

How to test and validate this PR

EVETEST_EVE_VERSION=<version> make evetest NAME=TestPortScanSuite
EVETEST_EVE_VERSION=<version> EVETEST_HYPERVISOR=kubevirt make evetest NAME=TestThreeNodesPortScan

Results:

  • TestPortScanSuite (EVE-kvm and single-node EVE-k): passes; 22/tcp is the only open port. The variants take 5.6 and 9.2 min. (Run on the image with the first two commits; the 12346 server and the local-storage drop-in exist only in cluster mode.)

  • TestThreeNodesPortScan (0.0.0-port-scan-c903eb10, all four commits): passes in 28 min, 14.5 of them for the cluster to form. 21 open ports:

    • 22 on every address, offering publickey only;
    • 6443 and 10250 on each cluster IP, TLS with 401 without credentials;
    • 2379 and 2380 on each cluster IP, TLS requiring a client certificate;
    • 12346 on each cluster IP, TLS 1.3 with 401 without the token.

    The App-Tracker on every node's loopback interface reports all three nodes.

  • Cluster upgrade: evetest: upgrade a three-node EVE-k cluster #6673's TestThreeNodesUpgrade, extended with an App-Tracker check, passes upgrading a three-node cluster from 0.0.0-master-3f136150 (this branch's base) to 0.0.0-port-scan-c903eb10, one node at a time. After the upgrade every node's App-Tracker reports all three nodes. The check will be added to that test once both PRs merge.

Against 17.0.0-lts the three-node test fails on SSH password methods, on plain-HTTP 12346, and on Traefik's 80/443 and NodePorts.

Unit tests cover the token and certificate derivation (pinned to the same values in zedkube and kube-init), 401 without or with a wrong token, rejection of a peer with another join token, and the k3s disable merge.

Open test issues

  • A first join during a mixed-version upgrade is not tested: it needs a fourth node. The code suggests a join can block when the joining node and its bootstrap node run opposite sides of the 12346 change.
  • An upgrade from 17.0.0-lts is not yet tested: in evetest: upgrade a three-node EVE-k cluster #6673's test the cluster does not form on that release.

Changelog notes

  • Cleanup the optional SSH server configuration to only offers public-key authenticationto facilitate security testing.
  • On EVE-k clusters, the cluster status service on port 12346 requires TLS and authentication. The App-Tracker is available only on the node itself, at 127.0.0.1:12346, for example through edgeview.sh tcp/127.0.0.1:12346.
  • EVE-k clusters no longer run k3s's servicelb and Traefik by default.

PR Backports

  • 17.0-stable: Yes, the EVE-k fixes; a separate PR, since 17.0-stable still has the shell kube scripts.
  • 16.0-stable: No, test and hardening change.
  • 14.5-stable: No, test and hardening change.
  • 13.4-stable: No, test and hardening change.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

Comment thread evetest/tests/portscan/portscan.go Dismissed
@codecov

codecov Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 51.90840% with 63 lines in your changes missing coverage. Please review.
✅ Project coverage is 27.33%. Comparing base (5aa1c1b) to head (ed34462).

Files with missing lines Patch % Lines
pkg/pillar/cmd/zedkube/clusterstatus.go 0.00% 45 Missing ⚠️
pkg/pillar/cmd/zedkube/clusterstatusauth.go 79.06% 10 Missing and 8 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6665      +/-   ##
==========================================
+ Coverage   26.99%   27.33%   +0.34%     
==========================================
  Files         523      534      +11     
  Lines       95745    97560    +1815     
==========================================
+ Hits        25844    26669     +825     
- Misses      67846    68649     +803     
- Partials     2055     2242     +187     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark eriknordmark changed the title evetest: scan device ports from the network side evetest: port scans; sshd offers only publickey login Sep 28, 2026
@eriknordmark
eriknordmark force-pushed the port-scan branch 2 times, most recently from 0e34d37 to 168ae89 Compare September 29, 2026 00:04
@eriknordmark eriknordmark changed the title evetest: port scans; sshd offers only publickey login evetest: port scans; fix the exposure they found on EVE and EVE-k Sep 29, 2026
@eriknordmark
eriknordmark marked this pull request as ready for review September 29, 2026 06:53
Add tests that run nmap from the evetest container against every TCP
and UDP port a device exposes: TestPortScanSuite covers EVE-kvm and a
single-node EVE-k, and TestThreeNodesPortScan in TestNodeClusterSuite
covers each node of a three-node EVE-k cluster on both its management
address and its cluster IP, reusing TestThreeNodesCluster's devices.

Any open UDP port fails. SSH must offer publickey authentication only,
which nmap's ssh-auth-methods checks without sending credentials. Every
other open TCP port must accept only TLS 1.2+ with ciphers that
ssl-enum-ciphers grades A, refuse plaintext HTTP, and reject a request
without credentials with 401 or by demanding a client certificate.

The harness image gains nmap and nmap-scripts, hence the VERSION bump.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eriknordmark eriknordmark added the stable Should be backported to stable release(s) label Sep 29, 2026
eriknordmark and others added 4 commits September 29, 2026 00:04
sshd in the debug container ran with Alpine's defaults, which offer
password and keyboard-interactive authentication to anyone who
connects. No account has a password, so none of those logins could
succeed, but that can only be established by inspecting the image, not
from the network. Turning both methods off leaves what is allowed
unchanged and makes it observable: the evetest port scans now require
that SSH offers publickey only.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On EVE-k, the cluster status server on port 12346 of the cluster IP
spoke plain HTTP and answered anyone on the cluster network, including
the App-Tracker's per-application dumps. It now accepts only TLS 1.3
and only requests carrying a bearer token. Every node derives the token
and the server's certificate from the cluster join token, so kube-init
on a joining node and zedkube on its peers authenticate the server and
themselves without any key distribution.

The cluster IP now serves only what other nodes need, /status and
/app/. The operator-facing App-Tracker, including /cluster-app/, moves
to plain HTTP on 127.0.0.1:12346, which edgeview reaches as before via
tcp/127.0.0.1:12346.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every EVE-k cluster ran k3s's servicelb and Traefik, exposing Traefik on
80/443 of every node address. kube-init writes the local-storage drop-in
with a bare "disable:" key, which replaces the base config's servicelb
and traefik entries instead of adding to them. The "disable+" fix went
into the copy under cfg-manifests, which kube-init never reads, so the
dead copy is removed.

A unit test renders the drop-ins kube-init writes beside the base
config, applies k3s's merge rule, and fails if servicelb or traefik is
no longer disabled, or if a shipped drop-in shadows a rendered one.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
List the ports EVE's firewall admits on the cluster interface and how
each is protected, note that the flannel VXLAN overlay is neither
encrypted nor authenticated, and explain why a bare disable: key in a
k3s drop-in re-exposes Traefik.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULraDivKMXrdxJxycz6g6h
@eriknordmark
eriknordmark merged commit 6da1ba3 into lf-edge:master Sep 29, 2026
49 of 50 checks passed
@eriknordmark
eriknordmark deleted the port-scan branch October 8, 2026 15:15
shjala added a commit to shjala/eve that referenced this pull request Nov 5, 2026
Build the verified upstream release to include the musl password authentication bypass fix before Alpine 3.22 packages it. Preserve publickey-only access and register the source-built version in the SBOM.

Add an isolated regression that reproduces the bypass on 10.6p1 and verifies rejection, publickey login and SFTP on 10.6p2.

Signed-off-by: Shahriyar Jalayeri <shahriyar@posteo.de>
(cherry picked from commit 8553375)

Backport notes:
Retain 16.0 base image pins and refresh its installer debug image hash.
Carry the unchanged master register-sbom-pkg.sh helper locally because
this branch lacks it. The preceding commit backports master 97f726a
(original PR lf-edge#6665 commit b1d10d9) to disable password authentication.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants