Skip to content

Harden 802.1X (PNAC) re-authentication and removal handling - #6929

Merged
eriknordmark merged 7 commits into
lf-edge:masterfrom
milan-zededa:pnac-hardening
Oct 30, 2026
Merged

eriknordmark merged 7 commits into
lf-edge:masterfrom
milan-zededa:pnac-hardening

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Description

Hardening of the 802.1X (PNAC) support in nim, fixing problems found during
EVE 17.0.1 RC testing, where repeated re-authentications and trusted CA changes
could leave a port with a wrong IP address or a stale authentication status.

Changes in nim:

  • Detect 802.1X re-authentication through the wpa_supplicant control socket
    (EAP-SUCCESS/EAP-FAILURE events) instead of the wpa_cli -a action script.
    The script is only invoked for CONNECTED/DISCONNECTED, which are not emitted
    when a wired port re-authenticates, so the reported PNAC state could be stale
    and the DHCP lease was not re-acquired after the port moved between VLANs.
  • Send EAPOL-Logoff before stopping the supplicant, so that the authenticator
    immediately unauthorizes the port instead of keeping the session of a
    supplicant that no longer exists.
  • Re-acquire the DHCP lease when PNAC is removed from a port, using the existing
    DHCP reacquire mechanism. DPC verification now waits for the first reacquire
    to finish, so that the connectivity test is not run with the stale IP address
    of the authenticated network (which previously caused a rollback to the
    previous DPC).

Changes in evetest:

  • TestPNAC gets stress cycles of repeated re-authentications, trusted CA
    changes and SCEP profile replacements, verifying after each cycle that the
    reported status is updated and that the port gets the IP from the
    correct (authenticated or non-authenticated) VLAN. The number of cycles is set
    by EVETEST_PNAC_REAUTH_CYCLE_COUNT (default 3), which can be increased to
    look for resource growth. Memory usage is logged after each cycle, and an
    unexpected reboot fails the test.
  • The SDN restarts 802.1X authentication of stations when the authenticator
    configuration changes (new_sta), restarts hostapd when the CA or users change
    (hostapd does not reload TLS certificates), and reaps child processes
    (zombie hostapd previously made the SDN fail to stop it).
  • evetest/CLAUDE.md now describes how an AI agent can use the EVE console.

How to test and validate this PR

Run the automated test, optionally with more cycles:

EVETEST_PNAC_REAUTH_CYCLE_COUNT=30 make evetest NAME=TestPNAC

Pillar unit tests of the changed packages:

make -C pkg/pillar test TESTRUN=TestPNAC TESTPKGS=./dpcmanager/
make -C pkg/pillar test TESTPKGS=./dpcreconciler/genericitems/

Manual check: on a device with PNAC enabled on an uplink, change the trusted
CA or force the authenticator to re-authenticate the port, and verify that the
reported PNAC state changes and the port gets an IP from the expected network.
Then remove PNAC from the port and verify that the port acquires an address from
the non-authenticated network and that the authenticator sees EAPOL-Logoff.

Changelog notes

Improved reliability of 802.1X (port-based network access control): the port
status is updated on every re-authentication, the DHCP lease is re-acquired
after the port moves between authenticated and non-authenticated networks,
and the switch is notified when 802.1X is turned off on a port.

PR Backports

  • 17.0-stable: To be backported.
  • 16.0-stable: No, as PNAC is not supported there.
  • 14.5-stable: No, as PNAC is not supported there.
  • 13.4-stable: No, as PNAC is not supported there.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't check them.

@milan-zededa milan-zededa added the stable Should be backported to stable release(s) label Oct 26, 2026
@github-actions
github-actions Bot requested a review from europaul October 26, 2026 08:51
@milan-zededa milan-zededa added bug Something isn't working next-17.0.x-rc PR must be present in the next 17.0.x-lts release labels Oct 26, 2026
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@claude

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @milan-zededa's task in 6m 5s —— View job


Review complete — PR #6929: Harden 802.1X (PNAC) re-authentication and removal handling

  • Read pillar changes (wpa_ctrl.go, wpa_supplicant.go, dpcmanager, verify.go)
  • Read evetest changes (hostapd.go, net_adapter_test.go, helpers)
  • Analyze for bugs, concurrency, resource leaks, best practices
  • Post inline review comments

Summary

Solid, well-tested hardening change. I traced the two production-side state machines in detail and found no correctness bugs — just two low-severity robustness observations, posted inline:

  1. pnacStateTracker.stop() doesn't fence an in-flight flush() (wpa_ctrl.go). Correctness currently depends on the multi-second Delete path running before the state file is removed; a fragile timing invariant. Suggested a stopped flag under the mutex.
  2. ATTACH phase is not cancellable (wpa_ctrl.go): the ctx-watching goroutine starts after wpaCtrlAttach, so a cancellation during attach is deferred up to the 5s reply timeout. Bounded by the 10s stop timeout — slower shutdown, not a leak.

Things I verified as correct (no action needed):

  • dpcmanager/verify.go DHCP-reacquire gating: the new isDHCPReacquirePending() wait is bounded — runVerify re-arms pendingDpcTimer, and verifyDPC falls through after waitForAsyncRetries*dpcTestDuration with a warning, so verification cannot stall indefinitely even if no AddrChange/lease ever arrives. The pnacEnabled != port.PNAC.Enabled guard correctly distinguishes a DPC switch from a legitimate PNAC-removal reacquire.
  • wpaEventListener: keepalive PING / lost-connection detection, PONG filtering (non-< prefix ignored), ctx-cancel → DETACH+close with sync.Once, and the ctx.Err() check before error handling are all correct; no goroutine leak.
  • Create/Delete ordering: reconciler.ContinueInBackground/done() serializes Create before any Delete, so pms.stopWatcher is always set before Delete can run (no watcher-goroutine leak).
  • EAPOL-Logoff ordering in Delete (stop watcher → logoff while supplicant alive → stop supplicant) is correct.
  • evetest: zombie reaping (go cmd.Wait()), new_sta re-auth, and credential-change restart logic look correct for the test harness.

No blocking issues from my side.
· branch pnac-hardening

@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 58.82353% with 84 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.04%. Comparing base (a273c3e) to head (6b87882).
⚠️ Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
...illar/dpcreconciler/genericitems/wpa_supplicant.go 0.00% 51 Missing ⚠️
pkg/pillar/dpcreconciler/genericitems/wpa_ctrl.go 82.35% 14 Missing and 7 partials ⚠️
pkg/pillar/dpcmanager/verify.go 0.00% 7 Missing and 1 partial ⚠️
pkg/pillar/dpcmanager/dpcmanager.go 84.61% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6929      +/-   ##
==========================================
+ Coverage   28.47%   29.04%   +0.57%     
==========================================
  Files         540      552      +12     
  Lines       98258    99985    +1727     
==========================================
+ Hits        27975    29045    +1070     
- Misses      68009    68707     +698     
+ Partials     2274     2233      -41     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kicking off Eden tests.....

@lf-edge lf-edge deleted a comment from github-actions Bot Oct 29, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Oct 29, 2026
milan-zededa and others added 7 commits October 29, 2026 19:17
Extend TestPNAC with a configurable number of cycles
(EVETEST_PNAC_REAUTH_CYCLE_COUNT, default 3) that exercise 802.1X
more aggressively than a single enrollment and authentication:

- re-initialize the port on the authenticator twice and verify that
  LastAuthTimestamp advances each time and the port keeps an IP from
  the authenticated VLAN
- make the authenticator trust only an unrelated CA and verify that
  the failure is reported and the port gets a pre-auth VLAN IP
- restore the CA and verify that EVE recovers on its own
- disable PNAC and remove the SCEP profile, verify the certificate is
  removed and the port falls back to the pre-auth VLAN, then enroll a
  new certificate with a different key type and hash
- log EVE memory usage after every cycle and fail if it approaches
  the EVE memory limit, so a higher cycle count can expose growth

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Bumping Hostapd.ReauthGeneration only deauthenticated the stations.
For a wired supplicant that sends no frame at all, so the supplicant
kept considering its port authorized and never re-authenticated,
unlike with a real switch that re-initializes the port and sends a
new EAP-Request/Identity.

After deauthenticating, wait for hostapd to remove each station and
add it back with "hostapd_cli new_sta", which makes hostapd start a
new EAP exchange.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
…ol socket

nim learned about the 802.1X state from "wpa_cli -a", which only calls
its action script for CONNECTED and DISCONNECTED. When a wired port
re-authenticates (e.g. after the authenticator re-initializes it),
wpa_supplicant stays in the COMPLETED state and emits neither event.
As a result the PNAC state file was not updated, the reported
LastAuthTimestamp stayed stale and, if the switch had moved the port to
a different VLAN, DHCP was never reacquired. The same happened when
the authenticator rejected the certificate after such a re-init.

Attach to the control socket of wpa_supplicant directly and update the
PNAC state file also on CTRL-EVENT-EAP-SUCCESS and
CTRL-EVENT-EAP-FAILURE. Events are coalesced for a short moment so that
the supplicant state machines have settled before the network monitor
reads the status. The wpa_cli watcher process and its generated script
are no longer needed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
When PNAC is disabled or reconfigured, nim stops wpa_supplicant by
terminating the process. That does not make the supplicant send an
EAPOL-Logoff frame, so the authenticator keeps treating the port as
authorized and does not move it back to the pre-authentication VLAN.

Ask wpa_supplicant to log off through its control socket before it is
stopped. The event listener is stopped first, so the logoff is not
reported as a change of the authentication state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
When PNAC is removed from a port, the supplicant logs off and the switch
may move the port to a different VLAN, so the lease has to be reacquired
just like after a change of the authentication state. The scheduled
reacquire used to be cancelled whenever the current DPC had PNAC disabled.
Cancel it only if the PNAC configuration of the port differs from the one
at the time of scheduling, or the port has no DHCP client.

Also make DPC verification wait for the first DHCP reacquire of a port
to finish (bounded the same way as for async operations). Otherwise the
connectivity test of a new DPC can run with a lease obtained in the
previous VLAN, fail, and make nim fall back to the previous DPC.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The console was described as unusable by AI agents. It can be driven by
piping short commands into it, and it is the only way in when ssh does
not work because of networking problems, which is when it is most useful.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
After a PNAC state change, the DHCP client may obtain a lease from the new
VLAN by itself before the first scheduled reacquire runs. Releasing that
lease again caused a short loss of the IP address and of the DNS servers
right after the port was reported as authenticated, which made connections
started at that moment fail.

Cancel the reacquire tracker instead of releasing the lease when the subnet
of the port already differs from the one recorded when the PNAC event was
received.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@lf-edge lf-edge deleted a comment from github-actions Bot Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

evetest: skip decision

This PR only hardens 802.1X/PNAC re-authentication and removal handling (nim's wpa_supplicant/wpa_ctrl event handling, dpcmanager's PNAC-gated DHCP reacquire tracker and DPC-verify wait, and the evetest SDN hostapd/TestPNAC harness). TestPNAC (including its TestPNACWithoutProxy / TestPNACWithProxy variants) is kept as the direct, already-enhanced coverage for this change.

Skipping the following as unrelated (whole suites, none touch PNAC/802.1X/wired-port auth or the DHCP-reacquire mechanism):

  • TestApplicationConnectivitySuite
  • TestAppsSuite
  • TestBootstrapSuite
  • TestControllerFaultsSuite
  • TestDatastoreSuite
  • TestDiagSuite
  • TestHardwareSuite
  • TestKonvertSuite
  • TestKubeRegistrationSuite
  • TestLPSSuite
  • TestNodeClusterResilienceSuite
  • TestNodeClusterSuite
  • TestPcibackErrorSuite
  • TestPortScanSuite
  • TestSecuritySuite
  • TestStorageSuite
  • TestTelemetrySuite
  • TestUpgradeSuite
  • TestKvmToKRepartitionAppVolume
  • TestKvmToKRepartitionVolmig
  • TestPVCUploadReport
  • TestVaultKeyKeptOnTPMError

Skipping the following individual tests in TestDeviceConnectivitySuite as unrelated (none of them configure PNAC on a port, so the PNAC-gated DHCP-reacquire wait added to DPC verification and the wpa_supplicant control-socket/EAPOL-Logoff changes — both only active when a port's PNAC config is non-nil — are inert for them):

  • TestDHCPIPv4Only
  • TestStaticIPv4Only
  • TestDNSFunctionality
  • TestPortFailover
  • TestNetworkConfigFallback
  • TestIntermittentConnectivity
  • TestMgmtTrafficRoutedViaApp
  • TestDeviceIPv6Connectivity
  • TestDeviceNTPConfig
  • TestActiveBackupBond
  • TestLACPBond
  • TestVLANSubinterfaces
  • TestVLANSubinterfacesOnTopOfLAGs
  • TestCellularConnectivity
  • TestWifiConnectivity

An engineer can comment /evetest all to force the full suite to run instead, or /evetest <name> <name> ... to run only specific tests/suites.

@lf-edge lf-edge deleted a comment from github-actions Bot Oct 29, 2026
@eriknordmark
eriknordmark merged commit 88aefcf into lf-edge:master Oct 30, 2026
76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working next-17.0.x-rc PR must be present in the next 17.0.x-lts release stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants