Skip to content

Don't spawn long-running daemons on an undrained stdout PIPE; tear down NAT rules on AP stop - #21

Open
cjolly wants to merge 1 commit into
SensorsIot:mainfrom
cjolly:fix/daemon-pipe-deadlock-nat-teardown
Open

cjolly wants to merge 1 commit into
SensorsIot:mainfrom
cjolly:fix/daemon-pipe-deadlock-nat-teardown

Conversation

@cjolly

@cjolly cjolly commented Aug 6, 2026 •

Copy link
Copy Markdown

Last one from this round of digging (same disclosure as #19/#20: Claude
helped track these down and write the fixes). Two independent issues found
in the same investigation — bundled here since they came from one review
pass, happy to split if you'd rather review them separately.

1. Undrained stdout=PIPE deadlocks daemons

hostapd, dnsmasq, mosquitto, and OpenOCD were all spawned with
stdout=subprocess.PIPE, and nothing ever read that pipe. Once ~64KB of log
output accumulated (a few hours of dnsmasq's log-dhcp chatter, in the
case that surfaced this), the daemon blocked mid-write() and silently
stopped doing its job — DHCP stopped answering, so AP clients associated at
the kernel level but never got a lease. From the outside this looked like a
hung DUT (STA connect failed or timed out), while ps showed hostapd and
dnsmasq both alive and well. It only shows up after hours of runtime, so
it's easy to miss in a quick manual test.

2. NAT rules from ap_start(internet=True) were never torn down

ap_stop() had no corresponding call to remove the MASQUERADE/FORWARD rules
_enable_nat() added, so one internet-bridged AP session left the bridge in
place for every AP started afterward — silently breaking the network
isolation a "plain" (non-internet) test AP is supposed to guarantee.

Fix

Long-running daemon spawns now log to a file in their work dir instead of an
unread pipe (can't block, and the log survives for post-mortem). A new
_disable_nat() is called both when ap_start(internet=False) runs after a
prior internet-bridged session, and unconditionally in ap_stop().

Note for reviewers: an earlier version of this fix also scoped
mqtt_controller's pkill -f mosquitto to avoid killing unrelated brokers
on the box — that fix has since landed independently on main (with a
_port_owner() conflict check added too), so this PR only carries the
still-missing PIPE fix for mosquitto's own stdout, not the pkill change.

Test plan

  • Run an AP session for several hours with log-dhcp enabled, confirm
    DHCP keeps answering (no wedge)
  • ap_start(internet=True) → ap_stop() → ap_start(internet=False),
    confirm the second AP's clients have no route to eth0/LAN
  • Confirm iptables -t nat -L / -L FORWARD show no leftover rules
    after ap_stop()
  • Start/stop an OpenOCD debug session, confirm /tmp/openocd.log (or
    $OPENOCD_LOG) captures output and the daemon doesn't wedge on a
    long GDB session

…on AP stop

Rebased against main's restructuring. hostapd/dnsmasq already got an
equivalent PIPE fix upstream independently, so this now only covers the
two daemons that didn't: OpenOCD (debug_controller.py) and mosquitto
(mqtt_controller.py), both still spawned with stdout=subprocess.PIPE
with nothing ever draining it, which blocks the daemon in write() once
~64KB of output accumulates.

Also still missing upstream: ap_stop() never tore down the NAT bridge
rules _enable_nat() adds for ap_start(internet=True), so one
internet-bridged AP session left the bridge in place for every AP
started afterward. _disable_nat() now runs in ap_stop() and in
ap_start(internet=False), and the MASQUERADE rule is scoped to the AP
subnet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@cjolly
cjolly force-pushed the fix/daemon-pipe-deadlock-nat-teardown branch from b469a97 to 66b2f13 Compare August 27, 2026 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant