Skip to content

[13.4-stable] Fix bugs in reporting of application IPs and flows - #6903

Merged
eriknordmark merged 4 commits into
lf-edge:13.4-stablefrom
milan-zededa:13.4-backport-6718-app-ips-flows
Oct 26, 2026
Merged

eriknordmark merged 4 commits into
lf-edge:13.4-stablefrom
milan-zededa:13.4-backport-6718-app-ips-flows

Conversation

@milan-zededa

Copy link
Copy Markdown
Contributor

Description

Backport of #6718 to 13.4-stable.

Partial backport, cherry-picked with -x from 13c2702286, c597e4d344, 1ddeeb12f7 and ef7d4022db. All four cherry-picks are clean, no conflicts. Not backported: the evetest commit of the PR and zedrouter: make expiration of ARP-learned IPs configurable (a separate feature, not a fix).

These fixes concern reporting of application IPs and flows for a Switch NI (the test of the PR, TestSwitchNIDeletedWithAppsAttached, is not part of the 13.4 evetest backport (#6899) yet).

PR dependencies

None.

How to test and validate this PR

Covered by TestSwitchNIDeletedWithAppsAttached from master (not in 13.4-stable yet, see above). To run it, copy evetest/tests/networking/switch_ni_delete_test.go and its testsuite entry from master onto the evetest backport (#6899):

EVETEST_EVE_VERSION=<13.4-stable build with this PR> make evetest NAME=TestSwitchNIDeletedWithAppsAttached

Changelog notes

Fixed a deadlock of zedrouter when stopping a packet capture, packet capture not being retried when the interface does not exist yet, stale VIFs in the network instance state collector after NI config deletion, and attribution of application flows in Switch network instances.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

milan-zededa and others added 4 commits October 22, 2026 10:36
When a network instance with a running packet capture (Switch NI, or Local
NI with flow logging) was deleted while the collector had an IP assignment
update pending, zedrouter could hang forever and be rebooted by the
watchdog. Three Go routines were waiting on each other:

 - zedrouter main loop: StopCollectingForNI waits for the PCAP Go routine
   (pcapWG.Wait) while holding the collector lock,
 - PCAP Go routine: blocked on a plain send to the capturedPackets channel
   (outside of the select that watches for cancellation) once the
   100-slot buffer was full,
 - collector event loop: the only reader of capturedPackets, blocked on
   an unbuffered watcher channel that only the zedrouter main loop reads.

Break the cycle at all three places:

 - PCAP sends of captured packets are now abandoned when the PCAP is
   cancelled.
 - The event loop no longer blocks on watcher channels. They are buffered
   and an update which does not fit is dropped (with a warning), so a slow
   watcher cannot stall the collector. Since IP assignment updates are deltas,
   a dropped one would leave the watcher out of sync until the same IP changes
   again. The collector therefore remembers it and, on the next garbage
   collection of IP assignments, sends the full state of IP assignments of all
   VIFs again (repeated until the watcher accepts it).
 - StopCollectingForNI and UpdateCollectingForNI no longer hold the
   collector lock while waiting for the PCAP to stop.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit 13c2702)
The packet capture used to learn IP assignments of app VIFs (and DNS
requests) was started only once. If the interface to capture from did not
exist at that moment, the capture Go routine logged an error and exited,
while the collector kept considering the capture as running and never
restarted it. This happens at boot, when the mirror interface of a Switch
NI is created by the NI reconciler only after the state collecting for the
NI was started. The result was a Switch NI with no IP addresses reported for
any of its VIFs until the next reboot.

Keep retrying to open the interface, with a delay growing from 1 to 30
seconds, until the capture is stopped. The same retry is applied if a running
capture gets closed. The error is logged only when it changes, to avoid
flooding the logs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit c597e4d)
… deleted

When the controller deleted a network instance together with the apps using
it (e.g. during a redeploy), the NI itself is kept by zedrouter until the last
application interface is removed. The NI config is gone by then, however,
and updateVIFsForStateCollecting skipped the update of the NI state
collector with "failed to get config for network instance ..." for every
app that was being removed.

The collector therefore kept watching VIFs of already deleted apps for
the whole app teardown. It kept reporting IP assignment changes for them
(when their addresses expired) and reading ACL counters of deleted iptables
chains, producing errors and unnecessary events, and it provided the
IP assignment update which triggered the deadlock fixed earlier.

Use the NetworkInstanceStatus, which carries the last applied config and
exists for as long as the NI does, instead of looking up the config.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit 1ddeeb1)
With flow logging enabled, zedrouter marks connections of every app with the
ID of the app and of the matching ACL rule, so that flow records can be
attributed to the right app. For a Switch NI, the rules marking traffic
"coming into" a VIF were applied in the mangle PREROUTING chain, where
the destination VIF of bridged traffic is not known yet. The rules were
matched only by the bridge as the input interface and by the ACL, and
a chain of every VIF is traversed for every packet. The ingress ACL of the
VIF whose chain is traversed first (the one created last) therefore marked
virtually every connection of the NI as belonging to its app, before the
egress rules of the app that really originated the traffic had a chance to
do the marking.

The result were flow records (and ACL IDs) attributed to a wrong app. The state
collector could not match the IP addresses in such flows with the VIFs of the
app and dropped them with a "Flow entry without app IP address" warning,
logged for most of the flows seen in the NI.

Apply the ingress marking rules of Switch NI in the mangle FORWARD chain,
where the output port of the bridge is known and can be matched using
--physdev-out, in the same way as the filter table ingress ACLs are. Traffic
originated by apps remains marked by egress rules in PREROUTING. Local NI is not
affected, it only marks port-mapped traffic in PREROUTING, which is matched by
the destination IP address.

FORWARD is traversed after the device-wide ACLs of PREROUTING-device, which
mark ingress traffic with the mark of a control protocol (SSH, ICMP, DHCP, ...)
or, if not marked yet, with the default DROP mark. The marking chains of apps
therefore have to override marks that do not carry an application ID, otherwise
ingress traffic of apps would never get marked by the ACLs of the apps (and
would be dropped by the default DROP mark).

The mangle FORWARD chain also contains the TCP MSS clamping rule. The marking
chains finish with ACCEPT, which would end the traversal of the hook and skip
the clamping for ingress traffic of Switch NI. Therefore, separate marking
chains ending with RETURN are used for the ingress marking of Switch NI.

Also match VIFs by their exact name in all physdev matches. A "+" wildcard
was appended to the name for the sake of Xen, which is no longer supported
(qemu used to create a second "<vif>-emu" interface). With KVM and with the
EVE CNI of eve-k, the name of the VIF is exact, and the wildcard only made
a rule of a VIF match traffic of other VIFs whose name starts with the same
string (e.g. nbu1x4 and nbu1x40). It also prevented the per-VIF ACL drop
counters of the state collector from being matched, since it compares the
VIF name exactly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit ef7d402)
@codecov

codecov Bot commented Oct 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 32.66%. Comparing base (8077a3d) to head (3887980).
⚠️ Report is 319 commits behind head on 13.4-stable.

Additional details and impacted files
@@               Coverage Diff               @@
##           13.4-stable    #6903      +/-   ##
===============================================
+ Coverage        24.78%   32.66%   +7.88%     
===============================================
  Files                8        9       +1     
  Lines             1138     1090      -48     
===============================================
+ Hits               282      356      +74     
+ Misses             788      654     -134     
- Partials            68       80      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark
eriknordmark merged commit c857cc3 into lf-edge:13.4-stable Oct 26, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants