Repository navigation
[14.5-stable] Fix bugs in reporting of application IPs and flows - #6877
Merged
rene merged 4 commits intoOct 26, 2026
Merged
Conversation
milan-zededa
requested review from
christoph-zededa and
rouming
as code owners
October 20, 2026 07:31
4 of 7 tasks
milan-zededa
force-pushed
the
14.5-backport-6718-app-ips-flows
branch
2 times, most recently
from
October 20, 2026 07:53
f1498e7 to
4d7fd46
Compare
When a network instance with a running packet capture (Switch NI, or Local NI with flow logging) was deleted while the collector had an IP assignment update pending, zedrouter could hang forever and be rebooted by the watchdog. Three Go routines were waiting on each other: - zedrouter main loop: StopCollectingForNI waits for the PCAP Go routine (pcapWG.Wait) while holding the collector lock, - PCAP Go routine: blocked on a plain send to the capturedPackets channel (outside of the select that watches for cancellation) once the 100-slot buffer was full, - collector event loop: the only reader of capturedPackets, blocked on an unbuffered watcher channel that only the zedrouter main loop reads. Break the cycle at all three places: - PCAP sends of captured packets are now abandoned when the PCAP is cancelled. - The event loop no longer blocks on watcher channels. They are buffered and an update which does not fit is dropped (with a warning), so a slow watcher cannot stall the collector. Since IP assignment updates are deltas, a dropped one would leave the watcher out of sync until the same IP changes again. The collector therefore remembers it and, on the next garbage collection of IP assignments, sends the full state of IP assignments of all VIFs again (repeated until the watcher accepts it). - StopCollectingForNI and UpdateCollectingForNI no longer hold the collector lock while waiting for the PCAP to stop. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 13c2702)
The packet capture used to learn IP assignments of app VIFs (and DNS requests) was started only once. If the interface to capture from did not exist at that moment, the capture Go routine logged an error and exited, while the collector kept considering the capture as running and never restarted it. This happens at boot, when the mirror interface of a Switch NI is created by the NI reconciler only after the state collecting for the NI was started. The result was a Switch NI with no IP addresses reported for any of its VIFs until the next reboot. Keep retrying to open the interface, with a delay growing from 1 to 30 seconds, until the capture is stopped. The same retry is applied if a running capture gets closed. The error is logged only when it changes, to avoid flooding the logs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit c597e4d)
… deleted When the controller deleted a network instance together with the apps using it (e.g. during a redeploy), the NI itself is kept by zedrouter until the last application interface is removed. The NI config is gone by then, however, and updateVIFsForStateCollecting skipped the update of the NI state collector with "failed to get config for network instance ..." for every app that was being removed. The collector therefore kept watching VIFs of already deleted apps for the whole app teardown. It kept reporting IP assignment changes for them (when their addresses expired) and reading ACL counters of deleted iptables chains, producing errors and unnecessary events, and it provided the IP assignment update which triggered the deadlock fixed earlier. Use the NetworkInstanceStatus, which carries the last applied config and exists for as long as the NI does, instead of looking up the config. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 1ddeeb1)
With flow logging enabled, zedrouter marks connections of every app with the ID of the app and of the matching ACL rule, so that flow records can be attributed to the right app. For a Switch NI, the rules marking traffic "coming into" a VIF were applied in the mangle PREROUTING chain, where the destination VIF of bridged traffic is not known yet. The rules were matched only by the bridge as the input interface and by the ACL, and a chain of every VIF is traversed for every packet. The ingress ACL of the VIF whose chain is traversed first (the one created last) therefore marked virtually every connection of the NI as belonging to its app, before the egress rules of the app that really originated the traffic had a chance to do the marking. The result were flow records (and ACL IDs) attributed to a wrong app. The state collector could not match the IP addresses in such flows with the VIFs of the app and dropped them with a "Flow entry without app IP address" warning, logged for most of the flows seen in the NI. Apply the ingress marking rules of Switch NI in the mangle FORWARD chain, where the output port of the bridge is known and can be matched using --physdev-out, in the same way as the filter table ingress ACLs are. Traffic originated by apps remains marked by egress rules in PREROUTING. Local NI is not affected, it only marks port-mapped traffic in PREROUTING, which is matched by the destination IP address. FORWARD is traversed after the device-wide ACLs of PREROUTING-device, which mark ingress traffic with the mark of a control protocol (SSH, ICMP, DHCP, ...) or, if not marked yet, with the default DROP mark. The marking chains of apps therefore have to override marks that do not carry an application ID, otherwise ingress traffic of apps would never get marked by the ACLs of the apps (and would be dropped by the default DROP mark). The mangle FORWARD chain also contains the TCP MSS clamping rule. The marking chains finish with ACCEPT, which would end the traversal of the hook and skip the clamping for ingress traffic of Switch NI. Therefore, separate marking chains ending with RETURN are used for the ingress marking of Switch NI. Also match VIFs by their exact name in all physdev matches. A "+" wildcard was appended to the name for the sake of Xen, which is no longer supported (qemu used to create a second "<vif>-emu" interface). With KVM the name of the VIF is exact, and the wildcard only made a rule of a VIF match traffic of other VIFs whose name starts with the same string (e.g. nbu1x4 and nbu1x40). It also prevented the per-VIF ACL drop counters of the state collector from being matched, since it compares the VIF name exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit ef7d402)
milan-zededa
force-pushed
the
14.5-backport-6718-app-ips-flows
branch
from
October 22, 2026 14:16
4d7fd46 to
d472702
Compare
rene
approved these changes
Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6718 to
14.5-stable.Cherry-picked with
-xfrom13c2702286,c597e4d344,1ddeeb12f7andef7d4022db. All four cherry-picks are clean, no conflicts. Not backported: the evetest commit of the PR andzedrouter: make expiration of ARP-learned IPs configurable(a separate feature, not a fix).Without these fixes
TestSwitchNIDeletedWithAppsAttachedfails (reporting of application IPs and flows for a Switch NI). That test is not part of the 14.5 evetest backport (#6873) yet, because it needs this PR; it can be added once this is merged.PR dependencies
None.
How to test and validate this PR
Covered by
TestSwitchNIDeletedWithAppsAttachedfrom master (not in14.5-stableyet, see above). To run it, copyevetest/tests/networking/switch_ni_delete_test.goand its testsuite entry from master onto the evetest backport (#6873):Changelog notes
Fixed a deadlock of zedrouter when stopping a packet capture, packet capture not being retried when the interface does not exist yet, stale VIFs in the network instance state collector after NI config deletion, and attribution of application flows in Switch network instances.
Checklist
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.