Repository navigation
[16.0-stable] Fix bugs in reporting of application IPs and flows - #6844
Merged
rene merged 5 commits intoOct 19, 2026
Merged
Conversation
When a network instance with a running packet capture (Switch NI, or Local NI with flow logging) was deleted while the collector had an IP assignment update pending, zedrouter could hang forever and be rebooted by the watchdog. Three Go routines were waiting on each other: - zedrouter main loop: StopCollectingForNI waits for the PCAP Go routine (pcapWG.Wait) while holding the collector lock, - PCAP Go routine: blocked on a plain send to the capturedPackets channel (outside of the select that watches for cancellation) once the 100-slot buffer was full, - collector event loop: the only reader of capturedPackets, blocked on an unbuffered watcher channel that only the zedrouter main loop reads. Break the cycle at all three places: - PCAP sends of captured packets are now abandoned when the PCAP is cancelled. - The event loop no longer blocks on watcher channels. They are buffered and an update which does not fit is dropped (with a warning), so a slow watcher cannot stall the collector. Since IP assignment updates are deltas, a dropped one would leave the watcher out of sync until the same IP changes again. The collector therefore remembers it and, on the next garbage collection of IP assignments, sends the full state of IP assignments of all VIFs again (repeated until the watcher accepts it). - StopCollectingForNI and UpdateCollectingForNI no longer hold the collector lock while waiting for the PCAP to stop. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit ce0af7d)
The packet capture used to learn IP assignments of app VIFs (and DNS requests) was started only once. If the interface to capture from did not exist at that moment, the capture Go routine logged an error and exited, while the collector kept considering the capture as running and never restarted it. This happens at boot, when the mirror interface of a Switch NI is created by the NI reconciler only after the state collecting for the NI was started. The result was a Switch NI with no IP addresses reported for any of its VIFs until the next reboot. Keep retrying to open the interface, with a delay growing from 1 to 30 seconds, until the capture is stopped. The same retry is applied if a running capture gets closed. The error is logged only when it changes, to avoid flooding the logs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 93b1330)
… deleted When the controller deleted a network instance together with the apps using it (e.g. during a redeploy), the NI itself is kept by zedrouter until the last application interface is removed. The NI config is gone by then, however, and updateVIFsForStateCollecting skipped the update of the NI state collector with "failed to get config for network instance ..." for every app that was being removed. The collector therefore kept watching VIFs of already deleted apps for the whole app teardown. It kept reporting IP assignment changes for them (when their addresses expired) and reading ACL counters of deleted iptables chains, producing errors and unnecessary events, and it provided the IP assignment update which triggered the deadlock fixed earlier. Use the NetworkInstanceStatus, which carries the last applied config and exists for as long as the NI does, instead of looking up the config. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 3c9bb05)
An IP address of an app VIF learned from ARP snooping (statically configured IP, as opposed to those assigned by DHCP) was reported as assigned only for 10 minutes after the last ARP packet carrying it was seen. Apps that send ARP less often than that, which is common for addresses which are not used for much traffic (e.g. a second address of the VIF, also because the ARP cache entries of the peers are refreshed rarely), had the address repeatedly dropped from and re-added to the reported IP assignments, and every such change was published as an update of the NI and app network status. Add config property network.switch.arpsnoop.expiry (in seconds, default 600 as before) to change the expiration time. Note that EVE only stops reporting the expired address, it is not removed from the application. The garbage collection of expired IP assignments is run more often if a short expiration time is configured, so that the expiration is not delayed by up to a minute. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 2f129b6)
With flow logging enabled, zedrouter marks connections of every app with the ID of the app and of the matching ACL rule, so that flow records can be attributed to the right app. For a Switch NI, the rules marking traffic "coming into" a VIF were applied in the mangle PREROUTING chain, where the destination VIF of bridged traffic is not known yet. The rules were matched only by the bridge as the input interface and by the ACL, and a chain of every VIF is traversed for every packet. The ingress ACL of the VIF whose chain is traversed first (the one created last) therefore marked virtually every connection of the NI as belonging to its app, before the egress rules of the app that really originated the traffic had a chance to do the marking. The result were flow records (and ACL IDs) attributed to a wrong app. The state collector could not match the IP addresses in such flows with the VIFs of the app and dropped them with a "Flow entry without app IP address" warning, logged for most of the flows seen in the NI. Apply the ingress marking rules of Switch NI in the mangle FORWARD chain, where the output port of the bridge is known and can be matched using --physdev-out, in the same way as the filter table ingress ACLs are. Traffic originated by apps remains marked by egress rules in PREROUTING. Local NI is not affected, it only marks port-mapped traffic in PREROUTING, which is matched by the destination IP address. FORWARD is traversed after the device-wide ACLs of PREROUTING-device, which mark ingress traffic with the mark of a control protocol (SSH, ICMP, DHCP, ...) or, if not marked yet, with the default DROP mark. The marking chains of apps therefore have to override marks that do not carry an application ID, otherwise ingress traffic of apps would never get marked by the ACLs of the apps (and would be dropped by the default DROP mark). The mangle FORWARD chain also contains the TCP MSS clamping rule. The marking chains finish with ACCEPT, which would end the traversal of the hook and skip the clamping for ingress traffic of Switch NI. Therefore, separate marking chains ending with RETURN are used for the ingress marking of Switch NI. Also match VIFs by their exact name in all physdev matches. A "+" wildcard was appended to the name for the sake of Xen, which is no longer supported (qemu used to create a second "<vif>-emu" interface). With KVM and with the EVE CNI of eve-k, the name of the VIF is exact, and the wildcard only made a rule of a VIF match traffic of other VIFs whose name starts with the same string (e.g. nbu1x4 and nbu1x40). It also prevented the per-VIF ACL drop counters of the state collector from being matched, since it compares the VIF name exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 61a3b01)
milan-zededa
requested review from
christoph-zededa,
eriknordmark and
rouming
as code owners
October 18, 2026 17:41
4 of 7 tasks
rene
approved these changes
Oct 18, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 16.0-stable #6844 +/- ##
===============================================
- Coverage 29.12% 26.89% -2.24%
===============================================
Files 10 25 +15
Lines 1133 4332 +3199
===============================================
+ Hits 330 1165 +835
- Misses 768 2938 +2170
- Partials 35 229 +194 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6718 to
16.0-stable.The five pillar commits of the PR are cherry-picked with
-x(from the17.0-stableversions, which are the closest to 16.0); all of them appliedcleanly, without conflicts. The evetest commit of the PR is part of the
evetest backport (#6842).
The commits fix a zedrouter deadlock when a packet capture is stopped, retry
packet capture when it cannot be started, keep the NI state collector VIFs up
to date after the NI config is deleted, make the expiration of ARP-learned IPs
configurable, and fix the marking of app flows in Switch NI.
Without them
TestSwitchNIDeletedWithAppsAttachedfails.PR dependencies
None.
How to test and validate this PR
Covered by the evetest test
TestSwitchNIDeletedWithAppsAttached(added to16.0-stableby #6842):Changelog notes
Fixed several problems in the reporting of application IP addresses and network flows: a zedrouter deadlock when stopping a packet capture, packet capture that was not retried after a failure to start, stale VIF information after a network instance is deleted, and the marking of app flows in Switch network instances. The expiration of IP addresses learned through ARP is now configurable.
Checklist
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.