Repository navigation
pubsub: marshal and compress once for all subscribers - #7037
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #7037 +/- ##
==========================================
+ Coverage 29.06% 29.59% +0.52%
==========================================
Files 548 559 +11
Lines 98971 100628 +1657
==========================================
+ Hits 28763 29777 +1014
- Misses 67941 68624 +683
+ Partials 2267 2227 -40 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Claude finished @rene's task in 4m 18s —— View job Review of PR #7037 — pubsub: marshal and compress once for all subscribers
I reviewed the non-vendor pubsub changes (
One minor, non-blocking note posted inline on Vendor copies were not reviewed per instructions. |
| sub := ctxArg.(*SubscriptionImpl) | ||
| name := sub.nameString() | ||
| sub.log.Tracef("pubsub.handleModify(%s) key %s\n", name, key) | ||
| sum := sha256.Sum256(itemcb) |
There was a problem hiding this comment.
Minor (non-blocking): sha256.Sum256(itemcb) now runs on every handleModify, including the genuinely-changed case, where you still fall through to parseTemplate + cmp.Equal. So for a large object that actually changes, the hash is pure added work on top of the decode it was meant to avoid. The win only materializes on the unchanged/reconnect/populate paths. Given the measured net improvements this is clearly an acceptable trade-off — just noting the changed-path cost for the record.
The delete path correctly clears sums[key], so a delete+re-add with identical JSON still fires CreateHandler — good.
There was a problem hiding this comment.
Agreed, and it was more than noise: on my dev host SHA-256 runs at about 300 MB/s, which is 15% of one decode of an AppInstanceStatus and 45% of one decode of a 3.4 MB DevicePortConfigList, paid by every real change.
So the commit now hashes only where it pays: a persistent subscription, until it is synchronized. That covers loading the checkpoint and the publisher's resend at start. At sync the sums are dropped. After that, the publisher sends only changed JSON, which is decoded and compared exactly as before, with nothing hashed or kept. A reconnect's resend is decoded and compared as on master.
The start-up gain is unchanged. Starting a persistent subscription to 50 AppInstanceStatus takes 11,016 allocations of 1.9 MB (27,670 of 3.7 MB before), and to a 3.4 MB DevicePortConfigList 2,450 of 14.0 MB (20,080 of 18.7 MB). TestIdenticalNotDecoded now covers both phases.
TestControllerFaultsSuite: failure analysisTestDeferredQueueDropsReportedFailureClaude's conclusionRoot causeLive inspection shows the app ( This is a new failure, not in Regarding PR #7037: its actual diff is a pubsub internals rewrite ( |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
TestNodeClusterResilienceSuite: failure analysisTestVMAppPurgeDuringFailoverFailureClaude's conclusionRoot causeLive evidence confirms the exact mechanism already recorded in Per Regarding PR #7037: its code changes are confined to |
The counters of pubsub's own allocations came in with the last commit of lf-edge#5644, which was meant to be dropped before merge. They stay, so make them exact and keep them so. PublicationStats and SubscriptionStats passed a new PubStats or SubStats to LoadOrStore on every call, which allocated it each time: four allocations per published message spent on counting. Look the name up first; a new one is stored only the first time. Tests now hold the counters against the runtime's own count of allocations, and they found sites counted wrongly: []byte(key) passed to AppendEncode or EncodeToString is not copied, a buffer kept as a *[]byte moves its slice header to the heap, and the string of the persistence flag in hello escapes. TestAllocCountsSend and TestAllocCountsReceive compare the counters with testing.AllocsPerRun, and TestAllocSizeMatchesRuntime compares the size classes with MemStats.TotalAlloc, so a Go upgrade that changes what is allocated fails a test instead of making the counters drift. Both pass on Go 1.25 and 1.26. Under the race detector, which CI runs the tests with, the compiler does not optimise append(s, make([]T, n)...), so base64 decoding of a key above 32 bytes allocates twice: the receive test is skipped there, the send test runs. The pubsub-large counters went with pubsub-large; drop them. DeepCopy does not know the topic of what it copies, so report its counters in a "process" section of /debug/pubsub/stats instead of as a publication named "(DeepCopy)". Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A subscriber drops the connection on a frame above its frame size, or on one which does not decompress within its message size, and logs it. Count those in rejected_frames of /debug/pubsub/stats too, next to the values a publisher refuses, so that a mismatch of the size limits between two processes shows in the counters. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Every subscriber connection of a publication kept its own copy of what it had been sent, and on every change of any key compared the whole publication against it: GetAll deep-copied every item, a marshal and an unmarshal each, then every copy was marshalled again to compare it with the bytes sent before, through a linear search of the connection's map for each key. Sending an update copied the value once more into the message. A change of one key of a publication of N keys with S subscribers cost 2*N*S marshals and N*S unmarshals, and each connection held a copy of the JSON of every key. Publish now marshals the item once and keeps the JSON next to it, in a Value which is never modified. DetermineDiffs hands every connection the current Value of each key, and a key is sent again only if its Value is not the one the connection was sent last: nothing is marshalled or compared, and all connections share the same bytes. An update is written as its header followed by the shared JSON, in one writev, without copying the value. An item which marshals to the JSON already published is unchanged, and is neither copied nor compared: Publish used to deep-copy every item and compare the copy with cmp.Equal, which walks byte slices element by element through reflection and computes an edit script for every slice. Other JSON is decoded into the copy Publish keeps and compared with cmp.Equal as before, so an item which the Equal methods of its type find equal to the one published, e.g. a set in another order, is still unchanged. Get and GetAll decode their copies from the JSON: an unmarshal, without the marshal of a deep copy. A subscription's handler gets its copy decoded from the message received, and only if the handler is set. A checkpoint is sent as this version of its type marshals it, as before. Measured in a test process publishing real types over the socket driver, counting every allocation of the process: a change of one of 50 AppInstanceStatus with 10 subscribers took 77,033 allocations of 12.8MB, now 24,526 of 2.6MB; republishing an unchanged one 439 allocations, now 63; GetAll of the 50, 3,069, now 1,369. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A value whose message is above the frame size is compressed with zstd. Publish compressed it to check that it fits and threw the result away, and then every subscriber connection compressed it again to send it: 1+S compressions of the whole message for S subscribers, each with a copy of the message as its input. CheckMaxSize now returns the compressed message it made, Publish keeps it in the Value next to the JSON, and every connection sends that frame as it is. A checkpoint loaded at start is compressed once there too. A value which does not fit is refused by Publish as before; one from a checkpoint is kept and skipped when it would be sent, as before. /debug/pubsub/stats counts the compressions of each publication in compressions, next to compressed_messages, which counts the compressed messages sent. For a DevicePortConfigList of 3.4MB with 5 subscribers, the wire messages built for a change go from 20.5MB to 3.4MB, the input of the one compression. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Publish and every subscription compare a changed item with the one they hold using cmp.Equal, which honours the Equal methods of the types. cmp.Equal compares a []byte element by element through reflection and computes an edit script for every pair of slices, quadratic in their length where they differ. The network types carry certificates as [][]byte in every port: for a DevicePortConfigList of 1.3MB of JSON, 4 DPCs of 8 ports with a chain of three certificates each, cmp.Equal takes 1.8s when nothing changed and 2m24s when the certificates were replaced. Compare []byte with bytes.Equal instead, with a nil slice still unequal to an empty one as before: 5.9ms and 78ms for the same lists. What is equal does not change; a test holds the results against cmp.Equal without the option. Starting a persistent subscription to a DevicePortConfigList of 3.4MB, which compares the list loaded from the checkpoint with the one the publisher sends, goes from 4.6s and 284MB allocated to 96ms and 19MB. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
09119db to
1be3fd5
Compare
A persistent subscription loads its items from the checkpoint at start, and the publisher then sends it every key again, mostly the same JSON. Each was decoded and compared with the item held, only to find it unchanged, and the decoded copy thrown away. Until such a subscription is synchronized, it now keeps the SHA-256 of the JSON each item was decoded from, and drops a value with the same sum without decoding or comparing it. The same JSON decodes to the same item, so no handler call is lost or added. Once synchronized, the publisher sends only changes, which are decoded and compared as before: the sums are dropped, so nothing is hashed or kept after start. /debug/pubsub/stats counts the updates which change nothing in unchanged. Starting a persistent subscription to 50 AppInstanceStatus goes from 27,670 allocations of 3.7MB to 11,016 of 1.9MB, and to a DevicePortConfigList of 3.4MB from 20,080 allocations of 18.7MB to 2,450 of 14.0MB, most of which is the message read. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A test waiting for something a crash would prevent used to wait out its timeout, or to carry its own fatal-log watch and reboot check. CrashWatch (EdgeDevice.WatchForCrash) packages the signs a crash leaves: the fatal an agent logged, the device's serial console, which the broker serves on demand and which names the dead agent or the watchdog's victim within seconds and whatever becomes of the device afterwards, and a reboot the test did not ask for. Reason returns the first of them, or "" while none is known. TestLayerDownloadErrorsBounded uses it instead of its own copy. PACFileProxyConfig lets a device configuration carry a proxy auto-config script inline, base64-encoded as the EVE API takes it, next to the manual, transparent and auto-discovered proxy configurations. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
pubsub delivers every object between pillar's agents as one message of at most 64 KiB; an object that outgrows it makes the publishing agent exit with a fatal and the watchdog reboot the device, and when the object came from the controller the device keeps crashing after every reboot, since the configuration is checkpointed and re-read. Six ordinary configurations or workloads get there, each measured on the real types so that its default overflows with margin: - TestManyStaticDNSEntries: a network instance with 800 static DNS entries (NetworkInstanceConfig, zedagent; 600 entries reach the limit). - TestManyPhysicalIOAdapters: a device model with 200 I/O adapters (PhysicalIOAdapterList, zedagent; 160 reach the limit). - TestHugeConfigItem: debug.enable.ssh with 200 RSA keys, a value of about 79 KB (ConfigItemValueMap, zedagent; 45 KB reach the limit). - TestLargePACFile: a 64 KB proxy auto-config script on the management port (DevicePortConfig, zedagent, and the port status and persisted port config list in nim; 36 KB reach the limit). - TestManyVolumesPerApp: an application with 100 volumes (AppInstanceStatus, zedmanager; 81 volumes reach the limit). - TestManyAppInterfaces: 15 applications with 20 network adapters each (NetworkMetrics, zedrouter; 225 interfaces reach the limit). Each test applies its object, waits until the device reports it applied or in error, and then requires two more metrics reports, which proves pillar outlived the publish; it stops at once, naming the cause, when the device crashes. Every size is a test parameter, so a value below the limit runs the same test as a check of the passing path. The tests form TestLimitsSuite on one device, applications last. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
TestLimitsSuite takes 31 minutes on a libvirt host, 20 of them in TestManyAppInterfaces, which deploys 15 applications. Tag it heavy, as the other suites which deploy many applications or nodes are, so that the evetest workflows keep it off the light runners. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
1be3fd5 to
3c2eea6
Compare
|
@rene done: the bump commit is out, and the branch is rebased on master with #7041. I'll open the bump PR for I also narrowed the subscriber-side hashing after the review note on |
TestNodeClusterSuite: failure analysisTestAppInstancePurgeFailureClaude's conclusionThis matches perfectly — only one retry ( Root causeThis is a reproduction of the exact mechanism already recorded in today's TestClusterVMAppFailureClaude's conclusionThat's enough confirmation of the missing label already shown earlier. I have sufficient evidence to write the root cause. Root causeAll three EVE devices and the k3s cluster are healthy right now ( This is a different specific trigger from the two earlier failures in this run, but it's the same general bug family already recorded in |
evetest: skipped suites for this PRThis PR is confined to Skipped as unrelated to pubsub's internals (Check A) — these suites
Skipped as redundant incidental coverage (Check B) — within
Everything else runs, including the new An engineer can comment |
Move the modules which require pkg/pillar to master c22104d, the merge of #7037: published values marshalled and compressed once for all subscribers, byte slices compared with bytes.Equal, the values a persistent subscription has loaded not decoded again when the publisher resends them at start, and the corrected allocation counters. pkg/newlog, pkg/edgeview, pkg/wwan/mmagent, pkg/kube/kube-init, pkg/installer, pkg/recovertpm and pkg/vtpm vendor pkg/pillar/pubsub; evetest and its SDN VM do not vendor and only take the new version. Done with make bump-eve-pillar. Only pkg/pillar moves; the vendored changes are in pubsub, and the wire protocol is the same. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
TestSecuritySuite: failure analysisTestVaultKeyModeRecoveryOnExt4FailureClaude's conclusionRoot causeLive evidence shows the device genuinely rebooted successfully — it was just slower than the test's fixed 5-minute budget, not hung. The reboot command landed on This is the identical failure signature ( PR #7037's diff touches only TestVaultKeyModeRecoveryOnZFSFailureClaude's conclusionConfirmed same PR as before (#7037) — same scope, only pubsub internals and evetest limits infra, nothing in reboot/nodeagent path. I have enough evidence for a conclusion. Root causeThis shares the identical root cause and mechanism as the already-investigated This is a reproduction of the same mechanism documented tonight in |
TestDeviceConnectivitySuite: failure analysisTestPNACWithoutProxyFailureClaude's conclusionRoot causeLive evidence: This looks directly related to PR #7037: it heavily rewrites |
TestAppsSuite: failure analysisTestPurgeNeverActivatedAppFailureClaude's conclusionRoot causeLive evidence ( This is a known, recurring issue: Regarding PR #7037: its diff only touches |
Description
Follow-up to #5644. That PR was merged together with its last commit, the allocation counters marked "DO NOT MERGE". This PR keeps the counters and makes them exact. It then does the follow-up planned in #5644: pubsub stops redoing the same work for every subscriber.
Before this PR, on every change of any key, each subscriber connection of a publication:
Publishhad already compressed it to check its size and thrown the result away.So one changed key of a publication with N keys and S subscribers cost 2·N·S marshals and N·S unmarshals. With this PR,
Publishmarshals once, every connection sends the same bytes, and a large message is compressed once.Commits
PublicationStats/SubscriptionStatsallocated a new stats struct on every call, four allocations per published message spent on counting. They now look the name up first.testing.AllocsPerRun,MemStats.TotalAlloc). They found three sites counted wrongly, all now fixed.-race, which CI uses, the compiler allocates differently, so only the receive guard test is skipped there.processsection of/debug/pubsub/stats.rejected_frames, next to the publisher'srefused.Publishkeeps the JSON next to the item in an immutableValue.DetermineDiffshands every connection the currentValueof each key. A key is sent again only if itsValueis not the one sent last, so nothing is marshalled or compared per connection, and all connections share the bytes.writev, without a copy.cmp.Equal, as before, so the types'Equalmethods still decide.Get/GetAlldecode their copies from the JSON (no marshal). A subscription handler gets its copy decoded from the received bytes, and only if the handler is set.CheckMaxSizereturns the compressed message it builds to check the size.Publishkeeps it in theValue, and every connection sends that frame.compressions.cmp.Equalcompares a[]bytebyte by byte through reflection, with an edit script per slice, which is quadratic where they differ. The network types carry certificates as[][]bytein every port.cmp.Equaltook 1.8 s with nothing changed and 2 min 24 s with the certificates replaced, in the publisher and in every subscriber. Withbytes.Equalfor[]byteit takes 5.9 ms and 78 ms.cmp.Equal.unchanged.TestLimitsSuite, six configurations or workloads whose objects outgrew the old 64 KB message. All six crashed EVE before pubsub: send messages as frames over unix stream sockets #5644. Here they become regression tests.The vendored pillar in
pkg/newlog,pkg/edgeview,pkg/wwan/mmagentandpkg/kube/kube-initis not bumped here. A separate PR bumps it once this one is merged, so that the modules pin a commit on master.Measured
A test process publishes real pillar types over the socket driver to S subscriptions on tmpfs. The "master" column is master with commits 1–2, so both columns are counted the same way. The figures are the runtime's exact counts for the whole process,
MemStats.MallocsandTotalAlloc, which includeencoding/json. Nothing is sampled.Publishof an unchanged AppInstanceStatusGetGetAllof 50For a change of a 3.4 MB DevicePortConfigList, pubsub's own counters show:
What remains is one marshal, one copy of the message as compression input, and the message each subscriber decompresses.
The whole-process time of that change is dominated by something outside pubsub.
DevicePortConfigList.LogModifybuildscmp.Diff(old, new)for a Trace-level log, so it is evaluated even with tracing off. For the two lists in this test, that onecmp.Difftakes 9.6 s and 686 MB in 1.08 million allocations to produce a 4.5 KB diff. It runs once in the publisher and once in every subscriber. 57LogModifymethods inpkg/pillar/typescallcmp.Diff, and some of them likewise discard the result. That is left for a separate PR.Measured on a device
Christoph Ostarek's limits tests ran on amd64 (evetest, libvirt). They use the evetest memory probe, which reads pubsub's counters from
/debug/pubsub/stats; the probe is not part of this PR. The "master" build is master with commits 1–2, so both builds count the same way. All runs passed on both builds.TestHugeConfigItem, 25 SSH keys, 3 runs per build, alternatingTestLimitsSuite, bytes pubsub allocates while each object is appliedTestManyStaticDNSEntriesTestManyPhysicalIOAdaptersTestHugeConfigItemTestLargePACFileTestManyVolumesPerAppTestManyAppInterfacesThe two large cases are the per-subscriber cost commit 3 removes. Every change of any of the many keys made every subscriber connection deep-copy and re-marshal all of them.
Compatibility
The wire protocol, checkpoints and the
Publication/SubscriptionAPI are unchanged. Handlers are called exactly when they were before. Publishers skip unchanged items as before: identical JSON now, otherwisecmp.Equalwith the types'Equalmethods.Two internal driver interfaces change; only the drivers in
pkg/pillar/pubsubimplement them:DriverPublisher.CheckMaxSizealso returns the prepared frame.LocalCollectionholds*Value.How to test and validate this PR
make -C pkg/pillar test. New tests:TestDetermineDiffsSharesValues: every connection gets the sameValue, a key is sent again only after a change, and deleted keys come first.TestPublicationCopies,TestHandlerGetsCopy:Get/GetAlland handlers get copies that callers can change.TestPublishHonoursEqual: an item equal by its type'sEqualmethod is still unchanged.TestCompressedOnce: one compression for three subscribers, per change.TestEqualOptsSameResult,TestEqualOptsFast: the comparer changes no result and is fast on 1 MB slices.TestIdenticalNotDecoded: while a persistent subscription starts, the same JSON is neither decoded nor handed to a handler; once it is synchronized, JSON is decoded and compared as before; equal items never reach handlers.TestAllocCountsSend,TestAllocCountsReceive,TestAllocSizeMatchesRuntime: the counters against the runtime.TestRejectedFrameCounted.curl -s http://127.1:6543/debug/pubsub/stats(aftereve http-debug).compressionsis at most one per change of a large value.unchangedcounts the keys resent to persistent subscriptions at start.process.DeepCopycounters stay at 0: pubsub no longer deep-copies.TestLimitsSuite(commits 7–8). It passed 6/6 on this PR, as on master.Changelog notes
No user-facing changes. Pubsub encodes and compresses each published value once for all subscribers, and compares large network configurations far faster, so EVE services use less memory and CPU.
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't check them. Tested on amd64 with evetest on libvirt; arm64 is not tested.