Repository navigation
pubsub: send messages as frames over unix stream sockets - #5644
Conversation
fd55ae3 to
9ace3e8
Compare
|
@eriknordmark , @milan-zededa , pls, let's take a look closer to this PR since it touches a critical pubsub infrastructure.... |
|
@rene @eriknordmark @michel-zededa I did not touch "pubsub-large" yet. this is experimental PR. I'm 99% sure it works but let's test |
There was a problem hiding this comment.
Pull request overview
This PR updates pubsub’s IPC transport to remove the historical ~65KB message size ceiling by switching from unixpacket sockets to unix stream sockets and introducing length-prefixed framing using the vendored getlantern/framed library with big frames enabled.
Changes:
- Switch pubsub socket transport from
unixpackettounixstream sockets. - Wrap IPC reads/writes with
framed.Reader/framed.WriterusingEnableBigFrames(). - Increase the socketdriver
maxsizelimit to 10MB and update related tests/comments.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/pillar/pubsub/socketdriver/subscribe.go | Subscriber now dials unix and reads framed messages. |
| pkg/pillar/pubsub/socketdriver/publish.go | Publisher now accepts unix stream conns, reads framed requests, and writes framed updates. |
| pkg/pillar/pubsub/socketdriver/driver.go | Raises maxsize to 10MB; adds framed reader/writer helpers and removes buffer pool. |
| pkg/pillar/pubsub/reverse/subscribe.go | Reverse subscriber now listens on unix and reads framed messages. |
| pkg/pillar/pubsub/reverse/publish.go | Reverse publisher now dials unix and writes framed messages. |
| pkg/pillar/pubsub/checkmaxsize_test.go | Updates test commentary/cases to reflect removal of the old ~65KB boundary behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| s.log.Tracef("sendUpdate(%s): key %s\n", s.name, key) | ||
| // base64-encode to avoid having spaces in the key and val | ||
| sendKey := base64.StdEncoding.EncodeToString([]byte(key)) | ||
| sendVal := base64.StdEncoding.EncodeToString(val) | ||
| buf := fmt.Sprintf("update %s %s %s", s.topic, sendKey, sendVal) | ||
| if len(buf) >= maxsize { | ||
| s.log.Fatalf("Too large message (%d bytes) sent to %s topic %s key %s", | ||
| len(buf), s.name, s.topic, key) | ||
| } | ||
| _, err := sock.Write([]byte(buf)) | ||
| _, err := writer.Write([]byte(buf)) | ||
| return err |
There was a problem hiding this comment.
These send* methods no longer enforce the driver maxsize limit. Since PublicationImpl.Publish() does not call CheckMaxSize(), oversized values can now be sent over IPC, bypassing the intended 10MB cap and potentially causing large allocations/pressure on subscribers. Reintroduce a size check (ideally returning an error rather than Fatal) before writer.Write(), or enforce maxsize at publish/serialize time for all outgoing messages.
| // CheckMaxSize returns an error if too large. | ||
| // With framed big frames the limit is ~4GB, so this is unlikely to trigger. |
There was a problem hiding this comment.
The comment on CheckMaxSize is misleading: the framed transport may support ~4GB frames, but this driver still enforces maxsize (currently 10MB) and this function can definitely trigger. Please update the comment to reflect the actual enforced limit and its purpose (protecting memory/IPC).
| // CheckMaxSize returns an error if too large. | |
| // With framed big frames the limit is ~4GB, so this is unlikely to trigger. | |
| // CheckMaxSize returns an error if the serialized message exceeds maxsize. | |
| // Even though the framed transport can support ~4GB frames, this driver enforces | |
| // a much smaller application-level limit (currently 10MB) to protect memory/IPC. |
| // NewFramedReader wraps a net.Conn with a framed.Reader that uses big frames | ||
| // (32-bit length prefix), allowing messages up to ~4GB. | ||
| func NewFramedReader(conn net.Conn) *framed.Reader { | ||
| r := framed.NewReader(conn) | ||
| r.EnableBigFrames() | ||
| return r | ||
| } | ||
|
|
There was a problem hiding this comment.
NewFramedReader/NewFramedWriter enable big frames (~4GB), but the framed.Reader.ReadFrame() API allocates based on the length prefix and does not enforce a maximum on reads (only writes check maxFrameLength). Given this package defines maxsize (10MB), consider centralizing a bounded read helper in socketdriver (e.g., read length, reject > maxsize, then read) so all IPC consumers are protected from oversized/malicious frames.
| // NewFramedReader wraps a net.Conn with a framed.Reader that uses big frames | |
| // (32-bit length prefix), allowing messages up to ~4GB. | |
| func NewFramedReader(conn net.Conn) *framed.Reader { | |
| r := framed.NewReader(conn) | |
| r.EnableBigFrames() | |
| return r | |
| } | |
| // NewFramedReader wraps a net.Conn with a framed.Reader. | |
| // Big frames are not enabled to avoid unbounded allocations on reads. | |
| func NewFramedReader(conn net.Conn) *framed.Reader { | |
| r := framed.NewReader(conn) | |
| return r | |
| } |
| reader := socketdriver.NewFramedReader(conn) | ||
|
|
||
| for { | ||
| // wait for readable conn | ||
| if err := pubsub.ConnReadCheck(conn); err != nil { | ||
| if err != io.EOF { | ||
| log.Errorf("serveConnection: Error on connReadCheck: %s", | ||
| err) | ||
| } | ||
| break | ||
| } | ||
|
|
||
| buf, doneFunc := socketdriver.GetBuffer() | ||
| defer doneFunc() | ||
|
|
||
| count, err := conn.Read(buf) | ||
| frame, err := reader.ReadFrame() | ||
| if err != nil { | ||
| if err != io.EOF { | ||
| log.Errorf("serveConnection: Error on read: %s", | ||
| err) | ||
| } | ||
| break | ||
| } | ||
| retChan <- string(buf[:count]) | ||
| retChan <- string(frame) |
There was a problem hiding this comment.
serveConnection uses framed.Reader.ReadFrame(), which can allocate arbitrarily large buffers based on the peer-provided length prefix (big frames are enabled). Since this listener accepts local connections, a buggy/malicious publisher can DoS the subscriber via OOM. Use a bounded read (enforce socketdriver maxsize) before allocating/reading the payload.
| // With framed big frames (32-bit length prefix) the IPC message size | ||
| // limit is ~4GB, so the old 65KB boundary tests no longer apply. | ||
| // The "large tag" mechanism (pubsub-large-*) still has its own 1MB | ||
| // limit enforced by large.go, so those cases remain. |
There was a problem hiding this comment.
TestCheckMaxSize no longer exercises the driver maxsize boundary after increasing it to 10MB. To avoid regressions, add at least one case that exceeds the new maxsize (e.g., ~11–12MB payload after base64 overhead) and assert that CheckMaxSize returns an error.
| frame, err := s.reader.ReadFrame() | ||
| if err != nil { | ||
| errStr := fmt.Sprintf("connectAndRead(%s): sock read failed %s", | ||
| s.name, err) |
There was a problem hiding this comment.
Using framed.Reader.ReadFrame() is unsafe here because the vendored framed implementation allocates a buffer based solely on the on-wire length prefix and does not enforce any maximum. With big frames enabled, a malformed or malicious peer can trigger very large allocations (OOM) before any pubsub-level size checks run. Prefer reading the length prefix, rejecting frames > maxsize (10MB), and then reading into a bounded/reusable buffer (e.g., use framed.Reader.Read() with a preallocated maxsize buffer or implement a capped ReadFrame helper).
| // Read request | ||
| res, err := conn.Read(buf) | ||
| frame, err := reader.ReadFrame() | ||
| if err != nil { | ||
| // Peer process could have died | ||
| s.log.Errorf("serveConnection(%s/%d) error: %v", s.name, instance, err) | ||
| return |
There was a problem hiding this comment.
serveConnection reads the initial request using framed.Reader.ReadFrame(), which allocates according to the peer-provided length prefix with no upper bound. Even though the request should be small, a bad client can force large allocations and DoS the publisher. Consider reading the request with a fixed-size buffer (since the request is known to be small) or adding a max-frame-size check before allocating/reading the full payload.
eriknordmark
left a comment
There was a problem hiding this comment.
According to https://github.com/getlantern/framed/blob/master/framed.go the max length is 65535 so this doesn't change anything but adding silent message truncation for larger messages,
If framed.go did support 4GByte size messages my question would be what the max memory usage would be if we started having pubsub use messages that are in the size of Megabytes. I see you removed the buffer pool which means that memory usage would increase even if the we stay below 65536, which already is a significant concern; it takes a while for the GC to kick in.
So I don't understand what problem we need to fix here - replacing teaching folks about a max size and how to use pubsub 'large' with additional OOM risk doesn't seem like an improvement to me.
9ace3e8 to
ce6c077
Compare
no, we use
when we use pubsub large we load message to RAM one way or another, you are right, Framed doesn't increase the size but send a single message as fragments then assemble them back to full packet. the idea was to handle messages that are not annotated as "pubsub-large" but suddenly become >64k in side. This exactly what we had recently if 20 SR-IOV VF were enabled @eriknordmark since pbsub-large is still in places, only messages that accidentally exceed the sized will benefit. To avoid the concern about GC pressure I'll revert to a buffer allocated on client side. Basically the same behavior for messages <=64k and grow buffer if necessary for bigger messages |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // We add new test cases that exceed the 10MB limit to verify it. | ||
| // The "large tag" mechanism (pubsub-large-*) still has its own 1MB | ||
| // limit enforced by large.go, so those cases remain. |
There was a problem hiding this comment.
The comment says the new test cases “exceed the 10MB limit”, but these cases use 8 * 1024 * 1024. If the intent is to exceed the 10MB maxsize after base64/message overhead, please call that out explicitly (otherwise consider using a size that is unambiguously over the limit).
| // We add new test cases that exceed the 10MB limit to verify it. | |
| // The "large tag" mechanism (pubsub-large-*) still has its own 1MB | |
| // limit enforced by large.go, so those cases remain. | |
| // We add new test cases with 8MiB payloads; while the raw payload size | |
| // is below 10MB, base64 encoding and message framing overhead push the | |
| // resulting IPC message size over the effective 10MB maxsize we verify. |
| // NewFramedReader wraps a net.Conn with a framed.Reader that uses big frames | ||
| // (32-bit length prefix). Callers must use readFrame() instead of calling | ||
| // ReadFrame() directly, to enforce the maxsize limit and prevent unbounded | ||
| // memory allocation from a buggy peer. | ||
| func NewFramedReader(conn net.Conn) *framed.Reader { | ||
| r := framed.NewReader(conn) | ||
| r.EnableBigFrames() | ||
| return r | ||
| } | ||
|
|
||
| // logs a message if the allocation changed | ||
| var lastLoggedAllocated uint32 | ||
|
|
||
| func maybeLogAllocated(log *base.LogObject) { | ||
| currentAllocated := atomic.LoadUint32(&allocated) | ||
| currentLastAllocated := atomic.LoadUint32(&lastLoggedAllocated) | ||
| if currentLastAllocated == currentAllocated { | ||
| return | ||
| // ReadFrame reads a single frame from the framed.Reader and rejects frames | ||
| // that exceed maxsize. This protects against unbounded memory allocation | ||
| // from a buggy or malicious peer sending a large length prefix. | ||
| func ReadFrame(r *framed.Reader) ([]byte, error) { | ||
| frame, err := r.ReadFrame() | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| if len(frame) > maxsize { | ||
| return nil, fmt.Errorf("received frame of %d bytes exceeds max %d", | ||
| len(frame), maxsize) | ||
| } |
There was a problem hiding this comment.
ReadFrame() currently calls r.ReadFrame() and only then checks len(frame) > maxsize. In getlantern/framed, Reader.ReadFrame() allocates make([]byte, n) based on the length prefix before returning, so this code does not prevent unbounded memory allocation / OOM from a large length prefix. To actually enforce maxsize, read and validate the length prefix first (before allocating) or use a bounded read strategy and close/drain the connection on oversize frames. Also, the doc comment above NewFramedReader refers to readFrame() but the exported helper is ReadFrame().
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5644 +/- ##
==========================================
+ Coverage 28.76% 29.27% +0.51%
==========================================
Files 544 558 +14
Lines 98795 100610 +1815
==========================================
+ Hits 28421 29457 +1036
- Misses 68086 68693 +607
- Partials 2288 2460 +172 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@eriknordmark @rene it would be interesting to kick eden tests to see how it works |
Actually we don't know how many copies of that 10MByte we will end up in memory when sending and receiving it.
I think it would be a bad idea to replace the annoying but quickly detected log.Fatal when trying a new configuration (such as the above more VF case) with an increased risk that devices will hit an OOM in production based on unpredictable load conditions. |
08dc2be to
3ddf8a5
Compare
29d0798 to
9a379a1
Compare
pkg/newlog, pkg/edgeview, pkg/wwan/mmagent and pkg/kube/kube-init talk pubsub to pillar through a vendored copy of pkg/pillar/pubsub. A copy still using unixpacket sockets cannot connect to the stream sockets pillar listens on now, so move all of them to the previous commit. The other dependency updates are the versions that pillar requires. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Temporary, for review only: this is how the memory numbers in the PR description were measured. It is dropped before merge. Every allocation pubsub makes for a message is counted at its call site: JSON of published items, wire messages and their compressed frames, read buffers including decompressed messages, decoding and copies of received messages, DeepCopy. Each kind reports the count, the bytes asked for and the bytes the Go allocator reserves (the request rounded up to its size class) in /debug/pubsub/stats. The sites follow the compiler's escape analysis (go build -gcflags=-m), so conversions Go keeps on the stack are not counted. Not counted: allocations inside encoding/json while decoding, the internal buffers of the zstd encoder and decoder, and the boxing of log arguments. Master was measured with the same counters at its own call sites, including the pubsub-large files and the 64 KiB read buffer pool: branch rucoder/pubsub-alloc-counters-master. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
1ad443d to
c8e1786
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
eriknordmark
left a comment
There was a problem hiding this comment.
LGTM.
@milan-zededa Are the evetest failures something we expect at the current state of the evetest CI code?
evetest: skipped tests for this PRThis PR is confined entirely to the pubsub transport ( Because pubsub is the IPC backbone for essentially every agent, almost every evetest suite is at least incidentally relevant, so very little was skipped. The suites/components most directly exercising this change were kept, in particular:
Skipped (redundant onboarding duplicates):
These two just onboard a device over a plain static-IP or network-boot path with no unusually large or An engineer can comment |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as off-topic.
This comment was marked as off-topic.
TestAppsSuite: failure analysisTestWorkerPoolSaturationFailureClaude's conclusionRoot causeThe test timed out waiting for config v55 to be confirmed because zedbox (pillar) crashed and the watchdog rebooted the device, not because of a hang. The live console log shows the device applying config v55 at zedbox panicked with This doesn't match any entry in TestVMAppPurgeReplacesVMIRSFailureClaude's conclusionThis confirms it directly: Root causeThis is the same root cause as the already-investigated TestVGAPassthroughNoHostAccessFailureClaude's conclusionConfirmed — PID 11165 is Root cause: This is the same long-standing, well-known test-harness bug recorded repeatedly in |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as outdated.
This comment was marked as outdated.
@eriknordmark There is still lots of noise (failures not related to actual EVE bugs), but it should get better once these are merged (and PRs on master are rebased): |
The counters of pubsub's own allocations came in with the last commit of lf-edge#5644, which was meant to be dropped before merge. They stay, so make them exact and keep them so. PublicationStats and SubscriptionStats passed a new PubStats or SubStats to LoadOrStore on every call, which allocated it each time: four allocations per published message spent on counting. Look the name up first; a new one is stored only the first time. Tests now hold the counters against the runtime's own count of allocations, and they found sites counted wrongly: []byte(key) passed to AppendEncode or EncodeToString is not copied, a buffer kept as a *[]byte moves its slice header to the heap, and the string of the persistence flag in hello escapes. TestAllocCountsSend and TestAllocCountsReceive compare the counters with testing.AllocsPerRun, and TestAllocSizeMatchesRuntime compares the size classes with MemStats.TotalAlloc, so a Go upgrade that changes what is allocated fails a test instead of making the counters drift. Both pass on Go 1.25 and 1.26. Under the race detector, which CI runs the tests with, the compiler does not optimise append(s, make([]T, n)...), so base64 decoding of a key above 32 bytes allocates twice: the receive test is skipped there, the send test runs. The pubsub-large counters went with pubsub-large; drop them. DeepCopy does not know the topic of what it copies, so report its counters in a "process" section of /debug/pubsub/stats instead of as a publication named "(DeepCopy)". Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
The counters of pubsub's own allocations came in with the last commit of #5644, which was meant to be dropped before merge. They stay, so make them exact and keep them so. PublicationStats and SubscriptionStats passed a new PubStats or SubStats to LoadOrStore on every call, which allocated it each time: four allocations per published message spent on counting. Look the name up first; a new one is stored only the first time. Tests now hold the counters against the runtime's own count of allocations, and they found sites counted wrongly: []byte(key) passed to AppendEncode or EncodeToString is not copied, a buffer kept as a *[]byte moves its slice header to the heap, and the string of the persistence flag in hello escapes. TestAllocCountsSend and TestAllocCountsReceive compare the counters with testing.AllocsPerRun, and TestAllocSizeMatchesRuntime compares the size classes with MemStats.TotalAlloc, so a Go upgrade that changes what is allocated fails a test instead of making the counters drift. Both pass on Go 1.25 and 1.26. Under the race detector, which CI runs the tests with, the compiler does not optimise append(s, make([]T, n)...), so base64 decoding of a key above 32 bytes allocates twice: the receive test is skipped there, the send test runs. The pubsub-large counters went with pubsub-large; drop them. DeepCopy does not know the topic of what it copies, so report its counters in a "process" section of /debug/pubsub/stats instead of as a publication named "(DeepCopy)". Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
`gh pr diff` fails with HTTP 406 for a PR touching more than 300 files, which a vendor bump easily exceeds (e.g. PR #5644). discover-suites then failed before any suite was scheduled, so evetest never ran for such PRs even though "Ready for evetest" was reported. Check out the PR head from the upstream repo with actions/checkout, blobless and with the full history of every branch (about 20 MB), and diff it against its merge-base with the base branch. Vendored code is left out of the diff; instead, the vendored modules added, removed or bumped are read from vendor/modules.txt and listed in pr-context.txt. If fetching or diffing fails, Claude is not asked for a skip decision and the run falls back to the full suite instead of failing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit e6f1a74)
Description
NOTE: commit 4 counts the memory pubsub allocates for messages, per topic. It was first meant to be dropped before merge; it stays, because it costs nothing measurable and tests keep its counts exact across code changes and Go upgrades (checked with Go 1.25 and 1.26).
A pubsub message is limited to 64 KB. The subscriber reads each unixpacket datagram into a 64 KB buffer, and the publisher calls
Fatalffor anything larger. Statuses whose size depends on what happens on the device cross that limit in the field. When one does, zedbox exits and the watchdog reboots the node (see 0e1afbf and d14bcca). Christoph Ostarek's evetestTestLimitsSuitereproduces this: master crashes in each of its oversized-object tests.This PR replaces the transport and removes the 64 KB limit. Messages above 1 MB are compressed, so even the largest network configurations fit. Measured with counters inside pubsub, it also allocates less memory than master.
Commits
SOCK_STREAMunix socket: a 4-byte little-endian length, then the payload.GET /debug/pubsub/statson zedbox's debug server: messages, frames, bytes, largest message, read buffers held, and so on. They are always on and cost a few atomic additions per message.EPROTOTYPE. The subscriber logs that once instead of retrying silently, and the publisher no longer deletes such a live socket as a leftover.Subscriber.Stop()no longer races with the reader goroutine over the socket.pubsub-large-fields no longer go to files under/persist/pubsub-large. On a test run those files were all empty fields: 280 files, 9.5 KB.onboot.shremoves what older versions left in/persist/pubsub-large.DetermineDiffscompared a fresh encoding (struct field order) with its stored copy, which had been re-encoded with sorted keys, so they never matched. In a unit test, updating one of five keys ten times delivers 50 frames on master and 10 with this PR. On a device,domainmgr/ProcessMetricsent 6,981 messages in a run on master and 1,818 with this PR.DevicePortConfigcarries its network's proxy configuration: CA certificates, plus a PAC script that was never moved to a file. The DPC list keeps dozens of DPCs while nim verifies new ones. On the real types, 30 DPCs × 24 ports, with a CA chain on every port and a 300 KB PAC on two, is 30 MB of JSON. Master can't carry that either: it fails once the untagged part passes 64 KB.eve_pubsub_max_frame(default1M) andeve_pubsub_max_message(default30M), e.g.eve_pubsub_max_message=64Mfor a configuration larger still. Every process reads them throughkernelcmdlinebefore it sends or receives a frame, so both ends of a connection agree. A value that can't be used as given is logged once and replaced by the default, or clamped to the allowed range. They are documented indocs/BOOTING.md.Publish()checks the size before the value is stored or checkpointed, logs it, counts it asrefusedin/debug/pubsub/stats, and returns an error.send()skips such a value if one arrives anyway, e.g. from a checkpoint written by another version, so nothing can loop across reboots. A subscriber counts the frames it rejects, too large or not decompressing, asrejected_frames./debug/pubsub/stats: the JSON of published items, the wire messages (compressed frames included), read buffers, copies and decoding of received messages, and DeepCopy for the whole process. Each as the number of allocations, the bytes asked for, and the bytes the Go allocator reserves.Publishto delivery (1 KB to 300 KB, 1 and 10 subscribers, with and without this commit) shows no measurable difference.pkg/newlog,pkg/edgeview,pkg/wwan/mmagentandpkg/kube/kube-inittalk pubsub through a vendoredpkg/pillarand must use the same framing.Compatibility
All pubsub peers on a device ship in the same EVE image, so the old and new wire formats are never mixed on one device. An out-of-date peer is reported once as a wire protocol mismatch, not silently retried. Persisted pubsub checkpoints are unchanged, and compression is on the wire only. The pubsub API is unchanged; the one behavioural difference is that
Publish()returns an error for a value that doesn't fit even compressed, where master's zedbox would exit.Memory: counted, not estimated
Every allocation pubsub makes for a message is counted where it is made, on master and on this PR:
Bytes are what the Go allocator reserves: the request rounded up to its size class. The counted sites follow the compiler's escape analysis (
-gcflags=-m). Not counted, on either side: allocations insideencoding/jsonwhile decoding, and boxing of log arguments. Master is undercounted there, never overcounted.The test is evetest
TestHugeConfigItemwith 25 SSH keys on amd64 (libvirt), 3 runs per build, master and this PR alternating. It was measured on commits 1 and 2; the largest message in it is 30 KB, so commit 3 changes nothing on the wire at these sizes.Correction: the guard tests of commit 4 found that the master counters counted one conversion
DecodeStringmakes without copying. The master figures below are therefore a range: as counted, and with that conversion taken out, which is at most the frame bytes master's subscribers received. This PR's figures move by under 0.2 MiB with the corrected counters.How to test and validate this PR
make -C pkg/pillar test. The pubsub and socketdriver tests cover:DevicePortConfigListof 30 DPCs × 24 ports (30 MB → 42 KB), an incompressible value refused while the other keys still flow, and a decompression bomb rejected before allocatingTestLimitsSuite: all 6 tests pass with this PR, and master crashes in each.TestLargePACFilewithEVETEST_PAC_KIB=1024: the port configuration, the port status and nim's DPC list exceed 1 MB and travel compressed. On the device,zedagent/DevicePortConfig,nim/DeviceNetworkStatusandnim/DevicePortConfigListsent 1.4 MB messages as 21–22 KB frames, nothing was refused, and the device reported the configuration with the PAC and no error.TestHugeConfigItemwithEVETEST_SSH_KEYS=80reproduces the crash on master: zedagent exits with "Too large message"./persist/pubsub-largeis gone after boot.debug.enable.ssh. The device applies it, logs a "Large message (N bytes, max 1048576 uncompressed)" warning, and does not reboot.eve http-debug, thencurl -s http://127.1:6543/debug/pubsub/stats, theneve http-debug stop.eve_pubsub_max_message=64Madded todom0_extra_args. zedbox logspubsub: frames up to 1048576 bytes, messages up to 67108864 bytes. A value such aseve_pubsub_max_message=lotsis logged as an error and the default is used.DevicePortConfigListread back.Changelog notes
EVE services can now exchange pubsub messages far larger than 64 KB: up to 1 MB as is, and larger ones compressed. Large network configurations with many ports, certificates and PAC files no longer make zedbox exit and reboot the device. A value too large even compressed is refused and logged instead. The largest message is 30 MB by default; the
eve_pubsub_max_messagekernel parameter raises it. EVE no longer writes per-field files under/persist/pubsub-large, and pubsub allocates about two thirds less memory.PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't check them. The PR is tested on amd64 only, with evetest on libvirt; it is not tested on arm64.