Skip to content

pubsub: send messages as frames over unix stream sockets - #5644

Merged
eriknordmark merged 5 commits into
lf-edge:masterfrom
rucoder:rucoder/pubsub-msg-size
Nov 4, 2026
Merged

eriknordmark merged 5 commits into
lf-edge:masterfrom
rucoder:rucoder/pubsub-msg-size

Conversation

@rucoder

@rucoder rucoder commented Mar 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

NOTE: commit 4 counts the memory pubsub allocates for messages, per topic. It was first meant to be dropped before merge; it stays, because it costs nothing measurable and tests keep its counts exact across code changes and Go upgrades (checked with Go 1.25 and 1.26).

A pubsub message is limited to 64 KB. The subscriber reads each unixpacket datagram into a 64 KB buffer, and the publisher calls Fatalf for anything larger. Statuses whose size depends on what happens on the device cross that limit in the field. When one does, zedbox exits and the watchdog reboots the node (see 0e1afbf and d14bcca). Christoph Ostarek's evetest TestLimitsSuite reproduces this: master crashes in each of its oversized-object tests.

This PR replaces the transport and removes the 64 KB limit. Messages above 1 MB are compressed, so even the largest network configurations fit. Measured with counters inside pubsub, it also allocates less memory than master.

Commits

  1. pubsub: send messages as frames over unix stream sockets
    • Every message is a frame on a SOCK_STREAM unix socket: a 4-byte little-endian length, then the payload.
    • A frame carries up to 1 MB; commit 3 compresses anything larger. A warning is logged whenever a publication sends a new largest message above 64 KB.
    • Update values are sent without base64; keys stay base64-encoded. An update is a quarter smaller, and nobody encodes or decodes the value.
    • Read buffers come from pools by size class (1, 4, 16 and 64 KB). A reader takes one only once a frame's header has arrived, and gives it back as soon as the frame is consumed. Idle subscriptions, which is most of the 250 or so in zedbox, hold no buffer at all. A frame above 64 KB gets a one-off buffer.
    • Counters per topic are served at GET /debug/pubsub/stats on zedbox's debug server: messages, frames, bytes, largest message, read buffers held, and so on. They are always on and cost a few atomic additions per message.
    • A peer still on a unixpacket socket gets EPROTOTYPE. The subscriber logs that once instead of retrying silently, and the publisher no longer deletes such a live socket as a leftover. Subscriber.Stop() no longer races with the reader goroutine over the socket.
    • The reverse pubsub and ipcmonitor use the same framing.
  2. pubsub: carry fields tagged pubsub-large inside the message
    • With 1 MB messages, pubsub-large- fields no longer go to files under /persist/pubsub-large. On a test run those files were all empty fields: 280 files, 9.5 KB.
    • Every subscriber no longer decodes and re-encodes every message on every topic to look for such files.
    • The JSON names stay, so persisted checkpoints read back unchanged. onboot.sh removes what older versions left in /persist/pubsub-large.
    • This also fixes a master bug: a publication resent all its keys to every subscriber whenever one key changed. DetermineDiffs compared a fresh encoding (struct field order) with its stored copy, which had been re-encoded with sorted keys, so they never matched. In a unit test, updating one of five keys ten times delivers 50 frames on master and 10 with this PR. On a device, domainmgr/ProcessMetric sent 6,981 messages in a run on master and 1,818 with this PR.
  3. pubsub: compress messages above 1MB, refuse what does not fit
    • The network types repeat their content. Every port of a DevicePortConfig carries its network's proxy configuration: CA certificates, plus a PAC script that was never moved to a file. The DPC list keeps dozens of DPCs while nim verifies new ones. On the real types, 30 DPCs × 24 ports, with a CA chain on every port and a 300 KB PAC on two, is 30 MB of JSON. Master can't carry that either: it fails once the untagged part passes 64 KB.
    • A message above 1 MB is compressed with zstd and sent in a frame flagged by the top bit of its length, transparently to the types. The list above goes over the socket as 42 KB. By default a frame carries at most 1 MB and a message at most 30 MB uncompressed, which the decoder checks before it allocates. Messages up to 1 MB, almost all of them, are unchanged on the wire.
    • Both sizes can be set on the kernel command line: eve_pubsub_max_frame (default 1M) and eve_pubsub_max_message (default 30M), e.g. eve_pubsub_max_message=64M for a configuration larger still. Every process reads them through kernelcmdline before it sends or receives a frame, so both ends of a connection agree. A value that can't be used as given is logged once and replaced by the default, or clamped to the allowed range. They are documented in docs/BOOTING.md.
    • A value that doesn't fit even compressed is refused, never fatal. Publish() checks the size before the value is stored or checkpointed, logs it, counts it as refused in /debug/pubsub/stats, and returns an error. send() skips such a value if one arrives anyway, e.g. from a checkpoint written by another version, so nothing can loop across reboots. A subscriber counts the frames it rejects, too large or not decompressing, as rejected_frames.
    • A frame in a buffer of its own (decompressed, or above 64 KB) is no longer copied: the subscriber takes the value straight from it.
    • Cost: compressing the 30 MB list takes 35 ms and decompressing it 16 ms, against 166 ms to marshal it and 217 ms to unmarshal it.
  4. pubsub: count the memory pubsub allocates for messages
    • Counted where it is allocated, per topic, in /debug/pubsub/stats: the JSON of published items, the wire messages (compressed frames included), read buffers, copies and decoding of received messages, and DeepCopy for the whole process. Each as the number of allocations, the bytes asked for, and the bytes the Go allocator reserves.
    • Which conversions and buffers reach the heap is up to the compiler and the standard library, so tests hold the counters against the runtime's own exact count of allocations on the send and receive paths, and the size classes against the bytes the runtime accounts. A code change or a Go upgrade that changes either fails a test instead of skewing the numbers.
    • Cost: one atomic addition per counted allocation, about 20 per message. A benchmark of Publish to delivery (1 KB to 300 KB, 1 and 10 subscribers, with and without this commit) shows no measurable difference.
    • Master was measured with the same counters at its own call sites, for the tables below: rucoder/pubsub-alloc-counters-master.
  5. Update the vendored pillar for the framed pubsub transport. pkg/newlog, pkg/edgeview, pkg/wwan/mmagent and pkg/kube/kube-init talk pubsub through a vendored pkg/pillar and must use the same framing.

Compatibility

All pubsub peers on a device ship in the same EVE image, so the old and new wire formats are never mixed on one device. An out-of-date peer is reported once as a wire protocol mismatch, not silently retried. Persisted pubsub checkpoints are unchanged, and compression is on the wire only. The pubsub API is unchanged; the one behavioural difference is that Publish() returns an error for a value that doesn't fit even compressed, where master's zedbox would exit.

pubsub-counted pubsub-limits

Memory: counted, not estimated

Every allocation pubsub makes for a message is counted where it is made, on master and on this PR:

  • JSON of published items
  • wire messages, and base64 on master
  • read buffers
  • decoding and copies of received messages
  • pubsub-large files and the JSON around them
  • DeepCopy

Bytes are what the Go allocator reserves: the request rounded up to its size class. The counted sites follow the compiler's escape analysis (-gcflags=-m). Not counted, on either side: allocations inside encoding/json while decoding, and boxing of log arguments. Master is undercounted there, never overcounted.

The test is evetest TestHugeConfigItem with 25 SSH keys on amd64 (libvirt), 3 runs per build, master and this PR alternating. It was measured on commits 1 and 2; the largest message in it is 30 KB, so commit 3 changes nothing on the wire at these sizes.

Correction: the guard tests of commit 4 found that the master counters counted one conversion DecodeString makes without copying. The master figures below are therefore a range: as counted, and with that conversion taken out, which is at most the frame bytes master's subscribers received. This PR's figures move by under 0.2 MiB with the corrected counters.

allocated by pubsub master this PR change
bytes, zedbox start to test end 61.7–67.1 MiB 21.6 MiB −65% to −68%
allocations, same period 136,380–152,712 59,834 −56% to −61%
bytes while the large property is applied (~15 s) 25.3–27.6 MiB 8.8 MiB −65% to −68%
by part, MiB, whole run master this PR
send: build the wire message 27.8 3.3
receive: decode and copy 10.4–15.8 3.1
pubsub-large: write fields to files 4.4 0
pubsub-large: read fields back 5.0 0
receive: read buffers 0.8 0.4
JSON of published items 5.2 5.9
DeepCopy 8.1 8.9
  • Read buffers: master hands out 64 KB for every read, 510 MiB over a run. This PR averages 2.6 KB per read, 7.7 MiB in total.
  • JSON and DeepCopy: the PR doesn't change them. They are slightly higher here only because these runs published slightly more items. They are now the largest pubsub cost, and caching the JSON per key is planned as a follow-up.
  • Process level, 3 runs each: the PR's numbers are within master's run-to-run spread.
process level master this PR
zedbox RSS, mean 126.8 ± 0.4 MiB 126.5 ± 0.5 MiB
zedbox RSS, peak 134.0 ± 1.2 MiB 135.0 ± 1.4 MiB
pillar cgroup RSS, peak 78.4 ± 1.4 MiB 79.5 ± 0.8 MiB
device MemAvailable, mean 7,147.9 MiB 7,149.6 MiB

How to test and validate this PR

  • Unit tests: make -C pkg/pillar test. The pubsub and socketdriver tests cover:
    • framing, and frames above 64 KB and up to the limit
    • buffer pooling, the counters, and stopping a subscription while its publisher sends
    • pubsub-large fields arriving inline
    • compression: a 20 MB value end to end, a real-types DevicePortConfigList of 30 DPCs × 24 ports (30 MB → 42 KB), an incompressible value refused while the other keys still flow, and a decompression bomb rejected before allocating
    • the kernel parameters: defaults, size suffixes, values that don't parse, clamping, and a message size below the frame size
  • evetest:
    • Christoph Ostarek's TestLimitsSuite: all 6 tests pass with this PR, and master crashes in each.
    • TestLargePACFile with EVETEST_PAC_KIB=1024: the port configuration, the port status and nim's DPC list exceed 1 MB and travel compressed. On the device, zedagent/DevicePortConfig, nim/DeviceNetworkStatus and nim/DevicePortConfigList sent 1.4 MB messages as 21–22 KB frames, nothing was refused, and the device reported the configuration with the PAC and no error.
    • TestHugeConfigItem with EVETEST_SSH_KEYS=80 reproduces the crash on master: zedagent exits with "Too large message".
  • On a device:
    1. Boot and onboard. No agent logs a wire protocol mismatch.
    2. /persist/pubsub-large is gone after boot.
    3. Push a configuration larger than 64 KB, for example many SSH keys in debug.enable.ssh. The device applies it, logs a "Large message (N bytes, max 1048576 uncompressed)" warning, and does not reboot.
    4. Look at the counters: eve http-debug, then curl -s http://127.1:6543/debug/pubsub/stats, then eve http-debug stop.
    5. Boot with eve_pubsub_max_message=64M added to dom0_extra_args. zedbox logs pubsub: frames up to 1048576 bytes, messages up to 67108864 bytes. A value such as eve_pubsub_max_message=lots is logged as an error and the default is used.
  • Upgrade: from the previous release, the device comes up, all agents connect, and persisted statuses such as DevicePortConfigList read back.

Changelog notes

EVE services can now exchange pubsub messages far larger than 64 KB: up to 1 MB as is, and larger ones compressed. Large network configurations with many ports, certificates and PAC files no longer make zedbox exit and reboot the device. A value too large even compressed is refused and logged instead. The largest message is 30 MB by default; the eve_pubsub_max_message kernel parameter raises it. EVE no longer writes per-field files under /persist/pubsub-large, and pubsub allocates about two thirds less memory.

PR Backports

  • 17.0-stable: No.
  • 16.0-stable: No.
  • 14.5-stable: No.
  • 13.4-stable: No.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't check them. The PR is tested on amd64 only, with evetest on libvirt; it is not tested on arm64.

@rene

rene commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

@eriknordmark , @milan-zededa , pls, let's take a look closer to this PR since it touches a critical pubsub infrastructure....

@rucoder

rucoder commented Mar 2, 2026 •

Copy link
Copy Markdown
Contributor Author

@rene @eriknordmark @michel-zededa I did not touch "pubsub-large" yet. this is experimental PR. I'm 99% sure it works but let's test

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates pubsub’s IPC transport to remove the historical ~65KB message size ceiling by switching from unixpacket sockets to unix stream sockets and introducing length-prefixed framing using the vendored getlantern/framed library with big frames enabled.

Changes:

  • Switch pubsub socket transport from unixpacket to unix stream sockets.
  • Wrap IPC reads/writes with framed.Reader/framed.Writer using EnableBigFrames().
  • Increase the socketdriver maxsize limit to 10MB and update related tests/comments.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
pkg/pillar/pubsub/socketdriver/subscribe.go Subscriber now dials unix and reads framed messages.
pkg/pillar/pubsub/socketdriver/publish.go Publisher now accepts unix stream conns, reads framed requests, and writes framed updates.
pkg/pillar/pubsub/socketdriver/driver.go Raises maxsize to 10MB; adds framed reader/writer helpers and removes buffer pool.
pkg/pillar/pubsub/reverse/subscribe.go Reverse subscriber now listens on unix and reads framed messages.
pkg/pillar/pubsub/reverse/publish.go Reverse publisher now dials unix and writes framed messages.
pkg/pillar/pubsub/checkmaxsize_test.go Updates test commentary/cases to reflect removal of the old ~65KB boundary behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 409 to 415
s.log.Tracef("sendUpdate(%s): key %s\n", s.name, key)
// base64-encode to avoid having spaces in the key and val
sendKey := base64.StdEncoding.EncodeToString([]byte(key))
sendVal := base64.StdEncoding.EncodeToString(val)
buf := fmt.Sprintf("update %s %s %s", s.topic, sendKey, sendVal)
if len(buf) >= maxsize {
s.log.Fatalf("Too large message (%d bytes) sent to %s topic %s key %s",
len(buf), s.name, s.topic, key)
}
_, err := sock.Write([]byte(buf))
_, err := writer.Write([]byte(buf))
return err

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These send* methods no longer enforce the driver maxsize limit. Since PublicationImpl.Publish() does not call CheckMaxSize(), oversized values can now be sent over IPC, bypassing the intended 10MB cap and potentially causing large allocations/pressure on subscribers. Reintroduce a size check (ideally returning an error rather than Fatal) before writer.Write(), or enforce maxsize at publish/serialize time for all outgoing messages.

Copilot uses AI. Check for mistakes.
Comment on lines +441 to +442
// CheckMaxSize returns an error if too large.
// With framed big frames the limit is ~4GB, so this is unlikely to trigger.

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment on CheckMaxSize is misleading: the framed transport may support ~4GB frames, but this driver still enforces maxsize (currently 10MB) and this function can definitely trigger. Please update the comment to reflect the actual enforced limit and its purpose (protecting memory/IPC).

Suggested change
// CheckMaxSize returns an error if too large.
// With framed big frames the limit is ~4GB, so this is unlikely to trigger.
// CheckMaxSize returns an error if the serialized message exceeds maxsize.
// Even though the framed transport can support ~4GB frames, this driver enforces
// a much smaller application-level limit (currently 10MB) to protect memory/IPC.

Copilot uses AI. Check for mistakes.
Comment on lines 244 to 251
// NewFramedReader wraps a net.Conn with a framed.Reader that uses big frames
// (32-bit length prefix), allowing messages up to ~4GB.
func NewFramedReader(conn net.Conn) *framed.Reader {
r := framed.NewReader(conn)
r.EnableBigFrames()
return r
}

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NewFramedReader/NewFramedWriter enable big frames (~4GB), but the framed.Reader.ReadFrame() API allocates based on the length prefix and does not enforce a maximum on reads (only writes check maxFrameLength). Given this package defines maxsize (10MB), consider centralizing a bounded read helper in socketdriver (e.g., read length, reject > maxsize, then read) so all IPC consumers are protected from oversized/malicious frames.

Suggested change
// NewFramedReader wraps a net.Conn with a framed.Reader that uses big frames
// (32-bit length prefix), allowing messages up to ~4GB.
func NewFramedReader(conn net.Conn) *framed.Reader {
r := framed.NewReader(conn)
r.EnableBigFrames()
return r
}
// NewFramedReader wraps a net.Conn with a framed.Reader.
// Big frames are not enabled to avoid unbounded allocations on reads.
func NewFramedReader(conn net.Conn) *framed.Reader {
r := framed.NewReader(conn)
return r
}

Copilot uses AI. Check for mistakes.
Comment thread pkg/pillar/pubsub/reverse/subscribe.go Outdated
Comment on lines +73 to +84
reader := socketdriver.NewFramedReader(conn)

for {
// wait for readable conn
if err := pubsub.ConnReadCheck(conn); err != nil {
if err != io.EOF {
log.Errorf("serveConnection: Error on connReadCheck: %s",
err)
}
break
}

buf, doneFunc := socketdriver.GetBuffer()
defer doneFunc()

count, err := conn.Read(buf)
frame, err := reader.ReadFrame()
if err != nil {
if err != io.EOF {
log.Errorf("serveConnection: Error on read: %s",
err)
}
break
}
retChan <- string(buf[:count])
retChan <- string(frame)

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serveConnection uses framed.Reader.ReadFrame(), which can allocate arbitrarily large buffers based on the peer-provided length prefix (big frames are enabled). Since this listener accepts local connections, a buggy/malicious publisher can DoS the subscriber via OOM. Use a bounded read (enforce socketdriver maxsize) before allocating/reading the payload.

Copilot uses AI. Check for mistakes.
Comment thread pkg/pillar/pubsub/checkmaxsize_test.go Outdated
Comment on lines +46 to +49
// With framed big frames (32-bit length prefix) the IPC message size
// limit is ~4GB, so the old 65KB boundary tests no longer apply.
// The "large tag" mechanism (pubsub-large-*) still has its own 1MB
// limit enforced by large.go, so those cases remain.

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestCheckMaxSize no longer exercises the driver maxsize boundary after increasing it to 10MB. To avoid regressions, add at least one case that exceeds the new maxsize (e.g., ~11–12MB payload after base64 overhead) and assert that CheckMaxSize returns an error.

Copilot uses AI. Check for mistakes.
Comment on lines +269 to 272
frame, err := s.reader.ReadFrame()
if err != nil {
errStr := fmt.Sprintf("connectAndRead(%s): sock read failed %s",
s.name, err)

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using framed.Reader.ReadFrame() is unsafe here because the vendored framed implementation allocates a buffer based solely on the on-wire length prefix and does not enforce any maximum. With big frames enabled, a malformed or malicious peer can trigger very large allocations (OOM) before any pubsub-level size checks run. Prefer reading the length prefix, rejecting frames > maxsize (10MB), and then reading into a bounded/reusable buffer (e.g., use framed.Reader.Read() with a preallocated maxsize buffer or implement a capped ReadFrame helper).

Copilot uses AI. Check for mistakes.
Comment on lines 290 to 295
// Read request
res, err := conn.Read(buf)
frame, err := reader.ReadFrame()
if err != nil {
// Peer process could have died
s.log.Errorf("serveConnection(%s/%d) error: %v", s.name, instance, err)
return

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serveConnection reads the initial request using framed.Reader.ReadFrame(), which allocates according to the peer-provided length prefix with no upper bound. Even though the request should be small, a bad client can force large allocations and DoS the publisher. Consider reading the request with a fixed-size buffer (since the request is known to be small) or adding a max-frame-size check before allocating/reading the full payload.

Copilot uses AI. Check for mistakes.

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to https://github.com/getlantern/framed/blob/master/framed.go the max length is 65535 so this doesn't change anything but adding silent message truncation for larger messages,

If framed.go did support 4GByte size messages my question would be what the max memory usage would be if we started having pubsub use messages that are in the size of Megabytes. I see you removed the buffer pool which means that memory usage would increase even if the we stay below 65536, which already is a significant concern; it takes a while for the GC to kick in.

So I don't understand what problem we need to fix here - replacing teaching folks about a max size and how to use pubsub 'large' with additional OOM risk doesn't seem like an improvement to me.

@rucoder
rucoder force-pushed the rucoder/pubsub-msg-size branch from 9ace3e8 to ce6c077 Compare March 2, 2026 16:07
@github-actions
github-actions Bot requested a review from eriknordmark March 2, 2026 16:07
@rucoder

rucoder commented Mar 2, 2026 •

Copy link
Copy Markdown
Contributor Author

According to https://github.com/getlantern/framed/blob/master/framed.go the max length is 65535 so this doesn't change anything but adding silent message truncation for larger messages,

no, we use EnableBigFrames to overcome this limitation

If framed.go did support 4GByte size messages my question would be what the max memory usage would be if we started having pubsub use messages that are in the size of Megabytes. I see you removed the buffer pool which means that memory usage would increase even if the we stay below 65536, which already is a significant concern; it takes a while for the GC to kick in.

So I don't understand what problem we need to fix here - replacing teaching folks about a max size and how to use pubsub 'large' with additional OOM risk doesn't seem like an improvement to me.

when we use pubsub large we load message to RAM one way or another, you are right, Framed doesn't increase the size but send a single message as fragments then assemble them back to full packet. the idea was to handle messages that are not annotated as "pubsub-large" but suddenly become >64k in side. This exactly what we had recently if 20 SR-IOV VF were enabled

@eriknordmark since pbsub-large is still in places, only messages that accidentally exceed the sized will benefit. To avoid the concern about GC pressure I'll revert to a buffer allocated on client side. Basically the same behavior for messages <=64k and grow buffer if necessary for bigger messages

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/pillar/pubsub/checkmaxsize_test.go Outdated
Comment on lines +48 to +50
// We add new test cases that exceed the 10MB limit to verify it.
// The "large tag" mechanism (pubsub-large-*) still has its own 1MB
// limit enforced by large.go, so those cases remain.

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says the new test cases “exceed the 10MB limit”, but these cases use 8 * 1024 * 1024. If the intent is to exceed the 10MB maxsize after base64/message overhead, please call that out explicitly (otherwise consider using a size that is unambiguously over the limit).

Suggested change
// We add new test cases that exceed the 10MB limit to verify it.
// The "large tag" mechanism (pubsub-large-*) still has its own 1MB
// limit enforced by large.go, so those cases remain.
// We add new test cases with 8MiB payloads; while the raw payload size
// is below 10MB, base64 encoding and message framing overhead push the
// resulting IPC message size over the effective 10MB maxsize we verify.

Copilot uses AI. Check for mistakes.
Comment on lines 245 to 266
// NewFramedReader wraps a net.Conn with a framed.Reader that uses big frames
// (32-bit length prefix). Callers must use readFrame() instead of calling
// ReadFrame() directly, to enforce the maxsize limit and prevent unbounded
// memory allocation from a buggy peer.
func NewFramedReader(conn net.Conn) *framed.Reader {
r := framed.NewReader(conn)
r.EnableBigFrames()
return r
}

// logs a message if the allocation changed
var lastLoggedAllocated uint32

func maybeLogAllocated(log *base.LogObject) {
currentAllocated := atomic.LoadUint32(&allocated)
currentLastAllocated := atomic.LoadUint32(&lastLoggedAllocated)
if currentLastAllocated == currentAllocated {
return
// ReadFrame reads a single frame from the framed.Reader and rejects frames
// that exceed maxsize. This protects against unbounded memory allocation
// from a buggy or malicious peer sending a large length prefix.
func ReadFrame(r *framed.Reader) ([]byte, error) {
frame, err := r.ReadFrame()
if err != nil {
return nil, err
}
if len(frame) > maxsize {
return nil, fmt.Errorf("received frame of %d bytes exceeds max %d",
len(frame), maxsize)
}

Copilot AI Mar 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReadFrame() currently calls r.ReadFrame() and only then checks len(frame) > maxsize. In getlantern/framed, Reader.ReadFrame() allocates make([]byte, n) based on the length prefix before returning, so this code does not prevent unbounded memory allocation / OOM from a large length prefix. To actually enforce maxsize, read and validate the length prefix first (before allocating) or use a bounded read strategy and close/drain the connection on oversize frames. Also, the doc comment above NewFramedReader refers to readFrame() but the exported helper is ReadFrame().

Copilot uses AI. Check for mistakes.
@codecov

codecov Bot commented Mar 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.27%. Comparing base (2727419) to head (c8e1786).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5644      +/-   ##
==========================================
+ Coverage   28.76%   29.27%   +0.51%     
==========================================
  Files         544      558      +14     
  Lines       98795   100610    +1815     
==========================================
+ Hits        28421    29457    +1036     
- Misses      68086    68693     +607     
- Partials     2288     2460     +172     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rucoder

rucoder commented Mar 2, 2026

Copy link
Copy Markdown
Contributor Author

@eriknordmark @rene it would be interesting to kick eden tests to see how it works

@rucoder
rucoder marked this pull request as draft March 2, 2026 17:45
@eriknordmark

Copy link
Copy Markdown
Contributor

when we use pubsub large we load message to RAM one way or another, you are right, Framed doesn't increase the size but send a single message as fragments then assemble them back to full packet. the idea was to handle messages that are not annotated as "pubsub-large" but suddenly become >64k in side. This exactly what we had recently if 20 SR-IOV VF were enabled

Actually we don't know how many copies of that 10MByte we will end up in memory when sending and receiving it.
I see at least one additional copy (where it gets sent over a channel).

@eriknordmark since pbsub-large is still in places, only messages that accidentally exceed the sized will benefit. To avoid the concern about GC pressure I'll revert to a buffer allocated on client side. Basically the same behavior for messages <=64k and grow buffer if necessary for bigger messages

I think it would be a bad idea to replace the annoying but quickly detected log.Fatal when trying a new configuration (such as the above more VF case) with an increased risk that devices will hit an OOM in production based on unpredictable load conditions.

@rucoder
rucoder force-pushed the rucoder/pubsub-msg-size branch 4 times, most recently from 08dc2be to 3ddf8a5 Compare March 6, 2026 10:57
@github-actions
github-actions Bot requested a review from jsfakian March 6, 2026 10:57
@rucoder
rucoder force-pushed the rucoder/pubsub-msg-size branch 3 times, most recently from 29d0798 to 9a379a1 Compare March 6, 2026 11:23
pkg/newlog, pkg/edgeview, pkg/wwan/mmagent and pkg/kube/kube-init talk
pubsub to pillar through a vendored copy of pkg/pillar/pubsub. A copy
still using unixpacket sockets cannot connect to the stream sockets pillar
listens on now, so move all of them to the previous commit.

The other dependency updates are the versions that pillar requires.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Temporary, for review only: this is how the memory numbers in the PR
description were measured. It is dropped before merge.

Every allocation pubsub makes for a message is counted at its call site:
JSON of published items, wire messages and their compressed frames, read
buffers including decompressed messages, decoding and copies of received
messages, DeepCopy. Each kind reports the count, the
bytes asked for and the bytes the Go allocator reserves (the request
rounded up to its size class) in /debug/pubsub/stats. The sites follow
the compiler's escape analysis (go build -gcflags=-m), so conversions Go
keeps on the stack are not counted.

Not counted: allocations inside encoding/json while decoding, the
internal buffers of the zstd encoder and decoder, and the boxing of log
arguments.

Master was measured with the same counters at its own call sites,
including the pubsub-large files and the 64 KiB read buffer pool:
branch rucoder/pubsub-alloc-counters-master.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
@rucoder
rucoder force-pushed the rucoder/pubsub-msg-size branch from 1ad443d to c8e1786 Compare November 3, 2026 09:11
@github-actions
github-actions Bot requested a review from eriknordmark November 3, 2026 09:26
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@milan-zededa Are the evetest failures something we expect at the current state of the evetest CI code?

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

evetest: skipped tests for this PR

This PR is confined entirely to the pubsub transport (pkg/pillar/pubsub/**, ipcmonitor, the /debug/pubsub/stats endpoint, onboot.sh's /persist/pubsub-large cleanup, and the vendored-pillar bumps in newlog, edgeview, kube-init and wwan/mmagent needed to keep those peers on the same wire framing). No agent business logic changed. The change is already covered by extensive new unit tests in pkg/pillar/pubsub (framing, size limits, compression, buffer pools, pubsub-large inlining, stats, the Stop() race).

Because pubsub is the IPC backbone for essentially every agent, almost every evetest suite is at least incidentally relevant, so very little was skipped. The suites/components most directly exercising this change were kept, in particular:

  • TestBootstrapWithProxy and TestPNAC (CA certs / PAC data — the pubsub-large ProxyCertPEM field and the large-DevicePortConfig scenario this PR is specifically about)
  • TestAppUserData (CloudInitUserData, another pubsub-large field)
  • TestControllerEncryptCertChange / TestControllerSigningCertChange (CipherData)
  • TestKonvertSuite, TestKubeRegistrationSuite, TestNodeClusterSuite/TestNodeClusterResilienceSuite (exercise kube-init, one of the four vendored-pillar consumers updated here, and the pubsub-large-tagged kube types)
  • TestCellularConnectivity / TestWifiConnectivity (exercise mmagent, another updated vendored consumer)
  • TestDeviceLogs and friends (exercise newlogd, another updated vendored consumer)

Skipped (redundant onboarding duplicates):

  • TestBootstrapWithStaticIP (both variants)
  • TestBootstrapWithNetworkBoot

These two just onboard a device over a plain static-IP or network-boot path with no unusually large or pubsub-large-tagged payload — the same onboarding/config-distribution path is already covered, with normal-to-larger configs, by TestBootstrapWithLastResort (kept as the baseline) and by TestBootstrapWithProxy/TestBootstrapWithMgmtVLAN/TestBootstrapWithLACPBond (kept, since they add port-config complexity relevant to larger DevicePortConfig/DeviceNetworkStatus messages).

An engineer can comment /evetest all to force the full suite to run instead, or /evetest <name> <name> ... to run only specific tests/suites.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as off-topic.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestAppsSuite: failure analysis

TestWorkerPoolSaturation

Failure
Device "edge-dev" failed to confirm the latest config (version 55): timed out waiting for LastProcessedConfig >= 2026-11-04 04:23:09.77633744 +0000 UTC
Claude's conclusion
Root cause

The test timed out waiting for config v55 to be confirmed because zedbox (pillar) crashed and the watchdog rebooted the device, not because of a hang. The live console log shows the device applying config v55 at 04:23:09–04:23:11 (6 apps' content-trees/volumes transitioning INVALID → RESOLVING_TAG, consistent with TestWorkerPoolSaturation setting volumemgr.worker.pool.size=1 and submitting 6 pool apps at once), then at 04:24:13 the console prints:

Rebooting EVE. Reason: panic: send on closed channel
Watchdog report for IMGA EVE version 0.0.0-pr5644-c8e1786e-kvm-amd64 at 2026-11-04T04:24:13+00:00: 3 /run/zedbox.pid

zedbox panicked with panic: send on closed channel (a goroutine writing to an already-closed channel — plausibly in the worker-pool dispatch path the test is specifically exercising at pool size 1) and the watchdog force-rebooted the device, which is why zedagent never reported LastProcessedConfig for v55 and why my own evetest eve ssh/direct connection attempts were refused mid-investigation — the device was actively re-booting (console shows a fresh containerd/zedbox boot sequence starting at 04:25:59, coming back up right as I checked).

This doesn't match any entry in live-ledger.md (no prior send on closed channel or TestWorkerPoolSaturation occurrences), so it's a new failure tonight, not a previously-known recurring issue. It's also unrelated to any of the other failure signatures already catalogued in the ledger (downloader tag-resolution hangs, missed pubsub republish races, kvm_async_pf hung guests) — this is a distinct, concrete pillar panic triggered specifically by the worker-pool-saturation scenario. No PR diff was provided for this run to judge relatedness to in-flight code changes.

TestVMAppPurgeReplacesVMIRS

Failure
Device "edge-dev": expected 0 reboot(s) but observed 1
Claude's conclusion

This confirms it directly: NodeAgentStatus on edge-dev shows RebootReason:"panic: send on closed channel..." at 2026-11-04T04:24:13Z, with the stack trace pointing at pkg/pillar/cmd/downloader/download.go:473 (objectMetadata.func1) — the exact same panic/reboot already diagnosed for the preceding TestWorkerPoolSaturation failure in this run.

Root cause

This is the same root cause as the already-investigated TestWorkerPoolSaturation failure in this run: zedbox panicked with panic: send on closed channel in downloader.objectMetadata (pkg/pillar/cmd/downloader/download.go:473), which caused the watchdog to force-reboot edge-dev at 04:24:13. TestVMAppPurgeReplacesVMIRS reused the same device right afterward — its gotest.json output shows the test body itself immediately skipped (HYPERVISOR is kvm: this test asserts on VMIRS objects and PVCs, which only exist on kubevirt), and the reported failure is purely the harness's teardown checkRebootCounts (devstate.go:407) catching the leftover reboot from the prior test's crash — not a bug specific to this test or to VMIRS/purge logic at all. NodeAgentStatus confirms RestartCounter:2 and the matching panic stack trace, directly corroborating the earlier finding. This exact panic signature doesn't appear anywhere in live-ledger.md, so it's not a previously-known/recurring issue — it's new tonight, and both failures in this run trace back to the single TestWorkerPoolSaturation panic.

TestVGAPassthroughNoHostAccess

Failure
Host processes interfered with PCI devices while the VGA controller 0000:00:01.0 was passed through to the application:
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=11165 parent=runc:[1:CHILD]/11164 grandparent=runc:[0:PARENT]/11163
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=11165 parent=runc:[1:CHILD]/11164 grandparent=runc:[0:PARENT]/11163
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=11165 parent=runc:[1:CHILD]/11164 grandparent=runc:[0:PARENT]/11163
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=11165 parent=runc:[1:CHILD]/11164 grandparent=runc:[0:PARENT]/11163
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=11165 parent=runc:[1:CHILD]/11164 grandparent=runc:[0:PARENT]/11163
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=11165 parent=runc:[1:CHILD]/11164 grandparent=runc:[0:PARENT]/11163
Kernel messages mentioning 0000:00:01.0:
[    1.449695] [    T1] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint
[    1.454119] [    T1] pci 0000:00:01.0: BAR 0 [mem 0x80000000-0x80ffffff pref]
[    1.454828] [    T1] pci 0000:00:01.0: BAR 2 [mem 0x8264d000-0x8264dfff]
[    1.455512] [    T1] pci 0000:00:01.0: ROM [mem 0xffff0000-0xffffffff pref]
[    1.456278] [    T1] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.777347] [    T1] pci 0000:00:01.0: vgaarb: setting as boot VGA device
[    1.777347] [    T1] pci 0000:00:01.0: vgaarb: bridge control possible
[    1.777914] [    T1] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.967532] [    T1] pci 0000:00:01.0: Adding to iommu group 1
[  156.928268] [ T2675] vfio-pci 0000:00:01.0: vgaarb: deactivate vga console
[  156.928276] [ T2675] vfio-pci 0000:00:01.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  191.960660] [ T7666] vfio-pci 0000:00:01.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[  192.267151] [ T7666] vfio-pci 0000:00:01.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  590.432156] [T11188] vfio-pci 0000:00:01.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
Claude's conclusion

Confirmed — PID 11165 is vga-passthrough-app's own qemu process (-name 585b7279-...1.1), and TID 11188 within it is the kernel-renamed vCPU thread CPU 0/KVM, exactly the thread flagged as "host interference."

Root cause: This is the same long-standing, well-known test-harness bug recorded repeatedly in live-ledger.md (first seen 2026-10-29, recurring on 2026-10-30, 2026-10-31/11-01, and again here tonight). pciAccess.isViolation() (evetest/tests/apps/pciaccess_helpers_test.go:408-413) only whitelists PCI config-space writes when comm has prefix "qemu-system-", but QEMU's vCPU thread renames itself to CPU N/KVM while sharing the owning process's PID — so the config-space writes made by the app's own vCPU thread during its [app-restart] re-initialization get misclassified as external host interference. Live evidence here confirms the exact same mechanism: PID 11165 is vga-passthrough-app's own qemu process (domain 585b7279-...1.1), and TID 11188 in that process is literally CPU 0/KVM, matching the flagged comm. This is unrelated to the panic/reboot root cause already identified earlier in this run (TestWorkerPoolSaturation/TestVMAppPurgeReplacesVMIRS's panic: send on closed channel in downloader.objectMetadata) — it's a distinct, pre-existing test-infra defect, not a product regression, and has never been fixed despite recurring on nearly every run since late September.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as off-topic.

@github-actions

This comment was marked as outdated.

@milan-zededa

milan-zededa commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

LGTM.

@milan-zededa Are the evetest failures something we expect at the current state of the evetest CI code?

@eriknordmark There is still lots of noise (failures not related to actual EVE bugs), but it should get better once these are merged (and PRs on master are rebased):

@eriknordmark
eriknordmark merged commit 394c0a7 into lf-edge:master Nov 4, 2026
68 of 79 checks passed
rucoder added a commit to rucoder/eve that referenced this pull request Nov 5, 2026
The counters of pubsub's own allocations came in with the last commit
of lf-edge#5644, which was meant to be dropped before merge. They stay, so make
them exact and keep them so.

PublicationStats and SubscriptionStats passed a new PubStats or SubStats
to LoadOrStore on every call, which allocated it each time: four
allocations per published message spent on counting. Look the name up
first; a new one is stored only the first time.

Tests now hold the counters against the runtime's own count of
allocations, and they found sites counted wrongly: []byte(key) passed to
AppendEncode or EncodeToString is not copied, a buffer kept as a *[]byte
moves its slice header to the heap, and the string of the persistence
flag in hello escapes. TestAllocCountsSend and TestAllocCountsReceive
compare the counters with testing.AllocsPerRun, and
TestAllocSizeMatchesRuntime compares the size classes with
MemStats.TotalAlloc, so a Go upgrade that changes what is allocated
fails a test instead of making the counters drift. Both pass on Go
1.25 and 1.26. Under the race detector, which CI runs the tests with,
the compiler does not optimise append(s, make([]T, n)...), so base64
decoding of a key above 32 bytes allocates twice: the receive test is
skipped there, the send test runs.

The pubsub-large counters went with pubsub-large; drop them. DeepCopy
does not know the topic of what it copies, so report its counters in a
"process" section of /debug/pubsub/stats instead of as a publication
named "(DeepCopy)".

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
rene pushed a commit that referenced this pull request Nov 5, 2026
The counters of pubsub's own allocations came in with the last commit
of #5644, which was meant to be dropped before merge. They stay, so make
them exact and keep them so.

PublicationStats and SubscriptionStats passed a new PubStats or SubStats
to LoadOrStore on every call, which allocated it each time: four
allocations per published message spent on counting. Look the name up
first; a new one is stored only the first time.

Tests now hold the counters against the runtime's own count of
allocations, and they found sites counted wrongly: []byte(key) passed to
AppendEncode or EncodeToString is not copied, a buffer kept as a *[]byte
moves its slice header to the heap, and the string of the persistence
flag in hello escapes. TestAllocCountsSend and TestAllocCountsReceive
compare the counters with testing.AllocsPerRun, and
TestAllocSizeMatchesRuntime compares the size classes with
MemStats.TotalAlloc, so a Go upgrade that changes what is allocated
fails a test instead of making the counters drift. Both pass on Go
1.25 and 1.26. Under the race detector, which CI runs the tests with,
the compiler does not optimise append(s, make([]T, n)...), so base64
decoding of a key above 32 bytes allocates twice: the receive test is
skipped there, the send test runs.

The pubsub-large counters went with pubsub-large; drop them. DeepCopy
does not know the topic of what it copies, so report its counters in a
"process" section of /debug/pubsub/stats instead of as a publication
named "(DeepCopy)".

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
eriknordmark pushed a commit that referenced this pull request Nov 14, 2026
`gh pr diff` fails with HTTP 406 for a PR touching more than 300 files,
which a vendor bump easily exceeds (e.g. PR #5644). discover-suites
then failed before any suite was scheduled, so evetest never ran for
such PRs even though "Ready for evetest" was reported.

Check out the PR head from the upstream repo with actions/checkout,
blobless and with the full history of every branch (about 20 MB), and
diff it against its merge-base with the base branch. Vendored code is
left out of the diff; instead, the vendored modules added, removed or
bumped are read from vendor/modules.txt and listed in pr-context.txt.

If fetching or diffing fails, Claude is not asked for a skip decision
and the run falls back to the full suite instead of failing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit e6f1a74)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants