Skip to content

pubsub: marshal and compress once for all subscribers - #7037

Merged
rene merged 9 commits into
lf-edge:masterfrom
rucoder:rucoder/pubsub-cache
Nov 5, 2026
Merged

rene merged 9 commits into
lf-edge:masterfrom
rucoder:rucoder/pubsub-cache

Conversation

@rucoder

@rucoder rucoder commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #5644. That PR was merged together with its last commit, the allocation counters marked "DO NOT MERGE". This PR keeps the counters and makes them exact. It then does the follow-up planned in #5644: pubsub stops redoing the same work for every subscriber.

Before this PR, on every change of any key, each subscriber connection of a publication:

  • deep-copied every item of the publication, a marshal and an unmarshal each;
  • marshalled every copy again and compared it with what it had sent last, finding each key with a linear search of its map;
  • kept its own copy of the JSON of every key;
  • copied the value into a new message for every send;
  • compressed a message above 1 MB again, although Publish had already compressed it to check its size and thrown the result away.

So one changed key of a publication with N keys and S subscribers cost 2·N·S marshals and N·S unmarshals. With this PR, Publish marshals once, every connection sends the same bytes, and a large message is compressed once.

Commits

  1. pubsub: correct the allocation counters and test them
    • PublicationStats/SubscriptionStats allocated a new stats struct on every call, four allocations per published message spent on counting. They now look the name up first.
    • Guard tests hold the counters against the runtime's exact count of allocations (testing.AllocsPerRun, MemStats.TotalAlloc). They found three sites counted wrongly, all now fixed.
    • Go 1.25 and 1.26 pass. Under -race, which CI uses, the compiler allocates differently, so only the receive guard test is skipped there.
    • The pubsub-large counters go. DeepCopy moves to a process section of /debug/pubsub/stats.
  2. pubsub: count the frames a subscriber rejects: rejected_frames, next to the publisher's refused.
  3. pubsub: marshal a published value once and share it
    • Publish keeps the JSON next to the item in an immutable Value.
    • DetermineDiffs hands every connection the current Value of each key. A key is sent again only if its Value is not the one sent last, so nothing is marshalled or compared per connection, and all connections share the bytes.
    • An update is written as its header plus the shared JSON in one writev, without a copy.
    • An item whose JSON equals the published JSON is unchanged at once. Only different JSON goes on to cmp.Equal, as before, so the types' Equal methods still decide.
    • Get/GetAll decode their copies from the JSON (no marshal). A subscription handler gets its copy decoded from the received bytes, and only if the handler is set.
  4. pubsub: compress a large value once for all its subscribers
    • CheckMaxSize returns the compressed message it builds to check the size. Publish keeps it in the Value, and every connection sends that frame.
    • New counter: compressions.
  5. pubsub: compare byte slices with bytes.Equal in cmp.Equal
    • cmp.Equal compares a []byte byte by byte through reflection, with an edit script per slice, which is quadratic where they differ. The network types carry certificates as [][]byte in every port.
    • For a 1.3 MB DevicePortConfigList, cmp.Equal took 1.8 s with nothing changed and 2 min 24 s with the certificates replaced, in the publisher and in every subscriber. With bytes.Equal for []byte it takes 5.9 ms and 78 ms.
    • A nil slice still differs from an empty one, and a test checks that the results match plain cmp.Equal.
  6. pubsub: do not decode what a persistent subscription has loaded
    • At start, a persistent subscription loads its checkpoint, and the publisher then resends every key, mostly the same JSON. Until the subscription is synchronized, it keeps the SHA-256 of each item's JSON and drops the same JSON without decoding it.
    • Once synchronized, the publisher sends only changes, which are decoded and compared as before. The sums are dropped, so nothing is hashed or kept after start.
    • The same JSON decodes to the same item, so handler calls are exactly as before.
    • New counter: unchanged.
  7. evetest infra: crash watch and PAC proxy config, by Christoph Ostarek.
  8. evetest test: objects that outgrow a pubsub message, by Christoph Ostarek: TestLimitsSuite, six configurations or workloads whose objects outgrew the old 64 KB message. All six crashed EVE before pubsub: send messages as frames over unix stream sockets #5644. Here they become regression tests.
  9. evetest: run TestLimitsSuite on the full runners only. The suite takes 31 minutes, 20 of them deploying 15 applications, so it is tagged heavy like the other suites that deploy many applications.

The vendored pillar in pkg/newlog, pkg/edgeview, pkg/wwan/mmagent and pkg/kube/kube-init is not bumped here. A separate PR bumps it once this one is merged, so that the modules pin a commit on master.

Measured

A test process publishes real pillar types over the socket driver to S subscriptions on tmpfs. The "master" column is master with commits 1–2, so both columns are counted the same way. The figures are the runtime's exact counts for the whole process, MemStats.Mallocs and TotalAlloc, which include encoding/json. Nothing is sampled.

per operation master this PR
1 of 50 AppInstanceStatus (4.2 KB) changes, 1 subscriber 9,803 allocs, 1.48 MB, 11.5 ms 4,466 allocs, 0.46 MB, 3.2 ms
same, 5 subscribers 39,660 allocs, 6.51 MB, 14.1 ms 13,415 allocs, 1.39 MB, 4.5 ms
same, 10 subscribers 77,033 allocs, 12.82 MB, 17.7 ms 24,606 allocs, 2.56 MB, 5.5 ms
Publish of an unchanged AppInstanceStatus 439 allocs, 51 KB, 0.53 ms 63 allocs, 12 KB, 0.03 ms
Get 64 allocs, 12.8 KB 30 allocs, 6.5 KB
GetAll of 50 3,069 allocs, 641 KB 1,370 allocs, 323 KB
persistent subscription start, 50 AppInstanceStatus 35,890 allocs, 5.26 MB, 46 ms 11,013 allocs, 1.88 MB, 14 ms
persistent subscription start, DevicePortConfigList of 3.4 MB 112,159 allocs, 326 MB, 4.6 s 2,442 allocs, 14.0 MB, 77 ms

For a change of a 3.4 MB DevicePortConfigList, pubsub's own counters show:

subscribers master: pubsub allocates this PR compressions per change (master / PR)
1 27.2 MB 10.2 MB 2 / 1
3 61.2 MB 17.0 MB 4 / 1
5 95.2 MB 23.9 MB 6 / 1

What remains is one marshal, one copy of the message as compression input, and the message each subscriber decompresses.

The whole-process time of that change is dominated by something outside pubsub. DevicePortConfigList.LogModify builds cmp.Diff(old, new) for a Trace-level log, so it is evaluated even with tracing off. For the two lists in this test, that one cmp.Diff takes 9.6 s and 686 MB in 1.08 million allocations to produce a 4.5 KB diff. It runs once in the publisher and once in every subscriber. 57 LogModify methods in pkg/pillar/types call cmp.Diff, and some of them likewise discard the result. That is left for a separate PR.

Measured on a device

Christoph Ostarek's limits tests ran on amd64 (evetest, libvirt). They use the evetest memory probe, which reads pubsub's counters from /debug/pubsub/stats; the probe is not part of this PR. The "master" build is master with commits 1–2, so both builds count the same way. All runs passed on both builds.

TestHugeConfigItem, 25 SSH keys, 3 runs per build, alternating master this PR
bytes pubsub allocates, zedbox start to test end 21.6 MiB 4.9 MiB (−77%)
allocations, same period 60,215 30,774 (−49%)
bytes while the property is applied 8.2 MiB 1.6 MiB (−80%)
of which DeepCopy, whole run 8.9 MiB 0
TestLimitsSuite, bytes pubsub allocates while each object is applied master this PR
TestManyStaticDNSEntries 12.4 MiB 2.4 MiB (−81%)
TestManyPhysicalIOAdapters 7.1 MiB 1.2 MiB (−82%)
TestHugeConfigItem 7.0 MiB 1.9 MiB (−73%)
TestLargePACFile 28.4 MiB 6.3 MiB (−78%)
TestManyVolumesPerApp 1,011 MiB 130 MiB (−87%)
TestManyAppInterfaces 772 MiB 65 MiB (−92%)

The two large cases are the per-subscriber cost commit 3 removes. Every change of any of the many keys made every subscriber connection deep-copy and re-marshal all of them.

Compatibility

The wire protocol, checkpoints and the Publication/Subscription API are unchanged. Handlers are called exactly when they were before. Publishers skip unchanged items as before: identical JSON now, otherwise cmp.Equal with the types' Equal methods.

Two internal driver interfaces change; only the drivers in pkg/pillar/pubsub implement them:

  • DriverPublisher.CheckMaxSize also returns the prepared frame.
  • LocalCollection holds *Value.

How to test and validate this PR

  • Unit tests: make -C pkg/pillar test. New tests:
    • TestDetermineDiffsSharesValues: every connection gets the same Value, a key is sent again only after a change, and deleted keys come first.
    • TestPublicationCopies, TestHandlerGetsCopy: Get/GetAll and handlers get copies that callers can change.
    • TestPublishHonoursEqual: an item equal by its type's Equal method is still unchanged.
    • TestCompressedOnce: one compression for three subscribers, per change.
    • TestEqualOptsSameResult, TestEqualOptsFast: the comparer changes no result and is fast on 1 MB slices.
    • TestIdenticalNotDecoded: while a persistent subscription starts, the same JSON is neither decoded nor handed to a handler; once it is synchronized, JSON is decoded and compared as before; equal items never reach handlers.
    • TestAllocCountsSend, TestAllocCountsReceive, TestAllocSizeMatchesRuntime: the counters against the runtime.
    • TestRejectedFrameCounted.
  • On a device: boot and onboard, then read curl -s http://127.1:6543/debug/pubsub/stats (after eve http-debug).
    • compressions is at most one per change of a large value.
    • unchanged counts the keys resent to persistent subscriptions at start.
    • The process.DeepCopy counters stay at 0: pubsub no longer deep-copies.
  • evetest: TestLimitsSuite (commits 7–8). It passed 6/6 on this PR, as on master.
  • Upgrade: checkpoints and the wire protocol are unchanged.

Changelog notes

No user-facing changes. Pubsub encodes and compresses each published value once for all subscribers, and compares large network configurations far faster, so EVE services use less memory and CPU.

PR Backports

  • 17.0-stable: No.
  • 16.0-stable: No.
  • 14.5-stable: No.
  • 13.4-stable: No.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't check them. Tested on amd64 with evetest on libvirt; arm64 is not tested.

@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.84530% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.59%. Comparing base (6758baf) to head (3c2eea6).
⚠️ Report is 13 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/pubsub/publish.go 85.91% 5 Missing and 5 partials ⚠️
pkg/pillar/pubsub/socketdriver/publish.go 83.67% 7 Missing and 1 partial ⚠️
pkg/pillar/pubsub/emptydriver.go 0.00% 2 Missing ⚠️
pkg/pillar/pubsub/subscribe.go 92.85% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7037      +/-   ##
==========================================
+ Coverage   29.06%   29.59%   +0.52%     
==========================================
  Files         548      559      +11     
  Lines       98971   100628    +1657     
==========================================
+ Hits        28763    29777    +1014     
- Misses      67941    68624     +683     
+ Partials     2267     2227      -40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene

rene commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 4m 18s —— View job


Review of PR #7037 — pubsub: marshal and compress once for all subscribers

  • Read the core pubsub changes (publish.go, subscribe.go, socketdriver)
  • Review the new Value/sharing logic and frame handling
  • Review the equal/compress/sha changes
  • Post review feedback

I reviewed the non-vendor pubsub changes (publish.go, subscribe.go, driver.go, stats.go, util.go, socketdriver/{publish,frame,subscribe}.go, limits.go). No blocking issues found. The design is sound and I verified the correctness-sensitive paths:

  • Shared Value + pointer-identity diffing (DetermineDiffs): each Publish that changes a key allocates a fresh *Value, and the deleted-key detection (len(localCollection) > current) is correct because the collection always ends ⊇ current keys, so extra entries are exactly the deletions. The unchanged fast-path keeps the same pointer, so identical re-publishes don't resend — correct.
  • bytes.Equal(old.value.JSON, b) fast path then cmp.Equal(…, equalOpts) fallback preserves the "skip if equal by the type's Equal method" semantics, including keeping the old JSON/checkpoint when equal-but-different-JSON. Matches prior behavior.
  • equalOpts []byte comparer: (a==nil)==(b==nil) && bytes.Equal(a,b) reproduces default cmp results (nil vs empty still differ) while avoiding the quadratic edit-script path. Only exact []byte matches; named types fall back to reflection (no correctness change).
  • Pre-compressed Wire: maxFrameSize()/maxMessageSize() are fixed per-process (kernel cmdline, sync.Once), so a frame compressed once at Publish time stays valid for every later subscriber send — no runtime-limit-change hazard.
  • Subscriber sums (SHA-256) skip: the same JSON yields the same decoded item, so a sum-match implies cmp.Equal would have been true → no handler call is skipped that would otherwise fire. handleDelete clears sums[key], keeping sums and km.key consistent so delete+re-add still triggers CreateHandler.
  • Concurrency: shared *Value/JSON/Wire are immutable after store and only read concurrently by per-connection serveConnection goroutines; net.Buffers.WriteTo only mutates the local bufs slice. sums is mutated only on the serial populate/ProcessChange path.

One minor, non-blocking note posted inline on subscribe.go about the per-change sha256 cost on the genuinely-changed path (added overhead there, win only on unchanged/reconnect/populate) — acceptable given the measured improvements.

Vendor copies were not reviewed per instructions.
· branch rucoder/pubsub-cache

Comment thread pkg/pillar/pubsub/subscribe.go Outdated
sub := ctxArg.(*SubscriptionImpl)
name := sub.nameString()
sub.log.Tracef("pubsub.handleModify(%s) key %s\n", name, key)
sum := sha256.Sum256(itemcb)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor (non-blocking): sha256.Sum256(itemcb) now runs on every handleModify, including the genuinely-changed case, where you still fall through to parseTemplate + cmp.Equal. So for a large object that actually changes, the hash is pure added work on top of the decode it was meant to avoid. The win only materializes on the unchanged/reconnect/populate paths. Given the measured net improvements this is clearly an acceptable trade-off — just noting the changed-path cost for the record.

The delete path correctly clears sums[key], so a delete+re-add with identical JSON still fires CreateHandler — good.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and it was more than noise: on my dev host SHA-256 runs at about 300 MB/s, which is 15% of one decode of an AppInstanceStatus and 45% of one decode of a 3.4 MB DevicePortConfigList, paid by every real change.

So the commit now hashes only where it pays: a persistent subscription, until it is synchronized. That covers loading the checkpoint and the publisher's resend at start. At sync the sums are dropped. After that, the publisher sends only changed JSON, which is decoded and compared exactly as before, with nothing hashed or kept. A reconnect's resend is decoded and compared as on master.

The start-up gain is unchanged. Starting a persistent subscription to 50 AppInstanceStatus takes 11,016 allocations of 1.9 MB (27,670 of 3.7 MB before), and to a 3.4 MB DevicePortConfigList 2,450 of 14.0 MB (20,080 of 18.7 MB). TestIdenticalNotDecoded now covers both phases.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestControllerFaultsSuite: failure analysis

TestDeferredQueueDropsReported

Failure

Timed out after 480.001s.
When passed a matcher, ReceiveMatcher's channel *must* receive something.
Claude's conclusion
Root cause

Live inspection shows the app (fa138ac7…) genuinely reached HALTED locally: zedmanager logs Set State from DomainStatus from 118 to 119 at 09:19:35.303, and evetest eve ssh confirms AppInstanceStatus shows State:119 (HALTED). zedagent correctly noticed this and built/queued a ZInfoApp{state:HALTED} report at 09:19:35.319, right as the test called ClearControllerFaults(). But the very next deferred-queue send attempt (09:19:35.334→.341) still got a 404 from the fault proxy — a brief race between the test clearing the fault and the retry firing — and since 404 is treated as a non-retriable status in controllerconn/deferred.go, that message was immediately and permanently dropped (dropping message, controller responded 404 Not Found), never retried. Because AppInstanceStatus doesn't change again afterward, nothing ever re-triggers another PublishAppInfoToZedCloud for that app, so the controller is stuck seeing the pre-halt RUNNING ZInfoApp forever (confirmed live via evetest eve app-info, still state:RUNNING 12+ minutes later) — hence settleApp's Eventually(...).Should(Receive(...HALTED...)) timing out at 480s.

This is a new failure, not in live-ledger.md (no prior entry for TestDeferredQueueDropsReported or this suite's failure signature) — first occurrence of this race.

Regarding PR #7037: its actual diff is a pubsub internals rewrite (pkg/pillar/pubsub/{publish,subscribe,util}.go) plus unrelated evetest changes (evetest/devconfig.go, evetest/edgedevice.go, tests/apps/layer_download_errors_test.go, tests/limits/*). It does not touch controllerconn/deferred.go, zedagent, zedmanager, or evetest/tests/controllerfaults/* — and the pubsub notification path for this app's status update worked correctly end-to-end (zedmanager published, zedagent received and acted on it). So this looks like a pre-existing race/design gap between the test's ClearControllerFaults() timing and the deferred-queue's "404 = permanent drop, no resend" policy, independent of this PR's changes.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterResilienceSuite: failure analysis

TestVMAppPurgeDuringFailover

Failure

Timed out after 900.000s.
The function passed to Eventually failed at /evetest/tests/cluster/purge_during_failover_test.go:169 with:
expected exactly one VMIRS for the app, found []
Expected
    <[]string | len:0, cap:0>: nil
to have length 1
Claude's conclusion
Root cause

Live evidence confirms the exact mechanism already recorded in live-ledger.md for this same run (TestNodeClusterResilienceSuite, TestVMAppPurgeDuringFailover): edge-dev1 (the app's DNID node) is EVE_DEVICE_STATE_SUSPECT/NotReady (deliberately powered off), kubectl get vmirs/vmi -A on edge-dev2 returns nothing at all, and AppInstanceStatus for failover-purge-app (uuid 43db3656-...) shows PurgeInprogress:4, PurgeStartedAt:2026-11-05T10:09:02Z, Activated:false, IsDesignatedNodeID:false. zedmanager on edge-dev2 is right now looping getKubeAppActivateStatus(43db3656-...): not designated, node 92d78f5e-..., onTheDevice false, statusRunning false every few seconds and suppressing publishAppInstanceStatus — i.e. once the purge tore down the old VMIRS, nothing can create a new one because no surviving node considers itself eligible to act for the downed designated node, matching the test's found [].

Per live-ledger.md, this is the same known/recurring bug that has reproduced in every nightly run since 2026-10-29 (run #1) through last night (2026-11-04, run #7) — tonight is at least the 8th consecutive occurrence, not a new issue. It does not share a root cause with the other failure investigated in this run (TestDNIDandBackupDNID's app-op-leader-flag race), which is a distinct mechanism.

Regarding PR #7037: its code changes are confined to pkg/pillar/pubsub (a decode-avoidance/hashing optimization in subscribe.go/publish.go/socketdriver) plus unrelated evetest/tests/limits additions and docs — nothing touches zedmanager, zedkube, domainmgr, or the DNID/backup-DNID logic responsible for this deadlock. Combined with this exact failure having recurred nightly for a week under many different PRs, this looks pre-existing/environmental to the test's known zedmanager gap rather than caused by this PR.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rucoder , you cannot bump pillar in the same PR. Master is broken because of that. Please, remove the bump commit from your PR, we will do the following:

  1. Merge my PR to fix master (#7041)
  2. Merge this PR without bumping pillar
  3. You open a new PR bumping pillar, we merge, master becomes in sync

The counters of pubsub's own allocations came in with the last commit
of lf-edge#5644, which was meant to be dropped before merge. They stay, so make
them exact and keep them so.

PublicationStats and SubscriptionStats passed a new PubStats or SubStats
to LoadOrStore on every call, which allocated it each time: four
allocations per published message spent on counting. Look the name up
first; a new one is stored only the first time.

Tests now hold the counters against the runtime's own count of
allocations, and they found sites counted wrongly: []byte(key) passed to
AppendEncode or EncodeToString is not copied, a buffer kept as a *[]byte
moves its slice header to the heap, and the string of the persistence
flag in hello escapes. TestAllocCountsSend and TestAllocCountsReceive
compare the counters with testing.AllocsPerRun, and
TestAllocSizeMatchesRuntime compares the size classes with
MemStats.TotalAlloc, so a Go upgrade that changes what is allocated
fails a test instead of making the counters drift. Both pass on Go
1.25 and 1.26. Under the race detector, which CI runs the tests with,
the compiler does not optimise append(s, make([]T, n)...), so base64
decoding of a key above 32 bytes allocates twice: the receive test is
skipped there, the send test runs.

The pubsub-large counters went with pubsub-large; drop them. DeepCopy
does not know the topic of what it copies, so report its counters in a
"process" section of /debug/pubsub/stats instead of as a publication
named "(DeepCopy)".

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A subscriber drops the connection on a frame above its frame size, or
on one which does not decompress within its message size, and logs it.
Count those in rejected_frames of /debug/pubsub/stats too, next to the
values a publisher refuses, so that a mismatch of the size limits
between two processes shows in the counters.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Every subscriber connection of a publication kept its own copy of what
it had been sent, and on every change of any key compared the whole
publication against it: GetAll deep-copied every item, a marshal and an
unmarshal each, then every copy was marshalled again to compare it with
the bytes sent before, through a linear search of the connection's map
for each key. Sending an update copied the value once more into the
message. A change of one key of a publication of N keys with S
subscribers cost 2*N*S marshals and N*S unmarshals, and each connection
held a copy of the JSON of every key.

Publish now marshals the item once and keeps the JSON next to it, in a
Value which is never modified. DetermineDiffs hands every connection the
current Value of each key, and a key is sent again only if its Value is
not the one the connection was sent last: nothing is marshalled or
compared, and all connections share the same bytes. An update is written
as its header followed by the shared JSON, in one writev, without
copying the value.

An item which marshals to the JSON already published is unchanged, and
is neither copied nor compared: Publish used to deep-copy every item and
compare the copy with cmp.Equal, which walks byte slices element by
element through reflection and computes an edit script for every slice.
Other JSON is decoded into the copy Publish keeps and compared with
cmp.Equal as before, so an item which the Equal methods of its type find
equal to the one published, e.g. a set in another order, is still
unchanged.

Get and GetAll decode their copies from the JSON: an unmarshal, without
the marshal of a deep copy. A subscription's handler gets its copy
decoded from the message received, and only if the handler is set.

A checkpoint is sent as this version of its type marshals it, as
before.

Measured in a test process publishing real types over the socket
driver, counting every allocation of the process: a change of one of 50
AppInstanceStatus with 10 subscribers took 77,033 allocations of 12.8MB,
now 24,526 of 2.6MB; republishing an unchanged one 439 allocations, now
63; GetAll of the 50, 3,069, now 1,369.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A value whose message is above the frame size is compressed with zstd.
Publish compressed it to check that it fits and threw the result away,
and then every subscriber connection compressed it again to send it:
1+S compressions of the whole message for S subscribers, each with a
copy of the message as its input.

CheckMaxSize now returns the compressed message it made, Publish keeps
it in the Value next to the JSON, and every connection sends that frame
as it is. A checkpoint loaded at start is compressed once there too. A
value which does not fit is refused by Publish as before; one from a
checkpoint is kept and skipped when it would be sent, as before.

/debug/pubsub/stats counts the compressions of each publication in
compressions, next to compressed_messages, which counts the compressed
messages sent. For a DevicePortConfigList of 3.4MB with 5 subscribers,
the wire messages built for a change go from 20.5MB to 3.4MB, the input
of the one compression.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Publish and every subscription compare a changed item with the one they
hold using cmp.Equal, which honours the Equal methods of the types.
cmp.Equal compares a []byte element by element through reflection and
computes an edit script for every pair of slices, quadratic in their
length where they differ. The network types carry certificates as
[][]byte in every port: for a DevicePortConfigList of 1.3MB of JSON, 4
DPCs of 8 ports with a chain of three certificates each, cmp.Equal takes
1.8s when nothing changed and 2m24s when the certificates were replaced.

Compare []byte with bytes.Equal instead, with a nil slice still unequal
to an empty one as before: 5.9ms and 78ms for the same lists. What is
equal does not change; a test holds the results against cmp.Equal
without the option. Starting a persistent subscription to a
DevicePortConfigList of 3.4MB, which compares the list loaded from the
checkpoint with the one the publisher sends, goes from 4.6s and 284MB
allocated to 96ms and 19MB.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
@rucoder
rucoder force-pushed the rucoder/pubsub-cache branch from 09119db to 1be3fd5 Compare November 5, 2026 11:56
rucoder and others added 4 commits November 5, 2026 12:03
A persistent subscription loads its items from the checkpoint at start,
and the publisher then sends it every key again, mostly the same JSON.
Each was decoded and compared with the item held, only to find it
unchanged, and the decoded copy thrown away.

Until such a subscription is synchronized, it now keeps the SHA-256 of
the JSON each item was decoded from, and drops a value with the same
sum without decoding or comparing it. The same JSON decodes to the same
item, so no handler call is lost or added. Once synchronized, the
publisher sends only changes, which are decoded and compared as before:
the sums are dropped, so nothing is hashed or kept after start.
/debug/pubsub/stats counts the updates which change nothing in
unchanged. Starting a persistent subscription to 50 AppInstanceStatus
goes from 27,670 allocations of 3.7MB to 11,016 of 1.9MB, and to a
DevicePortConfigList of 3.4MB from 20,080 allocations of 18.7MB to 2,450
of 14.0MB, most of which is the message read.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A test waiting for something a crash would prevent used to wait out its
timeout, or to carry its own fatal-log watch and reboot check. CrashWatch
(EdgeDevice.WatchForCrash) packages the signs a crash leaves: the fatal an
agent logged, the device's serial console, which the broker serves on
demand and which names the dead agent or the watchdog's victim within
seconds and whatever becomes of the device afterwards, and a reboot the
test did not ask for. Reason returns the first of them, or "" while none
is known. TestLayerDownloadErrorsBounded uses it instead of its own copy.

PACFileProxyConfig lets a device configuration carry a proxy auto-config
script inline, base64-encoded as the EVE API takes it, next to the manual,
transparent and auto-discovered proxy configurations.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
pubsub delivers every object between pillar's agents as one message of at
most 64 KiB; an object that outgrows it makes the publishing agent exit
with a fatal and the watchdog reboot the device, and when the object came
from the controller the device keeps crashing after every reboot, since the
configuration is checkpointed and re-read. Six ordinary configurations or
workloads get there, each measured on the real types so that its default
overflows with margin:

- TestManyStaticDNSEntries: a network instance with 800 static DNS entries
  (NetworkInstanceConfig, zedagent; 600 entries reach the limit).
- TestManyPhysicalIOAdapters: a device model with 200 I/O adapters
  (PhysicalIOAdapterList, zedagent; 160 reach the limit).
- TestHugeConfigItem: debug.enable.ssh with 200 RSA keys, a value of
  about 79 KB (ConfigItemValueMap, zedagent; 45 KB reach the limit).
- TestLargePACFile: a 64 KB proxy auto-config script on the management
  port (DevicePortConfig, zedagent, and the port status and persisted
  port config list in nim; 36 KB reach the limit).
- TestManyVolumesPerApp: an application with 100 volumes
  (AppInstanceStatus, zedmanager; 81 volumes reach the limit).
- TestManyAppInterfaces: 15 applications with 20 network adapters each
  (NetworkMetrics, zedrouter; 225 interfaces reach the limit).

Each test applies its object, waits until the device reports it applied or
in error, and then requires two more metrics reports, which proves pillar
outlived the publish; it stops at once, naming the cause, when the device
crashes. Every size is a test parameter, so a value below the limit runs
the same test as a check of the passing path. The tests form
TestLimitsSuite on one device, applications last.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
TestLimitsSuite takes 31 minutes on a libvirt host, 20 of them in
TestManyAppInterfaces, which deploys 15 applications. Tag it heavy, as
the other suites which deploy many applications or nodes are, so that
the evetest workflows keep it off the light runners.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
@rucoder
rucoder force-pushed the rucoder/pubsub-cache branch from 1be3fd5 to 3c2eea6 Compare November 5, 2026 12:04
@rucoder

rucoder commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@rene done: the bump commit is out, and the branch is rebased on master with #7041. I'll open the bump PR for pkg/newlog, pkg/edgeview, pkg/wwan/mmagent and pkg/kube/kube-init once this one is merged.

I also narrowed the subscriber-side hashing after the review note on subscribe.go. It now applies only while a persistent subscription starts; details in that thread.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterSuite: failure analysis

TestAppInstancePurge

Failure
timed out after 10m0.020774515s (excluding download) waiting for app "37efe5ff-cead-4891-b4ee-f1b9027fc88f" (purge-app) on device "edge-dev" to reach RUNNING state (last state: CREATING_VOLUME)
Claude's conclusion

This matches perfectly — only one retry (retryFailedClusterVolumeCreate attempt 1/12) happened at 10:36:05, which itself hit the stale-file error (not the retriable DNS error) at 10:43:28, after which gcVolumes just keeps logging found unknown format volume for the stale .img with no further create attempts, confirming the volume is permanently stuck exactly as before.

Root cause

This is a reproduction of the exact mechanism already recorded in today's live-ledger.md run #8 entry for TestNodeClusterSuite/TestAppInstancePurge (same test, different app UUID: 018b84ae... there vs 37efe5ff... now) — this is the second occurrence tonight, not a new issue. Live logs confirm the identical sequence: at 10:29:51 the test's node-drain/cordon step evicted coredns-75548b996f-fnm7h along with CDI/KubeVirt/Longhorn pods (zedkube: nodedrain-step:pod-evict-progress ... evicting pod kube-system/coredns-...), which killed DNS for the whole single-node cluster; volumemgr's RolloutDiskToPVC then failed every ~30s from 10:30:20–10:39:46 with dial tcp: lookup longhorn-backend on 10.43.0.10:53 ... connection refused. The one retry (retryFailedClusterVolumeCreate attempt 1/12 at 10:36:05) raced against DNS recovery and instead hit a second, non-retriable error at 10:43:28: CopyImgDirToRawImg: Failed to create raw image file ... /persist/vault/volumes/8ebcd67a-...-pvc-0.img already exists — confirmed live, the file exists on disk timestamped 10:36, and gcVolumes is still logging found unknown format volume for it with no further create attempts since. Because this "already exists" error isn't classified as a transient cluster-storage error, the volume is permanently wedged in CREATING_VOLUME, producing the 10-minute timeout exactly as observed. This doesn't look related to PR #7037's actual diff, which only touches docs/IPC.md (pubsub documentation) and evetest/devconfig.go (adds a PAC-file proxy config option) — neither touches volumemgr, zedkube drain/cordon logic, or DNS — so this is a self-cordon/DNS-outage/stale-volume-file test-isolation bug in TestAppInstancePurge itself, pre-existing and unrelated to the PR under test.

TestClusterVMApp

Failure
Timed out waiting for app "ee3e48b6-8703-4be2-a957-3cc993a5383c" to be scheduled in cluster "test-cluster"
Claude's conclusion

That's enough confirmation of the missing label already shown earlier. I have sufficient evidence to write the root cause.

Root cause

All three EVE devices and the k3s cluster are healthy right now (evetest eve/cluster info shows all nodes Ready/JOINED) — this is not a repeat of the two earlier TestClusterVMApp failures tonight (SDN/broker gRPC tunnel deaths with devices going SUSPECT). Instead, live kubectl get nodes --show-labels on edge-dev1 shows edge-dev3 has no tie-breaker-node label at all (edge-dev1/edge-dev2 both have tie-breaker-node=false), and cluster events confirm edge-dev3's node object was deleted and fully re-registered ~34-37 minutes ago ("Deleting node edge-dev3" → "Starting kubelet" / "RegisteredNode") as this test's setup reused devices from the prior TestTieBreakerCluster test and reconfigured the cluster from a tie-breaker topology to a 3-full-node one. Because Longhorn's longhorn-manager/engine-image/csi-plugin DaemonSets use the exact-match node selector tie-breaker-node=false, the missing label permanently excludes edge-dev3 from running any Longhorn component ("Manager pod is missing: node edge-dev3 has no manager pod running on it"), so the CDI-import PVC/pod for app ee3e48b6's disk — scheduled onto edge-dev3 — has been stuck 32+ minutes with repeated FailedAttachVolume: ... node edge-dev3 is not ready errors, and the app's VM/VMI object was never even created (kubectl get vm/vmi → "No resources found"), producing the 30-minute scheduling timeout.

This is a different specific trigger from the two earlier failures in this run, but it's the same general bug family already recorded in live-ledger.md as recurring and unfixed since 2026-11-02 (run #5): Longhorn's tie-breaker-node DaemonSet gating causing permanent volume-attach hangs on a node Longhorn doesn't run on (there it was PrepareVolume/WaitForLonghornReady hanging on an intentional tie-breaker node; here it's a node whose tie-breaker-node=false label failed to get (re)applied after a tie-breaker→full-member role transition on a reused device). PR #7037's diff only touches docs/IPC.md, evetest/devconfig.go/edgedevice.go (test-harness PAC-config and crash-watch helpers), and a vendored pillar pubsub-internals bump — none of it touches zedkube/kubeapi node-labeling or Longhorn/CDI code, so this looks like a pre-existing cluster-role-transition/labeling bug, not something introduced by this PR.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

evetest: skipped suites for this PR

This PR is confined to pkg/pillar/pubsub internals (marshal/compress/compare a
published value once for all subscribers, skip decoding unchanged checkpoint
items) plus the new TestLimitsSuite that regression-tests pubsub's message
size handling, and small evetest infra additions (CrashWatch,
PACFileProxyConfig) used only by that suite. The wire protocol, checkpoints
and the Publication/Subscription API are unchanged, and handlers are
called exactly as before, so the realistic regression surface is: large
config/status objects (the six TestLimitsSuite cases), byte-slice equality
for things like port certificates, and persistent-subscription checkpoint
replay (covered by TestUpgradeSuite).

Skipped as unrelated to pubsub's internals (Check A) — these suites
exercise domains with no large/complex pubsub objects and no reliance on the
changed comparison/compression/decode logic; a pubsub regression severe
enough to affect them would already show up in the suites left running
(bootstrap, connectivity, apps, limits):

  • TestKonvertSuite (KVM→K conversion, repartitioning, ZFS vault migration)
  • TestKubeRegistrationSuite (cluster registration, Longhorn disks)
  • TestNodeClusterSuite (kube cluster networking/apps)
  • TestNodeClusterResilienceSuite (kube cluster failover)
  • TestHardwareSuite (USB hotplug)
  • TestPortScanSuite (external port scanning)
  • TestPcibackErrorSuite (PCI assignment conflict reporting)
  • TestDiagSuite (diag CLI output)
  • TestDatastoreSuite (download-source protocols)
  • TestStorageSuite (zvol/disk-layout provisioning)
  • TestControllerFaultsSuite (controller HTTP retry/deferred queue, a
    different subsystem from the agent-to-agent pubsub bus this PR touches)

Skipped as redundant incidental coverage (Check B) — within
TestBootstrapSuite, the ...UsingOverrideJSON variants only vary how the
same bootstrap config is delivered to the device, an axis this PR doesn't
touch; each is kept once per network feature (plain TestBootstrapWithStaticIP,
TestBootstrapWithManualProxyConfig, TestBootstrapWithTransparentProxy,
TestBootstrapWithAutoDiscoveredProxy, TestBootstrapWithMgmtVLAN,
TestBootstrapWithLACPBond) and the override-JSON duplicate of each is
skipped:

  • TestBootstrapWithStaticIPUsingOverrideJSON
  • TestBootstrapWithManualProxyConfigUsingOverrideJSON
  • TestBootstrapWithTransparentProxyUsingOverrideJSON
  • TestBootstrapWithAutoDiscoveredProxyUsingOverrideJSON
  • TestBootstrapWithMgmtVLANUsingOverrideJSON
  • TestBootstrapWithLACPBondUsingOverrideJSON

Everything else runs, including the new TestLimitsSuite (the dedicated
regression test for this change), TestUpgradeSuite (checkpoint/decode
compatibility), TestApplicationConnectivitySuite and
TestDeviceConnectivitySuite (port-config/certificate comparisons),
TestSecuritySuite, TestAppsSuite, TestTelemetrySuite, and the rest of
TestBootstrapSuite.

An engineer can comment /evetest all to force the full suite to run
instead, or /evetest <name> <name> ... to run only specific tests/suites.

@rene
rene merged commit c22104d into lf-edge:master Nov 5, 2026
58 of 69 checks passed
@rucoder rucoder mentioned this pull request Nov 5, 2026
3 of 7 tasks
rene pushed a commit that referenced this pull request Nov 5, 2026
Move the modules which require pkg/pillar to master c22104d, the
merge of #7037: published values marshalled and compressed once for
all subscribers, byte slices compared with bytes.Equal, the values a
persistent subscription has loaded not decoded again when the
publisher resends them at start, and the corrected allocation
counters. pkg/newlog, pkg/edgeview, pkg/wwan/mmagent,
pkg/kube/kube-init, pkg/installer, pkg/recovertpm and pkg/vtpm vendor
pkg/pillar/pubsub; evetest and its SDN VM do not vendor and only take
the new version.

Done with make bump-eve-pillar. Only pkg/pillar moves; the vendored
changes are in pubsub, and the wire protocol is the same.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestSecuritySuite: failure analysis

TestVaultKeyModeRecoveryOnExt4

Failure
Timed out waiting for device "edge-dev" to reboot
Claude's conclusion
Root cause

Live evidence shows the device genuinely rebooted successfully — it was just slower than the test's fixed 5-minute budget, not hung. The reboot command landed on nodeagent/zedagent at 15:32:48 (NodeAgentStatus DeviceReboot:true), attestation state progressed normally through StateRestartWait → StateInternalQuoteWait → StateComplete by 15:32:59, and the device's logs stop cleanly shortly after (last entry ~15:33:02, no errors/panics). The VM's kernel didn't actually restart until ~15:36:12 (TPM/EFI dmesg lines at epoch 1793892972), a ~3-minute gap between the shutdown sequence completing and the power-cycle actually happening, and SSH only came back at 15:37:54 — just 7 seconds past the 5-minute timeout (rebootAndWait fired at 15:37:47). Post-boot load average (3.57 on a 4-vCPU box, 2 minutes after boot) is consistent with hypervisor/host contention slowing the VM's power-cycle rather than any pillar/kernel defect.

This is the identical failure signature (Timed out waiting for device "edge-dev" to reboot) and identical mechanism (slow VM power-cycle tipping over the fixed 5-min reboot timeout due to shared-host contention) first seen on this exact test one night ago (2026-11-04, run #7, per live-ledger.md), and recurring again tonight — a known/recurring issue since 2026-11-04, distinct from the older (2026-10-29 through 11-02) failure mode on this test, which was a stale hardcoded attempt-count assertion, not a reboot timeout.

PR #7037's diff touches only pkg/pillar/pubsub internals (socketdriver framing/compression/equality) and evetest test infrastructure (limits tests) — nothing in nodeagent, vaultmgr, baseosmgr, or the reboot/zboot path. So this failure looks unrelated to the PR; it's environmental host contention, not a regression introduced by this change.

TestVaultKeyModeRecoveryOnZFS

Failure
Timed out waiting for device "edge-dev" to reboot
Claude's conclusion

Confirmed same PR as before (#7037) — same scope, only pubsub internals and evetest limits infra, nothing in reboot/nodeagent path. I have enough evidence for a conclusion.

Root cause

This shares the identical root cause and mechanism as the already-investigated TestVaultKeyModeRecoveryOnExt4 failure earlier in this same run: a slow VM power-cycle/detection path tipping over the test's fixed 5-minute reboot budget, not a hang. Live evidence: the reboot was issued at 15:49:05 and the kernel actually came back up at 15:52:19–15:52:22 (confirmed both by the harness's reported boot time and the guest's own dmesg RTC line setting system clock to 2026-11-05T15:52:22 UTC) — about 3m15s, well within budget. But the harness's rebootAndWait didn't register the restart-counter increment until 15:54:06, 1m44s after the kernel had already booted, timing out at the 5:00 mark (15:54:05) just one second before detection succeeded — the delay is in controller-side reboot detection/reporting catching up, not the boot itself (SSH/pillar logs post-boot show clean startup with no errors).

This is a reproduction of the same mechanism documented tonight in live-investigation.md for the Ext4 variant (same signature: boot completes within or near budget, but harness confirmation lags just past the timeout), and per live-ledger.md this reboot-timeout failure mode on TestVaultKeyModeRecoveryOnZFS/Ext4 is known and recurring starting 2026-11-04 (run #7), distinct from the older 2026-10-29→11-02 failure mode (a stale attempt-count assertion). PR #7037's diff touches only pkg/pillar/pubsub internals and evetest limits-test infrastructure — nothing in nodeagent, vaultmgr, or the reboot/zboot path — so this looks unrelated to the PR and environmental (shared-host/VM contention) in nature.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestDeviceConnectivitySuite: failure analysis

TestPNACWithoutProxy

Failure

Timed out after 600.001s.
Expected to satisfy: Device has enrolled NEW certificate for 802.1x AND deleted the obsolete certificate
machineArch:"x86_64" cpuArch:"x86_64" platform:"x86_64" ncpu:4 memory:7917 storage:40006 powerCycleCounter:-1 minfo:{manufacturer:"QEMU" productName:"Standard PC (Q35 + ICH9, 2009)" version:"pc-q35-11.0" serialNumber:"EB5M9F36" UUID:"Not Settable" biosVendor:"EDK II" biosVersion:"unknown" biosReleaseDate:"02/02/2022"} assignableAdapters:{type:PhyIoNetEth name:"ethernet0" members:"ethernet0" usedByBaseOS:true ioAddressList:{macAddress:"da:39:78:d0:19:f2"} usage:PhyIoUsageMgmtAndApps} dns:{DNSservers:"127.0.0.1:53" DNSsearch:"test."} storageList:{mountPath:"/persist/lost+found"} storageList:{mountPath:"/" total:3958} storageList:{mountPath:"/persist/vault/volumes"} storageList:{mountPath:"/persist/netdump"} storageList:{device:"nbd6"} storageList:{device:"nbd2"} storageList:{mountPath:"/persist/ingested"} storageList:{mountPath:"/persist/agentdebug"} storageList:{device:"nbd0"} storageList:{device:"nbd5"} storageList:{device:"nbd15"} storageList:{device:"nbd9"} storageList:{mountPath:"/persist/vault"} storageList:{mountPath:"/hostfs" total:242} storageList:{device:"nbd13"} storageList:{mountPath:"/persist/log"} storageList:{mountPath:"/persist/clear"} storageList:{mountPath:"/persist/newlog"} storageList:{mountPath:"/persist/status"} storageList:{device:"vda4" total:5 partitionLabel:"CONFIG" partitionTypeGuid:"13307e62-cd9c-4920-8f9b-91b45828b798" partitionUuid:"ad6871ee-31f9-4cf3-9e09-6f7a25c30054"} storageList:{mountPath:"/persist/containerd-system-root"} storageList:{device:"nbd4"} storageList:{mountPath:"/persist/certs"} storageList:{mountPath:"/persist" total:40006 storageLocation:true} storageList:{device:"/persist/vector/data/buffer/v2/keep_sent_queue_socket/buffer-data-0.dat"} storageList:{mountPath:"/persist/vault/downloader"} storageList:{mountPath:"/persist/checkpoint"} storageList:{mountPath:"/persist/vault/verifier"} storageList:{device:"vda1" total:2048 partitionLabel:"EFI System" partitionTypeGuid:"c12a7328-f81f-11d2-ba4b-00a0c93ec93b" partitionUuid:"ad6871ee-31f9-4cf3-9e09-6f7a25c30051"} storageList:{mountPath:"/persist/patchEnvelopesUsageCache"} storageList:{device:"nbd14"} storageList:{device:"nbd10"} storageList:{mountPath:"/persist/patchEnvelopesCache"} storageList:{device:"vda3" total:10240 partitionLabel:"IMGB" partitionTypeGuid:"5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6" partitionUuid:"ad6871ee-31f9-4cf3-9e09-6f7a25c30053"} storageList:{device:"nbd7"} storageList:{mountPath:"/persist/nettrace"} storageList:{mountPath:"/persist/tmp"} storageList:{mountPath:"/persist/memory-monitor/output"} storageList:{device:"vda" total:65536} storageList:{device:"nbd8"} storageList:{mountPath:"/persist/clear/volumes"} storageList:{device:"vda2" total:10240 partitionLabel:"IMGA" partitionTypeGuid:"5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6" partitionUuid:"ad6871ee-31f9-4cf3-9e09-6f7a25c30052"} storageList:{device:"nbd11"} storageList:{device:"nbd12"} storageList:{mountPath:"/persist/vault/containerd"} storageList:{mountPath:"/config" total:5} storageList:{device:"nbd3"} storageList:{device:"/persist/vector/data/buffer/v2/dev_upload_socket/buffer-data-0.dat"} storageList:{device:"nbd1"} storageList:{device:"vda9" total:40953 partitionLabel:"P3" partitionTypeGuid:"5f24425a-2dfa-11e8-a270-7b663faccc2c" partitionUuid:"ad6871ee-31f9-4cf3-9e09-6f7a25c30059"} storageList:{device:"vda7" total:2048 partitionLabel:"EFI System" partitionTypeGuid:"c12a7328-f81f-11d2-ba4b-00a0c93ec93b" partitionUuid:"ad6871ee-31f9-4cf3-9e09-6f7a25c30056"} bootTime:{seconds:1793890489} swList:{activated:true partitionLabel:"IMGA" partitionDevice:"/dev/vda2" partitionState:"active" status:INSTALLED shortVersion:"0.0.0-pr7037-09119db5-kvm-amd64" downloadProgress:100 userStatus:UPDATED} swList:{partitionLabel:"IMGB" partitionDevice:"/dev/vda3" partitionState:"unused" status:INITIAL} HostName:"2ce3601a-1301-4a1b-915d-d65532033721" lastRebootReason:"NORMAL: First boot of device - at 2026-11-05T14:55:12.703344064Z" lastRebootTime:{seconds:1793890512 nanos:703344064} systemAdapter:{status:{version:1 key:"zedagent" timePriority:{seconds:1793890588 nanos:492391648} lastSucceeded:{seconds:1793890604 nanos:565776656} ports:{ifname:"eth0" name:"ethernet0" isMgmt:true free:true dhcpType:4 subnet:"172.20.20.0/24" domainname:"test." proxy:{} macAddr:"da:39:78:d0:19:f2" IPAddrs:"172.20.20.123" defaultRouters:"172.20.20.1" dns:{DNSservers:"10.16.16.25" DNSdomain:"test."} up:true err:{timestamp:{seconds:1793890604 nanos:565729733}} usage:PhyIoUsageMgmtAndApps networkUUID:"f34c1584-48cf-40c2-ac6e-ea010967f60e" mtu:1500 config_source:{origin:NETWORK_CONFIG_ORIGIN_CONTROLLER submitted_at:{seconds:1793890588 nanos:492391648}} pnac_status:{enabled:true state:SUPPLICANT_STATE_AUTHENTICATED last_auth_timestamp:{seconds:1793890599}}}}} HSMStatus:NOTFOUND dataSecAtRestInfo:{status:DATASEC_AT_REST_DISABLED info:"TPM is either absent or not in use" vaultList:{name:"Application Data Store" status:DATASEC_AT_REST_DISABLED vaultErr:{description:"TPM is either absent or not in use" timestamp:{seconds:1793890580 nanos:118193658} severity:SEVERITY_ERROR} pcrStatus:PCR_DISABLED unlock_method:VAULT_UNLOCK_METHOD_NO_TPM}} sec_info:{sha_root_ca:"k\xc8D\xd1s\xb5)\xff\xc0\xf0K\xd64Ol\xbc;C\x08A\xbc\xceR%\xb4(<\n˥\xad\xee"} configItemStatus:{configItems:{key:"app.allow.vnc" value:{value:"true"}} configItems:{key:"debug.default.loglevel" value:{value:"debug"}} configItems:{key:"debug.default.remote.loglevel" value:{value:"debug"}} configItems:{key:"debug.enable.console" value:{value:"true"}} configItems:{key:"debug.enable.ssh" value:{value:"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC+H1RQUqHjFBJgGpslC73XsLz8Fg5WpNPble9naKyWz1Um8D2bOtQK/yguCImPeBYcH7/73z8dtC6d+dT0UF26+o7Vh6RN/U2X/5nkaZr7oM5QwwZTsD7Nd2Szww9wrRhXvpV0aFgUBDM9BIF1qBQxLNd+Jp8uttrgF3zj/cm7+SXllG54sv8WFBMfTX7J8cQ1jxLyp/Sc6PXK0zBaVzZwhmCCmI6CIzJK6ahMRgXm2vSP6doYibkB3ETSskaCXSHxDiZoaQK2ZY+GqNZkUusbau43MXVPTiJknXqUcmXhQmwyMSltQ3G54jcgn4TDObSQnW7vGdLI7zIEAHnk5D1BmKzQUBh5aRLdhBtb6T3uvVtAqGgnXWKD+d2GjMiy4G31zfIlArvC3G8LxwsDoxQL0XKaFjnEmXIptVXC68zq+laIM8YGDDOCEc6RfczP7lA4p6rv0gQUfTNqy0P3a4ulvIDb4hET1Gkh+Azkuw1do9NIhXPxDBmPdkTnKwJv6XelpdCPCw1QT5o7WdOkUgorf/e03jGDnnn1QBWpSPB9gLB/oDmT/Gzm8tCYaq7ggwYUq1fSMBvnbFaclH1KVcC2Gwn8UPLt9HHa/mGywuISZYl3gy7ztlKyAjHEZH053U7I8OaMvv/CFo9aR4Teeb5848REZYAes+yJIz3lJV1K5Q=="}} configItems:{key:"newlog.allow.fastupload" value:{value:"true"}} configItems:{key:"scep.retry.interval" value:{value:"300" error:"***ParseItem: Error in parsing Item. Replacing it with existing Value. key: scep.retry.interval, value: 10, Existing Value: {Key:scep.retry.interval ItemType:1 IntValue:300 StrValue: BoolValue:false TriStateValue:0}. Err: value out of bounds. Parsed value: 10, Max: 3600, Min: 60"}} configItems:{key:"timer.config.interval" value:{value:"5"}} configItems:{key:"timer.deviceinfo.interval" value:{value:"30"}} configItems:{key:"timer.download.retry" value:{value:"60"}} configItems:{key:"timer.location.app.interval" value:{value:"20"}} configItems:{key:"timer.location.cloud.interval" value:{value:"300"}} configItems:{key:"timer.metric.interval" value:{value:"20"}} configItems:{key:"timer.port.testbetterinterval" value:{value:"60"}} configItems:{key:"timer.test.baseimage.update" value:{value:"60"}}} last_boot_reason:BOOT_REASON_FIRST hardware_watchdog_present:true reboot_inprogress:true capabilities:{HWAssistedVirtualization:true IOVirtualization:true} state:ZDEVICE_STATE_REBOOTING storage_info:{pool_name:"/persist" storage_type:STORAGE_TYPE_INFO_EXT4 storage_state:STORAGE_STATUS_ONLINE disks:{disk_name:{name:"/dev/vda9" serial:"unknown"} status:STORAGE_STATUS_ONLINE smart_status:"failed"}} attestation_info:{state:ATTESTATION_STATE_COMPLETE error:{description:"[ATTEST] No escrow data" timestamp:{seconds:1793890589 nanos:990966378} severity:SEVERITY_ERROR}} api_capability:API_CAPABILITY_DEFERRED_QUEUE_METRICS optional_capabilities:{hw_inventory_support:true} device_name:"edge-dev" enrolled_certs:{cert_enrollment_profile_name:"scep-test" subject:{common_name:"edge-dev" organization:"lf-edge" country:"US"} issuer:{common_name:"SCEP CA" organization:"Example" organizational_unit:"Lab" country:"US"} san_uri:"urn:Name:edge-dev" issue_timestamp:{seconds:1793889997} expiration_timestamp:{seconds:1825426597} renew_period_percent:50 sha256_fingerprint:"3b652abcb1383ddd125106ebdc672075a1c3457edf8f685b0fdccc1a02611232" status:CERT_STATUS_AVAILABLE}
Claude's conclusion
Root cause

Live evidence: edge-dev requested a reboot at 14:56:55 (part of the test's "swap SCEP profile while offline, reboot, and verify EVE resyncs persisted config against the newly received config" phase — see net_adapter_test.go:509-548), came back up and logged normally through 14:58:26, then went completely silent — no further pillar/nim/scepclient log lines, no IP ever acquired on eth0 (collect-info/ssh fail with "no reachable endpoint"/"no IP address" even now, 12+ minutes later), and evetest status still shows the stale, pre-swap cert (cert_enrollment_profile_name:"scep-test"), never the expected new-scep-profile. The last thing nim logged was loading the stale, pre-reboot DevicePortConfigList from checkpoint (still referencing CertEnrollmentProfileName:scep-test) and running its "initial reconcile" of generic iptables rules — it never got far enough to bring the port up with the new PNAC/SCEP config. evetest sdn status is clean (no config errors, management IP reachable) and the console is reachable, ruling out the usual SDN-VM/broker infra flake that dominates most other entries in live-ledger.md; this is the first occurrence of this specific failure signature for TestPNACWithoutProxy in the ledger, so it is not a known/recurring issue.

This looks directly related to PR #7037: it heavily rewrites pkg/pillar/pubsub (publish.go, subscribe.go, socketdriver/*) specifically around how a persistent subscription resynchronizes against a publisher's checkpointed/restarted state (new SHA‑256 "already seen this checkpoint bytes" short-circuit in handleModify, and Publish's new byte/cmp.Equal-based "unchanged" dedup using *Value pointers for diffing). The very test that failed exists specifically to exercise that code path (its own comment: "check that EVE will (eventually) properly resync persisted config against the received config after boot"), and the live symptom — nim stuck applying the stale checkpointed DPC after reboot and never progressing to the fresh controller config — is exactly the kind of bug this refactor could introduce (e.g. a publisher/subscriber incorrectly treating the freshly-fetched config as "unchanged" vs. the checkpoint and suppressing the notification nim needs to reconfigure the port). Given the strength of this correlation, I'd treat this as a likely regression from the pubsub changes in this PR rather than infra flakiness.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestAppsSuite: failure analysis

TestPurgeNeverActivatedApp

Failure
Failed to setup devices [edge-dev]: rpc error: code = Unknown desc = failed to setup EVE device "edge-dev": failed to get import storage "local": unexpected end of JSON input
Claude's conclusion
Root cause

Live evidence (evetest status, evetest sdn status) shows no EVE device or SDN was ever created — TestPurgeNeverActivatedApp failed inside Setup→setupEVEDevices (setup.go:852/harness.go:1316) at 14:28:39, only ~2.5 minutes after the harness/Adam/image-server came up cleanly (adam-stdout, gotest.json show normal startup at 14:26:22). The error, failed to get import storage "local": unexpected end of JSON input, is the broker truncating/failing to parse a Proxmox API response while provisioning storage for the device VM — this happened before any device or network existed, so there's no further device-side state to inspect.

This is a known, recurring issue: live-ledger.md documents the identical "...storage \"local\"...: unexpected end of JSON input" Proxmox API signature repeatedly — explicitly called out as a "well-established recurring pattern dating back weeks" in the 2026-11-05 run #8 TestThreeNodesUpgrade entry, tracing back to at least 2026-11-01 (run #4) and recurring through 2026-11-03/04 — consistent with Proxmox broker/host overload returning truncated API responses during VM/storage provisioning. It does not share a root cause with the only other prior TestPurgeNeverActivatedApp ledger entry (2026-10-29), which failed differently (SDN gRPC "no route to host" after the device VM was already provisioned).

Regarding PR #7037: its diff only touches evetest/devconfig.go (PAC proxy config), evetest/edgedevice.go (new CrashWatch helper), test files under evetest/tests/{apps,limits}, and pkg/pillar/pubsub — none of which touch the broker provisioning/storage-import path (setup.go/harness.go) where this failed. This looks pre-existing/environmental (Proxmox broker instability), not caused by this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants