Skip to content

Fix SBOM generation: bump pillar in modules left on an unreachable commit - #7041

Merged
rene merged 4 commits into
lf-edge:masterfrom
rene:fix-pillar-bump-sbom
Nov 5, 2026
Merged

rene merged 4 commits into
lf-edge:masterfrom
rene:fix-pillar-bump-sbom

Conversation

@rene

@rene rene commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

The Publish workflow on master has been failing in the "Collect and push SBOM and sources" step since #5644. The failure is in go-sources-and-licenses:

Error: failed to walk directory .../pkg: failed to get package
github.com/lf-edge/eve/pkg/pillar@v0.0.0-20261103090849-1d275e6fd307:
failed to get module github.com/lf-edge/eve/pkg/pillar: failed to get module zip: 404 Not Found

Commit 2cb4f79 pinned pkg/edgeview, pkg/newlog, pkg/kube/kube-init and pkg/wwan/mmagent to pillar at 1d275e6fd307. That commit is on the PR branch before the rebase, so it never reached master, and proxy.golang.org returns 404 for it. Builds still pass because they use the vendored copy. Only the sources collection downloads module zips.

The follow-up bump in 7cea040 skipped exactly those four modules. go get -u cannot upgrade from a version it cannot fetch (invalid version: unknown revision 1d275e6fd307). bump-eve-pillar runs it through find -execdir, which ignores the exit status of the commands it runs, so make still reported success.

This PR has four commits:

  1. Bump eve/pkg/pillar in the modules left on an unreachable commit: the vendor-only commit. It points the four modules at a master commit with go mod edit, then runs the usual go get -u, go mod tidy and go mod vendor. They now pin v0.0.0-20261105091144-7cea040d89f9. In each go.mod only the pillar line changes.
  2. Makefile: fail bump-* targets when a module cannot be bumped: one macro now drives all four bump-* targets. It bumps each tracked module that requires the dependency, keeps going past failures, and at the end exits non-zero with the list of modules that failed. Matching on the require line also skips the dependency's own module (go get failed in pkg/pillar on every bump-eve-pillar run) and untracked copies such as .claude/worktrees.
  3. pkg/debug: use the current eve-recovertpm image: 7cea040 changed the vendor tree of pkg/recovertpm, so its image tag changed. pkg/debug/Dockerfile still referenced the old tag, and make check-docker-hashes-consistency failed on master.
  4. pkg/installer: use the current eve-debug image: commit 3 changes the eve-debug image tag, which pkg/installer/Dockerfile pins.

How to test and validate this PR

Tested locally:

  • SBOM walk: the same walk tools/collect-sources.sh runs in CI:
    go install github.com/deitch/go-sources-and-licenses@v1.0.0
    go-sources-and-licenses sources -s "$PWD/pkg" --find --out /tmp/out --prefix golang \
      --template 'golang,{{.Module}}@{{.Version}},{{.Version}},{{.Path}}'
    On master (7cea040) it fails with the 404 above. On this branch it exits 0, and every pillar reference resolves to a master commit.
  • Builds: go build ./... with -mod=vendor in the four modules passes.
  • Bump target on master: make bump-eve-pillar with the new Makefile, run on master, exits 2:
    ERROR: failed to bump github.com/lf-edge/eve/pkg/pillar in: pkg/edgeview pkg/kube/kube-init pkg/newlog pkg/wwan/mmagent
    
    Before this change it exited 0.
  • Docker hashes: make check-docker-hashes-consistency passes. On master it fails with pkg/debug/Dockerfile uses lfedge/eve-recovertpm:f39bf74c... but c1a132f0... is built in this repo.
  • Proxy: proxy.golang.org returns 200 for the new pillar version and 404 for the old one.

After merge, the Publish workflow on master should pass the "Collect and push SBOM and sources" step.

Changelog notes

None. Build tooling only.

PR Backports

  • 17.0-stable: No, the broken pin exists only on master.
  • 16.0-stable: No, the broken pin exists only on master.
  • 14.5-stable: No, the broken pin exists only on master.
  • 13.4-stable: No, the broken pin exists only on master.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't check them. Build tooling and vendor change only: no device testing or documentation needed. It was verified locally with the SBOM sources walk CI runs.

🤖 Generated with Claude Code

rene and others added 2 commits November 5, 2026 11:06
Commit 2cb4f79 pinned pkg/edgeview, pkg/newlog, pkg/kube/kube-init
and pkg/wwan/mmagent to pillar v0.0.0-20261103090849-1d275e6fd307, a
commit of the PR branch that was rebased before merge and so never
reached master. proxy.golang.org returns 404 for it, which breaks
go-sources-and-licenses in the SBOM/sources step of the Publish
workflow.

The previous bump (7cea040) skipped these four modules: go get
cannot upgrade from a version it cannot fetch, and bump-eve-pillar
runs it through find -execdir, which ignores the failure. Point them
at a master commit with go mod edit first, then run the usual
go get -u, go mod tidy and go mod vendor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The bump targets ran go get through find -execdir, which ignores the
exit status of the commands it runs, so a module that could not be
bumped was skipped silently while make reported success. That is how
7cea040 left four modules pinned to an unreachable pillar commit.

Bump each tracked module that requires the dependency, keep going
past failures, and exit non-zero at the end with the list of modules
that failed. Matching on the require line also skips the module of
the dependency itself (go get always failed in pkg/pillar for
bump-eve-pillar) and untracked copies such as .claude/worktrees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The pillar bump in 7cea040 changed the vendor tree of
pkg/recovertpm, so its image tag changed, but pkg/debug/Dockerfile
still referenced the old one and make check-docker-hashes-consistency
failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Updating the eve-recovertpm image in pkg/debug changed the eve-debug
image tag, which pkg/installer/Dockerfile pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.57%. Comparing base (7cea040) to head (dc63d2e).
⚠️ Report is 17 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7041      +/-   ##
==========================================
+ Coverage   29.06%   29.57%   +0.50%     
==========================================
  Files         548      559      +11     
  Lines       98971   100566    +1595     
==========================================
+ Hits        28768    29738     +970     
- Misses      67932    68604     +672     
+ Partials     2271     2224      -47     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene
rene merged commit 6758baf into lf-edge:master Nov 5, 2026
45 of 46 checks passed
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestDiagSuite: failure analysis

TestDiagOutput

Failure
timed out after 5m0.001661986s (excluding download) waiting for app "d697db95-ae57-4635-9ecd-a1c86270705f" (diag-reader) on device "edge-dev" to reach RUNNING state (last state: HALTING)
Claude's conclusion
Root cause analysis

Live evidence: domainmgr/the KVM hypervisor backend did everything right up through launch — the log shows starting KVM domain d697db95…, the correct qemu-system-x86_64 args (Hypervisor args: […] -pidfile /run/hypervisor/kvm/d697db95….1.1/pid), created domainID 5899, and done launching qemu device model — but then waitForQmp polled query-status every 6s for 30s and never found the QMP socket (dial unix .../qmp: connect: no such file or directory), so the app went to HALTING. Live on the device right now: /run/hypervisor/kvm/ is empty (no per-domain dir), ctr … eve-user-apps tasks/containers ls shows nothing running, and ps aux shows only the idle qemu service's containerd-shim — i.e. the qemu process for this VM never actually materialized (or was torn down) despite domainmgr believing it launched it, and nothing in the logs (dmesg, pillar logs) records a crash/exit/OOM to explain why. /dev/kvm, VMX, and nested-virt are all fine, and memory/load on the host are nowhere near constrained (6.6GB free, loadavg 0.2–1.6), ruling out resource exhaustion as the obvious cause.

Ledger check: No prior entry in live-ledger.md mentions TestDiagOutput, diag-reader, QEMU Monitor/qmp, or HALTING — this is the first time this specific failure signature has shown up, not a known/recurring issue. It's also not a continuation of the other cascades recorded in this run's ledger (broker unreachability, kvm D-state wedges, SDN tunnel death) — edge-dev here is healthy and reachable.

PR #7041 relatedness: The diff is purely dependency/build plumbing — a Makefile refactor of the bump-* targets and base-image hash bumps (eve-debug, eve-recovertpm) plus vendored pkg/pillar go.mod/go.sum bumps in edgeview, kube-init, newlog, mmagent. It touches no domainmgr/hypervisor/qemu code paths, so this looks like an unrelated, environment-level flake (a one-off qemu/containerd launch race) rather than something this PR caused.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestLPSSuite: failure analysis

TestRadioSilence

Failure
Timed out waiting for device "edge-dev" to reboot
Claude's conclusion
Root cause

Live evidence shows the device actually rebooted successfully — it just took longer than the test's fixed timeout. The controller issued the reboot at 12:57:29, the app halted by 12:58:39, and the device's new boot (confirmed via live SSH: uptime shows "up 3 min" at 13:04:25, i.e. boot ~13:01) was reported back to Adam only at 13:02:45 (REBOOTING -> BOOTING -> ONLINE, restart counter 0→1) — just 16 seconds after evetest's 5‑minute rebootAndWait deadline expired at 13:02:29. Live top on the device right now shows 43% iowait and a load average of 9.07 on a 4-vCPU box only 3 minutes after boot, consistent with host/hypervisor contention slowing the VM's power-cycle rather than any EVE/pillar hang (pillar's own shutdown/boot sequence completed cleanly).

This is a known, recurring issue: the exact same "device reboots fine but exceeds the fixed 5-minute rebootAndWait budget due to host contention" mechanism is recorded in the ledger for 2026-11-03 (run #6), where it hit TestVaultKeyModeRecoveryOnExt4 and TestVaultKeyModeRecoveryOnZFS (first appearance of this failure mode, previously those tests only failed on stale assertion counts). Tonight's TestRadioSilence instance is a new test hitting the same recurring root cause, not an isolated bug.

PR #7041's diff is purely a vendor bump (eve-api/eve-libs pillar pointer bumps) and eve-debug/eve-recovertpm base-image pin updates, plus a Makefile refactor of the bump-* targets — none of it touches reboot/shutdown logic, so this failure looks unrelated to the PR and is better explained by shared Proxmox host-resource contention.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestAppsSuite: failure analysis

TestVMAppPurgeAfterPowerCycle

Failure

Timed out after 951.548s.
The function passed to Eventually failed at /evetest/tests/apps/purge_helpers_test.go:161 with:
expected the purge counter file to exist
Expected
    <bool>: false
to be true
Claude's conclusion

Good, that confirms the known issue dates back to run #1 (2026-10-29) and recurs every night since. I have a solid root cause now.

Root cause

This is not a pillar/zedmanager bug — it's the device going network-unreachable for ~16 minutes at exactly the moment the test needed to read a file. Adam's access log for this device (adam-stdout) shows a complete gap in check-ins from 13:57:42 to 14:13:57, which lines up precisely with the test's stuck Eventually polling window (it started waiting right after the post-purge reboot at 13:57:32 and gave up at 14:13:24, 951s later). The harness's own evetest eve ssh/SCP reads were equally unable to reach the device during that window, which is why purgeCounter()'s SCP-based read of /persist/status/zedmanager/UuidToNum/...json kept reporting "not found." Live inspection right now confirms the file exists and has always held the correct value (Number:1, unchanged since before the final reboot, mtime 13:56:24), and SSH to the device works fine — the device simply came back on its own right around the time the test timed out (device check-ins to Adam resume at 13:57→14:13:57, and evetest sdn status/ssh are healthy now). So the purge itself succeeded; the test just couldn't observe it during an unrelated connectivity blackout.

This shares its mechanism (not its specific symptom) with the extensive cascade of "SDN VM/gRPC tunnel becomes unreachable" failures already logged elsewhere in tonight's run (TestKonvertSuite's Shrink/Grow/etc.), and per live-ledger.md that broad instability is a known, recurring issue dating back to 2026-10-29 (run #1), recurring in every nightly run since. This specific test/failure signature (TestVMAppPurgeAfterPowerCycle / "purge counter file" timeout) doesn't appear by name in prior ledger entries, but it's a new manifestation of the same underlying infrastructure problem — notably milder here, since the device self-recovered rather than staying permanently SUSPECT.

Regarding PR #7041: its diff only touches the bump-* Makefile targets (refactoring them into a shared macro with failure handling) and the corresponding vendored eve/pkg/pillar version bumps plus two unrelated base-image hash bumps (eve-debug, eve-recovertpm) — no pillar, zedmanager, networking, or SDN code is touched. This failure is not plausibly caused by that PR; it looks purely environmental.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestApplicationConnectivitySuite: failure analysis

TestSwitchNIDeletedWithAppsAttached

Failure
timed out after 5m0.000860846s (excluding download) waiting for app "2722cde5-8e97-435b-8b6c-78e61abef02a" (app-1) on device "edge-dev" to reach RUNNING state (last state: RESOLVING_TAG)
Claude's conclusion
Root cause

Live evidence shows app-1's content tree (3e8d4a30-528e-428e-afe9-f0e67883bd16, image lfedge/evetest-ubuntu-ctr:1.2) is still wedged in RESOLVING_TAG right now (evetest eve app-info / VolumeStatus / ContentTreeStatus all confirm State:101, ContentSha256:""). The device's own newlog archive shows volumemgr published a ResolveConfig at 15:06:24.627, downloader's resolveHandler/resolveTagsToHash picked it up and called ociutil.manifestsDescImg(192.168.170.2:5000/lfedge/evetest-ubuntu-ctr:1.2) at 15:06:24.676 — and then never logged anything again for that resolve: by 15:12:03 (6+ minutes later, matching the 5-minute timeout) volumemgr is still repeating lookupResolveStatus… not found/Resolve status not found, i.e. the resolver goroutine hung instead of completing or erroring.

This is the identical signature already catalogued in live-ledger.md as a known, recurring downloader/registry-resolve-hang bug — a manifestsDescImg/tag-resolution (or sibling manifest/blob-fetch) call against the SDN's local registry hangs forever for one specific image while sibling pulls complete normally. That pattern first appears 2026-10-31 (TestFlowLog, TestLocalNetInstanceACLs), recurs 2026-11-02 and 2026-11-03 (TestFlowLog), and notably hit this exact test, TestSwitchNIDeletedWithAppsAttached, on 2026-11-04 (as a "download stalled at 57%" variant of the same mechanism). So today's failure is a continuation of a known, recurring infrastructure/downloader bug, not a new one, and it does not share a cause with any other failure investigated in tonight's run (device is healthy/ONLINE, no SDN-tunnel outage this time).

PR #7041's diff only touches the Makefile bump-target logic, pkg/debug/pkg/installer Dockerfile base-image pins, and go.mod/go.sum vendor bumps of eve/pkg/pillar in edgeview/kube-init/newlog/mmagent — it doesn't touch downloader, volumemgr, or the zedUpload/ociutil registry-resolve code path at all, so this failure looks unrelated to the PR under test.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestDeviceConnectivitySuite: failure analysis

TestPNACWithoutProxy

Failure

Unexpected error:
    <*errors.errorString | 0x57cabb0a700>: 
    curl http://http-server.test/helloworld failed: 
    {
        s: "curl http://http-server.test/helloworld failed: ",
    }
occurred
Claude's conclusion
Root cause

Live evidence shows this wasn't actually a networking/ACL problem: curl http://http-server.test/helloworld works instantly right now (Connected... 200 OK in <1s), the device's routing/ARP/DHCP state is healthy, and the SDN side logged zero port/VLAN/firewall activity during the failure window. The real signal is in the device's own pubsub/log stream (/persist/newlog/keepSentQueue/dev.log.*): every pillar agent (zedagent, nim, domainmgr, volumemgr, ledmanager, etc.) went completely silent from 12:45:58 to 12:47:36 — a 98‑second gap with no log lines from any agent at all, despite normally logging multiple times per second. The test's SSH command (curl .../helloworld) connected at 12:46:10 but produced no output and was killed by the harness's "stdout inactivity" watchdog at 12:46:24, squarely inside that silent window — i.e. the whole EVE guest (not just curl, not just PNAC/VLAN state) froze briefly and then resumed normally on its own.

This matches a known, recurring infra issue in live-ledger.md: the edge-dev qemu guest occasionally wedges due to a host-kernel KVM async-page-fault scheduling pathology (kvm_async_pf_task_wait_schedule, kernel 6.18.55-linuxkit-core), first seen 2026-11-02 and recurring repeatedly since (2026-11-03 cascades, etc.) on this same Proxmox broker infrastructure. This exact test (TestPNACWithoutProxy) hasn't hit this specific symptom before, but it's the same underlying host-side guest-freeze mechanism as those prior incidents, not a new/distinct bug.

PR #7041's diff is purely dependency/Dockerfile-base-image bumps (Makefile, pkg/debug, pkg/installer, pkg/edgeview, pkg/kube/kube-init, pkg/newlog, pkg/wwan/mmagent go.mod/go.sum and Dockerfiles) — it touches none of the PNAC, networking, or SSH code paths involved here, so this failure is unrelated to the PR and looks environmental (host-infra flake).

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestSecuritySuite: failure analysis

TestAppVTPMIdentitySurvivesReboots

Failure

Timed out after 300.002s.
The function passed to Eventually failed at /evetest/tests/security/vtpm_helpers_test.go:127 with:
read guest TPM NV marker: ERROR:tcti:src/tss2-tcti/tcti-device.c:451:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpmrm0: No such file or directory 
ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 
ERROR:tcti:src/tss2-tcti/tcti-device.c:451:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpm0: No such file or directory 
ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 
WARNING:tcti:src/util/io.c:262:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused 
ERROR:tcti:src/tss2-tcti/tcti-swtpm.c:613:Tss2_Tcti_Swtpm_Init() Cannot connect to swtpm TPM socket 
ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-swtpm.so.0 
WARNING:tcti:src/util/io.c:262:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused 
ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-mssim.so.0 
ERROR:tcti:src/tss2-tcti/tctildr-dl.c:269:tctildr_get_default() No standard TCTI could be loaded 
ERROR:tcti:src/tss2-tcti/tctildr.c:430:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI 
ERROR: Could not load tcti, got: "(null)"

Unexpected error:
    <*fmt.wrapError | 0x83dd6ee0780>: 
    failed to execute script over SSH for app 1f2e6f77-d0b5-41c4-a131-1c6069019a02: Process exited with status 1 (stderr: ERROR:tcti:src/tss2-tcti/tcti-device.c:451:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpmrm0: No such file or directory 
    ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 
    ERROR:tcti:src/tss2-tcti/tcti-device.c:451:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpm0: No such file or directory 
    ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 
    WARNING:tcti:src/util/io.c:262:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused 
    ERROR:tcti:src/tss2-tcti/tcti-swtpm.c:613:Tss2_Tcti_Swtpm_Init() Cannot connect to swtpm TPM socket 
    ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-swtpm.so.0 
    WARNING:tcti:src/util/io.c:262:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused 
    ERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-mssim.so.0 
    ERROR:tcti:src/tss2-tcti/tctildr-dl.c:269:tctildr_get_default() No standard TCTI could be loaded 
    ERROR:tcti:src/tss2-tcti/tctildr.c:430:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI 
    ERROR: Could not load tcti, got: "(null)"
    )
    {
        msg: "failed to execute script over SSH for app 1f2e6f77-d0b5-41c4-a131-1c6069019a02: Process exited with status 1 (stderr: ERROR:tcti:src/tss2-tcti/tcti-device.c:451:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpmrm0: No such file or directory \nERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 \nERROR:tcti:src/tss2-tcti/tcti-device.c:451:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpm0: No such file or directory \nERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0 \nWARNING:tcti:src/util/io.c:262:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused \nERROR:tcti:src/tss2-tcti/tcti-swtpm.c:613:Tss2_Tcti_Swtpm_Init() Cannot connect to swtpm TPM socket \nERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-swtpm.so.0 \nWARNING:tcti:src/util/io.c:262:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused \nERROR:tcti:src/tss2-tcti/tctildr-dl.c:169:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-mssim.so.0 \nERROR:tcti:src/tss2-tcti/tctildr-dl.c:269:tctildr_get_default() No standard TCTI could be loaded \nERROR:tcti:src/tss2-tcti/tctildr.c:430:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI \nERROR: Could not load tcti, got: \"(null)\"\n)",
        err: <*ssh.ExitError | 0x83dd66e6dc0>{
            Waitmsg: {status: 1, signal: "", msg: "", lang: ""},
        },
    }
occurred
Claude's conclusion
Root cause

This is the same domainmgr/vtpmd launch-race bug previously diagnosed in the ledger (2026-11-02, run #5), recurring here with a slightly different symptom. Live domainmgr logs on edge-dev show that at 13:49:38Z it called reserveAdapters(...) for vtpm-identity and asked vtpmd to launch the swtpm backend over its unix socket, but by 13:49:43Z that HTTP client call failed with "Failed to setup vTPM for 1f2e6f77-...1.1: failed to launch vTPM instance: ... context deadline exceeded (Client.Timeout exceeded while awaiting headers)". vtpmd's own logs show it was mid-way through "Trying to unseal disk key with PCRs [...]" at that same moment and only reported "vTPM launched SWTPM instance ... pid: 37" one second after domainmgr had already given up. Live evidence confirms the fallout exactly matches: the generated qemu domain config (/run/domainmgr/domains/dom1.cfg) for this boot has zero tpm/tpmdev entries, yet a live swtpm process and its /persist/swtpm/tpm-state-1f2e6f77-... state dir exist on the host — so the guest booted and was SSH-reachable but had no /dev/tpm0//dev/tpmrm0, causing the test's 300s Eventually TCTI probe to fail exactly as reported.

This is the same underlying mechanism as the 2026-11-02 (run #5) ledger entry for this same test — a race between vtpmd's unseal/launch latency and domainmgr's fixed client timeout for the vTPM setup call — making this a recurring, known issue (first seen 2026-11-02) rather than a new regression, though the specific timing/error string differs slightly (today it's an HTTP client deadline on the launch RPC itself, rather than the older swtpmTimeout/no-TPM-in-qemu-config path, but the net effect — TPM omitted from the domain config because domainmgr timed out before vtpmd finished — is identical). The PR under test (#7041) only touches Makefile bump-target refactoring and pkg/debug/pkg/installer base-image pin bumps plus vendored-pillar version bumps — nothing in domainmgr, vtpmd, or the hypervisor/vTPM code paths — so this failure looks pre-existing/environmental, not caused by this PR.

TestAppVTPMProtectedFailurePreservesState

Failure

Timed out after 300.001s.
The function passed to Eventually failed at /evetest/tests/security/vtpm_helpers_test.go:94 with:
Unexpected error:
    <*fmt.wrapError | 0x83dd62f9e40>: 
    failed to execute script over SSH for app 33a8bfd1-bfa8-49fd-92d4-f4694181d47d: Process exited with status 1 (stderr: )
    {
        msg: "failed to execute script over SSH for app 33a8bfd1-bfa8-49fd-92d4-f4694181d47d: Process exited with status 1 (stderr: )",
        err: <*ssh.ExitError | 0x83dd6857700>{
            Waitmsg: {status: 1, signal: "", msg: "", lang: ""},
        },
    }
occurred
Claude's conclusion

Confirmed — PR #7041 only touches Makefile bump-target refactoring, pkg/debug/pkg/installer image pin bumps, and vendored go.mod/go.sum bumps; nothing in domainmgr/vtpmd.

Root cause

Live domainmgr logs on edge-dev show the same mechanism as the earlier TestAppVTPMIdentitySurvivesReboots failure in this run: at 14:14:48.845Z domainmgr called reserveAdapters(...) for app 33a8bfd1-... (vtpm-protected-failure) and asked vtpmd to launch the swtpm backend over its unix socket, but at 14:14:54.474Z logged "Failed to setup vTPM for 33a8bfd1-....1.1: failed to launch vTPM instance: ... context deadline exceeded (Client.Timeout exceeded while awaiting headers)". Consistent with that, the live qemu domain config (/run/domainmgr/domains/dom1.cfg) has zero tpm/tpmdev entries even though a swtpm process and its /persist/swtpm/tpm-state-33a8bfd1-... state dir are running on the host — so the guest booted and was SSH-reachable but had no /dev/tpm0//dev/tpmrm0, causing writeGuestTPMMarker's in-guest script (which presumably just checks for the device) to exit 1 with empty stderr every time the harness connected, until the 300s Eventually timed out exactly as reported.

This shares the same root cause as the TestAppVTPMIdentitySurvivesReboots failure already investigated earlier in this run (live-investigation.md) — the vtpmd unseal/launch-latency vs. domainmgr fixed-timeout race that omits the TPM device from the qemu config — making it the second manifestation of that bug within this same run, not an independent issue. The ledger has no prior entry for TestAppVTPMProtectedFailurePreservesState by name, but the underlying mechanism was first documented there on 2026-11-02 (run #5) under the sibling test, so this is a recurring, known issue rather than new. Given PR #7041's diff is limited to build-tooling/image-pin/vendor bumps with no domainmgr/vtpmd changes, this failure is pre-existing/environmental and unrelated to the PR under test.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterResilienceSuite: failure analysis

TestDNIDandBackupDNID

Failure
timed out after 30m0.007338127s (excluding download) waiting for app "07599524-0ea9-478a-b748-dbab2671b415" (vm1) on device "edge-dev1" to reach RUNNING state (last state: BOOTING)
Claude's conclusion
Root cause

Live evidence confirms this is the exact same bug previously diagnosed in the ledger. On edge-dev1 (the DNID for app 07599524-.../vm1), zedkube reports IsDNidNode:true ScheduledOnThisNode:false AppKubeStatus:RunningState, and zedbox/domainmgr logs Info(07599524-...): vm1-07599-0 scheduled on a different node on every poll. kubectl get vmi confirms the VMI is actually Running/Ready on edge-dev3, not edge-dev1 — the VMIRS only expresses a preferred node affinity toward the DNID, so the scheduler placed it elsewhere with no outage involved. zedmanager's handleENClusterAppStatusImpl only derives RUNNING status from the cluster-wide view in the failover-triggered case, so when a pod simply lands off the DNID for normal scheduling reasons, the DNID node's AppInstanceStatus stays permanently wedged at BOOTING, causing the 30-minute timeout evetest hit.

This is a known, recurring issue per live-ledger.md: first observed 2026-10-31 (run #3), recurred 2026-11-01 (run #4), and now again today (run #8) — same code path (zedmanager/zedkube DNID/backup-DNID status-reporting gap), same signature each time.

The PR under test (#7041) only touches the Makefile bump-* targets, two base-image hash bumps in pkg/debug/pkg/installer Dockerfiles, and vendored pillar version-pin bumps in edgeview/kube-init/newlog/mmagent go.mod/go.sum (no pillar source changes). None of this touches zedmanager/zedkube/domainmgr code, so the failure is unrelated to this PR — it's a pre-existing, unfixed architectural gap in the DNID status-reporting logic.

TestVMAppPurgeDuringFailover

Failure

Timed out after 900.000s.
The function passed to Eventually failed at /evetest/tests/cluster/purge_during_failover_test.go:169 with:
expected exactly one VMIRS for the app, found []
Expected
    <[]string | len:0, cap:0>: nil
to have length 1
Claude's conclusion

This confirms edge-dev1 (the designated node) is unreachable/suspect — consistent with the deliberate failover scenario and matching the exact prior-run mechanism. I have enough evidence.

Root cause

This is the same known, recurring zedmanager/zedkube "purge-after-failover" deadlock already seen for TestVMAppPurgeDuringFailover — live evidence on edge-dev2 for app failover-purge-app (236ec65e-...) shows PurgeInprogress:4, PurgeStartedAt:2026-11-05T16:34:36Z, Activated:false, IsDesignatedNodeID:false, while kubectl get vmirs -A on the cluster returns "No resources found" — exactly the found [] the test asserts on. edge-dev1 (the designated node) is in EVE_DEVICE_STATE_SUSPECT and currently unreachable over SSH ("failed to detect any IP address"), matching the deliberately-downed-node failover scenario; once the purge tore down the VMIRS, nothing in the cluster recreated it.

This failure was already investigated earlier in this same run (see the prior write-up embedded in live-ledger.md's run #8 entry), describing the identical signature and attributing it to getKubeAppActivateStatus in pkg/pillar/cmd/zedmanager/zedmanager.go — today's occurrence corroborates that same conclusion with fresh live state. Per live-ledger.md, this exact mechanism has been a known/recurring issue in TestVMAppPurgeDuringFailover since 2026-10-29 (run #1) and has now reproduced in every nightly run through today (run #8) — at least 8 consecutive occurrences.

PR #7041 only touches Makefile bump-* target refactoring, two base-image hash bumps (pkg/debug, pkg/installer Dockerfiles), and vendored pillar version-pin bumps in edgeview/kube-init/newlog/mmagent go.mod/go.sum — no pillar source changes. This failure is unrelated to the PR; it's the same pre-existing, unfixed zedmanager purge/failover bug.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterSuite: failure analysis

TestClusterVMApp

Failure
Timed out waiting for app "5c18da19-074a-462b-82a5-bd89dcabbfea" to be scheduled in cluster "test-cluster"
Claude's conclusion
Root cause

Live inspection shows the app itself is stuck, not the harness/SDN: evetest eve app-info for 5c18da19-074a-462b-82a5-bd89dcabbfea is parked in CREATING_VOLUME, and its backing PVC-population pod cdi-upload-89c72331-...-pvc-0 has been ContainerCreating on edge-dev3 for 30+ minutes with repeated FailedAttachVolume: ... node edge-dev3 is not ready. kubectl get nodes.longhorn.io confirms edge-dev3 is Ready=False with reason ManagerPodMissing ("no manager pod running on it"), because the longhorn-manager DaemonSet only targets nodes labeled tie-breaker-node=false (2/2 ready — edge-dev1/edge-dev2), while edge-dev3 carries no tie-breaker-node label at all (unlike the other two). This device/cluster was reused straight from the prior TestTieBreakerCluster run, so edge-dev3 is almost certainly the designated tie-breaker/witness node on which Longhorn is deliberately never scheduled — but the CDI-generated, one-off cdi-upload-* pod isn't covered by tiebreaker.go's one-time nodeSelector patching of long-lived Deployments/DaemonSets, so the k8s scheduler placed it on the Longhorn-less tie-breaker node, where it can never attach its PVC and hangs forever.

This matches, but isn't byte-identical to, a standing bug already logged 3× in live-ledger.md for other tests (TestClusterPVCGCDeletesOnlyOrphans 2026-10-30/31/11-01, TestVMAppPurgeDuringFailover's neighbor): "tie-breaker node + Longhorn-dependent workload lands there and hangs forever," there via pillar's WaitForLonghornReady called unconditionally in volumehandlers/csihandler.go (still unconditional in current source — confirmed, not fixed). Here, pillar's own VolumeStatus on edge-dev3 actually shows CREATED_VOLUME/no error, so this specific instance is the Kubernetes-scheduler side of the same broader tie-breaker-scheduling gap rather than the exact documented pillar code path — a new angle on a known, recurring class of bug, not a brand-new independent mechanism. PR #7041 under test only bumps vendored-pillar versions in go.mod/go.sum across edgeview/kube-init/newlog/mmagent, refactors the bump-* Makefile targets, and repoints two Dockerfile base-image digests — it touches none of pkg/pillar/volumehandlers, pkg/pillar/kubeapi, or pkg/kube/kube-init/tiebreaker, so this failure is pre-existing/environmental and unrelated to the PR's changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants