Skip to content

evetest: Proxmox broker reliability fixes - #7020

Merged
rene merged 8 commits into
lf-edge:masterfrom
milan-zededa:evetest-broker-proxmox-fixes
Nov 4, 2026
Merged

rene merged 8 commits into
lf-edge:masterfrom
milan-zededa:evetest-broker-proxmox-fixes

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Description

Reliability fixes for the evetest broker's Proxmox device provider, found
while triaging the evetest nightly runs. One commit per fix:

  • Console websocket crash: the broker died with panic: send on closed channel
    inside go-proxmox's TermWebSocket when a VM's serial console
    connection broke (the latest go-proxmox release has the same code).
    Replaced it with our own termproxy client on top of gorilla/websocket.
  • Failed VM lookups mistaken for missing VMs: PVE answers both a missing
    VM and e.g. a failed worker fork with HTTP 500. Only PVE's "Configuration
    file ... does not exist" now counts as a missing VM. Teardown and failed
    creates no longer leak VMs on other errors, and can remove a placeholder VM
    left locked by a create that PVE could not start.
  • Orphans after a broker restart: at startup, the broker removes the VMs,
    xconnect VNets and import-storage volumes a previous run left behind (one
    crash left 22 VMs running). This relies on one broker per Proxmox host, as
    the installer sets up.
  • Restart on crash: the broker's systemd unit now has
    Restart=on-failure.
  • KVM async page faults: disabled on Proxmox VMs too, as the qemu
    provider already does. A guest hung on an async page fault while the host
    was swapping.
  • Failed Proxmox tasks reported as successful: go-proxmox's Task.Wait
    returns nil for a task that failed, and gives up at the first status poll
    that fails. waitTask now reports failed tasks and tolerates transient poll
    errors. Power operations skip a VM already in the target state and retry
    otherwise; VM lookups are retried too. This caused all 7 "SDN unreachable
    at setup" failures in the nightly runs: the SDN VM had failed to start, but
    the broker reported it as powered on.
  • VM starts during an SDN apply: an SDN apply restarts the zone's dnsmasq,
    and a VM starting meanwhile fails. Starts and SDN applies are now
    serialized.
  • Stale taps after a killed VM: after an OOM kill, the kernel left one of
    the VM's taps behind, and every later VM with the same VM ID failed to start
    with "Device or resource busy". The hookscript now removes the VM's stale
    taps at pre-start.

How to test and validate this PR

Covered by unit tests under evetest/broker/provider/ (fake termproxy server,
fake Proxmox task and VM endpoints): go test ./broker/provider/ from
evetest/.

End-to-end validation is the evetest nightly run against the Proxmox brokers:

  1. No broker crash and no orphaned ev-* VMs on the Proxmox hosts afterwards
    (qm list).
  2. No test failing at setup with "unable to connect to SDN gRPC service on any
    of the uplink IPs".
  3. If a VM start fails on Proxmox, the broker logs a retry carrying PVE's exit
    status instead of reporting the VM as powered on.

Changelog notes

No user-facing changes.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

No documentation changes are needed. The fixes run on the amd64 Proxmox
brokers, but are not yet proven by a nightly run (hence the draft). The
Proxmox hosts are amd64 only, so arm64 is not tested.

🤖 Generated with Claude Code

milan-zededa and others added 7 commits November 3, 2026 15:25
The Proxmox broker crashed with "panic: send on closed channel" inside
go-proxmox's TermWebSocket after a VM's serial console connection broke.
The function's close callback closes its result channels one second
after signalling its goroutines to stop. If the connection is already
broken, the writer goroutine fails to send the websocket close frame
and blocks reporting that on the unbuffered error channel, which
nobody reads any more. The channel is then closed under it and the
panic happens in a library goroutine, so it cannot be recovered and
takes down the broker together with every session it serves. The
latest go-proxmox release has the same code.

Talk the termproxy protocol directly with gorilla/websocket instead,
reusing only the library's authenticated API session. There are no
channels: Read and Write go straight to the websocket, writes are
serialized, every write has a deadline, a failed keepalive closes the
console so a silently dead connection cannot block Read forever, and
Close only uses methods gorilla allows concurrently with everything
else.

The tests run the client against a fake termproxy server, including
a connection that breaks underneath an active console. That is the
scenario that made the old code panic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
PVE answers a lookup of a VM that does not exist with an HTTP 500,
the same status it uses for any other server-side failure, such as
failing to fork an API worker on a host that is out of memory. Two
code paths misread such failures:

- When the create call failed, SetupDevice checked whether the VM
  existed anyway and quietly assumed nothing was created if that
  lookup failed. On a host out of memory, PVE had already written a
  placeholder config holding "lock: create" before failing to fork the
  create worker, and that VM was leaked. Even when found, it could not
  have been deleted, since a plain delete refuses a locked VM.
- Teardown treated any lookup error other than a transport error as
  proof that the VM was gone, and leaked the VM on a 500.

Add removeVM, which treats only PVE's "Configuration file ... does not
exist" as a missing VM, stops the VM if it is running (paused
included) and deletes it, overriding any lock left on it. Teardown and
both failed-setup paths use it. On a failed create the cleanup runs
before vmCreateMutex is released: if nothing was created, the VMID is
free again, and removing it after another device was given that ID
would delete that device's VM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
When the Proxmox broker crashed, every session it served lost its
teardown, and nothing removed the leftovers afterwards: the VMs kept
running and holding host memory, and their xconnect VNets and uploaded
firmware volumes stayed behind. One crash left 22 VMs, 27 VNets and 28
firmware volumes on the host.

Client sessions do not survive a broker restart, so at startup, before
serving any session, remove everything the provider creates: VMs named
with the "ev-" prefix, xconnect VNets of the evetest SDN zone, and
"ev-" volumes in the import storage. The uplink VNet is installer-owned
and stays. Also remove the nameless "lock: create" placeholder VMs that
PVE leaves behind when it fails to start a create worker, unless a task
is still running for them, so that a VM being created from the PVE UI is
spared.

This relies on running a single broker per Proxmox host, as the broker
installer sets up. The sweep is best-effort: failures are logged and the
broker starts anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The broker service had Restart=no, so a single crash took the broker
down until someone restarted it by hand, and every test run in the
meantime failed to connect. Restart it on failure instead. The broker
now removes what the crashed run left behind at startup, so a restarted
broker comes back clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The qemu provider starts device VMs without KVM async page faults
(-kvm-asyncpf,-kvm-asyncpf-int): on a host that overcommits memory, a
guest task faulting on a page the host swapped out can otherwise wait
forever for KVM's "page ready" wakeup. Proxmox VMs were still created
with plain "cpu: host", and a nightly run hit exactly that hang while
the Proxmox host was swapping: an app's qemu inside EVE sat in
kvm_async_pf_task_wait until the test timed out.

PVE's "cpu" option only accepts a fixed set of flags, so disable the
two properties as -global defaults of the host CPU model instead. On
PVE's QEMU 11.0 this leaves the flags of PVE's own -cpu option in
effect (checked over QMP: kvm-asyncpf and kvm-asyncpf-int read false,
kvm-pv-eoi from -cpu still true). Async page faults are x86-only, so
arm64 VMs are left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
waitTask waited with go-proxmox's Task.Wait, which returns nil as soon
as a task stops running, whether it succeeded or failed. A VM start
that failed was therefore reported as successful. In the nightly runs
this happened to seven SDN VMs whose QEMU exited at start ("start
failed: QEMU exited with code 1"): the broker logged that the SDN
device was powered on, then dialed its gRPC tunnel at an IP nobody
held for five minutes until the test failed with "no route to host".
The same blind spot hid failed VM creates, uploads, deletes and SDN
applies. Task.Wait also gave up at the first task status poll that
failed, so one lost response (e.g. a TLS handshake timeout) aborted a
power-off that may well have succeeded.

Poll tasks with our own loop instead: a task that PVE reports as
failed is an error carrying its exit status, up to five failed status
polls in a row are tolerated, and the overall deadline stays.

Reporting failures exposes cases that only worked because failures
were ignored, e.g. starting a VM that already runs. Power operations
now go through setVMPowerState, which reads the VM first and skips the
operation when the VM already is in the target state, and retries the
operation otherwise. The same check makes a retry after an attempt
whose response was lost count as done instead of repeating it. A
failed graceful shutdown is not retried: it means the guest ignored the
ACPI request. VM lookups by device name are retried too, except for a
VM that does not exist; until now a single truncated API response
("unexpected end of JSON input") failed a power-on outright.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Applying SDN changes restarts the evetest zone's dnsmasq. A VM that
starts meanwhile fails: PVE's network script reloads that dnsmasq to
add the VM's DHCP mapping, gets "dnsmasq@evetest.service is not
active, cannot reload", and QEMU exits. Concurrent sessions do this to
each other: one session's SDN apply for a new xconnect network killed
the SDN VM start of another, seven times across the nightly runs on
both Proxmox hosts.

Hold a read lock for each VM start and the write lock for each SDN
apply. With failed tasks now reported, such a start would also be
retried, but there's no reason to lose an attempt to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa milan-zededa added stable Should be backported to stable release(s) testing New test or test framework changes labels Nov 3, 2026
@codecov

codecov Bot commented Nov 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.26%. Comparing base (2727419) to head (86d282f).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7020      +/-   ##
==========================================
+ Coverage   28.76%   29.26%   +0.49%     
==========================================
  Files         544      555      +11     
  Lines       98795   100390    +1595     
==========================================
+ Hits        28421    29377     +956     
- Misses      68086    68770     +684     
+ Partials     2288     2243      -45     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

When a VM's QEMU process is killed (e.g. by the OOM killer), the kernel
can leak a reference to one of its tap devices. The tap then stays
behind with no process holding it, and every later VM that gets the
same VM ID fails to start with "could not configure /dev/net/tun
(tapNxM): Device or resource busy".

Remove any tap of the VM ID in the hookscript's pre-start phase, when
no QEMU runs for it. With a leaked reference, "ip link delete" frees
the name but then blocks waiting for the reference, so it runs
detached and the hook waits only for the name to disappear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa marked this pull request as ready for review November 4, 2026 16:12
@milan-zededa

Copy link
Copy Markdown
Contributor Author

Marking as ready for review and merge.
Successfully tested with PR #7024 and also with one evetest nightly run. The flakiness previously caused by broker is no longer observed.
(evetest harness flakiness is being addressed separately in #7024)

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rene
rene merged commit c28c009 into lf-edge:master Nov 4, 2026
42 of 47 checks passed
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestKonvertSuite: failure analysis

TestKvmToKRepartitionGeometry

Failure
EVE image "evebuild/pr:10.1.0-kvm-amd64" is not available (neither locally nor on Docker Hub). If running against the current HEAD (EVETEST_EVE_VERSION not explicitly set), build the image first with: make eve
Claude's conclusion

Confirmed — the PR exclusively modifies Proxmox VM lifecycle/broker internals (taps, SDN-apply races, console logging, retry logic), none of which touches image pull/registry or requirePinnedInitialVersion resolution. This failure occurred entirely in test setup before any Proxmox VM was created, so it's unrelated to this PR.

Have everything needed to write the root cause now.

Root cause

TestKvmToKRepartitionGeometry pins INITIAL_EVE_VERSION=10.1.0 (via requirePinnedInitialVersion, needing a real prior-release EVE image to convert from) and failed at setup.go:564 because evebuild/pr:10.1.0-kvm-amd64 doesn't exist on Docker Hub or locally on the broker ("pull access denied... repository does not exist"). Live checks confirm this failed purely in image-prep: evetest status shows "No EVE devices found", and evetest eve info/evetest sdn status both report no device was ever onboarded and no SDN client initialized.

This is the same known, recurring environment/test-config gap already logged repeatedly in live-ledger.md (first seen 2026-10-29, recurring 2026-10-30, 2026-10-31, and earlier today) — konvert tests calling requirePinnedInitialVersion resolve the pinned 10.1.0 tag through an unset repo parameter, which lands in a repo that only holds numeric CI build-id tags, never a real 10.1.0 semantic-release tag. The one new wrinkle tonight is the resolved repo name itself: previous nightly runs hit evebuild/nightly:10.1.0-kvm-amd64, while this PR-triggered run hit evebuild/pr:10.1.0-kvm-amd64 — same underlying mechanism, just a different default repo depending on run context, so it's the identical config gap, not a new bug.

The PR (#7020) under test only touches evetest/broker/provider/proxmox*.go and related deploy/term files (stale-tap cleanup, SDN-apply VM-start guarding, Proxmox task/retry handling, console logging) — nothing related to image-pull/registry resolution or requirePinnedInitialVersion. Since the failure happened entirely in setup before any Proxmox VM was created or any of this PR's changed code paths were exercised, this failure is environmental/pre-existing and unrelated to the PR's changes.

TestBackupCorruptRestore

Failure
EVE image "evebuild/pr:10.1.0-kvm-amd64" is not available (neither locally nor on Docker Hub). If running against the current HEAD (EVETEST_EVE_VERSION not explicitly set), build the image first with: make eve
Claude's conclusion

Confirmed: exact same failure mode and stack trace (setup.go:564 pushEVEImageToBroker → prepareImageForEVEDevice) as TestKvmToKRepartitionGeometry investigated earlier in this run, and no device/SDN was ever created here either.

Root cause: TestBackupCorruptRestore pins INITIAL_EVE_VERSION=10.1.0 via requirePinnedInitialVersion, and with initialEVERepoParamKey unset for this PR-triggered run it resolves to evebuild/pr — a CI repo that only holds numeric build-id tags, never a semantic release tag like 10.1.0-kvm-amd64, so the pull fails before any device/SDN is provisioned (confirmed live: evetest eve info/sdn status both report nothing was ever onboarded). This is the identical root cause already diagnosed for TestKvmToKRepartitionGeometry in this same run. Per live-ledger.md, this exact failure mode has been known/recurring since at least 2026-10-29 (where it hit TestBackupCorruptRestore itself plus four other TestKonvertSuite tests) and has recurred on most nightly runs since (10-30, 10-31, and earlier today), always as evebuild/nightly rather than evebuild/pr — so it's a long-standing environment/test-config gap, not new. PR #7020 only touches Proxmox broker/VM-lifecycle internals, unrelated to image-pull/registry resolution, and the failure happened entirely in setup before any of that code ran — so this failure is unrelated to the PR.

Shrink

Failure
EVE image "evebuild/pr:10.1.0-kvm-amd64" is not available (neither locally nor on Docker Hub). If running against the current HEAD (EVETEST_EVE_VERSION not explicitly set), build the image first with: make eve
Claude's conclusion

Already confirmed in both prior investigations in this same run — PR #7020 only touches Proxmox broker internals, unrelated to image resolution. No need to re-read the full diff.

Root cause

This is the identical root cause already diagnosed twice earlier in this same run (TestKvmToKRepartitionGeometry, TestBackupCorruptRestore): Shrink is a TestKonvertSuite test that pins INITIAL_EVE_VERSION=10.1.0 via requirePinnedInitialVersion, and with initialEVERepoParamKey unset for this PR-triggered run it resolves to the evebuild/pr repo — which, like evebuild/nightly in nightly runs, only holds numeric CI build-id tags, never a semantic release tag like 10.1.0-kvm-amd64. Live checks confirm this failed purely in image-prep before any device/SDN provisioning: evetest status shows "No EVE devices found," and evetest eve info/evetest sdn status both report no onboarded device and an uninitialized SDN client.

Per live-ledger.md, this exact failure mode for Shrink is a long-standing known/recurring issue going back to at least 2026-10-29 (run #1), and it recurred on most nightly runs since (10-30, 10-31, and earlier in today's own run, both via evebuild/nightly) — note this is distinct from a separate, later Shrink failure in the ledger (a VM-paused-on-ENOSPC issue after blob-reused redeploy), which is an unrelated bug that happened post-provisioning.

PR #7020 only touches Proxmox VM lifecycle/broker internals (stale-tap cleanup, SDN-apply VM-start guarding, task/retry handling, console logging) — nothing related to image-pull/registry resolution — and this failure occurred entirely in setup before any of that code ran, so it is environmental/pre-existing and unrelated to the PR.

PersistTooFull

Failure
EVE image "evebuild/pr:10.1.0-kvm-amd64" is not available (neither locally nor on Docker Hub). If running against the current HEAD (EVETEST_EVE_VERSION not explicitly set), build the image first with: make eve
Claude's conclusion

I have everything needed to write the root cause now.

Root cause

PersistTooFull failed identically to three other TestKonvertSuite tests already investigated earlier in this run (TestKvmToKRepartitionGeometry, TestBackupCorruptRestore, Shrink) — same error, same codepath (setup.go:564 → pushEVEImageToBroker → prepareImageForEVEDevice), occurring before any device/SDN provisioning. Live checks confirm this: evetest status shows "No EVE devices found," and evetest eve info/evetest sdn status both report no device was ever onboarded and no SDN client initialized.

PersistTooFull (like the other konvert tests) pins INITIAL_EVE_VERSION=10.1.0 via requirePinnedInitialVersion; with initialEVERepoParamKey unset for this PR-triggered run, it resolves to the evebuild/pr repo, which (like evebuild/nightly in nightly runs) only holds numeric CI build-id tags, never a semantic release tag like 10.1.0-kvm-amd64, so the pull fails before any provisioning occurs.

This is a known, long-recurring environment/test-config gap — the ledger shows PersistTooFull itself hitting this exact issue on prior nightlies (going back to at least 2026-10-29), always via evebuild/nightly there vs. evebuild/pr tonight (same mechanism, different default repo for this run context).

PR #7020 only touches evetest/broker/provider/proxmox*.go and related deploy files (stale-tap cleanup, SDN-apply VM-start guarding, Proxmox task/retry handling, console logging) — nothing related to image-pull/registry resolution — and the failure occurred entirely in setup before any of that code ran, so this is environmental/pre-existing and unrelated to the PR.

ZFSPersist

Failure
EVE image "evebuild/pr:10.1.0-kvm-amd64" is not available (neither locally nor on Docker Hub). If running against the current HEAD (EVETEST_EVE_VERSION not explicitly set), build the image first with: make eve
Claude's conclusion

Confirmed identical signature: same stack trace (pushEVEImageToBroker → setup.go:564 → prepareImageForEVEDevice), and evetest status/eve info both confirm no device was ever provisioned.

Root cause

ZFSPersist (part of TestKvmToKRepartitionRefused) pins INITIAL_EVE_VERSION=10.1.0 via requirePinnedInitialVersion, and with initialEVERepoParamKey unset for this PR-triggered run it resolves to the evebuild/pr repo — which only holds numeric CI build-id tags, never a semantic-release tag like 10.1.0-kvm-amd64 — so the pull fails before any device/SDN provisioning (evetest status/eve info confirm no device was ever onboarded). This is the same root cause already diagnosed for every other failure in this run (TestKvmToKRepartitionGeometry, TestBackupCorruptRestore, Shrink, PersistTooFull) — same error string, same setup.go:564 codepath. It's also a known, long-recurring issue per live-ledger.md: this exact ZFSPersist failure (then via evebuild/nightly) was first logged on 2026-10-29 alongside the other konvert tests, and has recurred on subsequent nightlies (10-30, 10-31, and earlier today). PR #7020 only touches Proxmox broker/VM-lifecycle internals (stale-tap cleanup, SDN-apply guarding, retry handling) — unrelated to image-pull/registry resolution — and the failure occurred entirely in setup before any PR-changed code ran, so this is environmental/pre-existing and unrelated to the PR.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterSuite: failure analysis

TestAppInstancePurge

Failure
Device "edge-dev": expected 0 reboot(s) but observed 1
Claude's conclusion

Confirmed via ledger: this exact failure has happened before, with identical mechanism.

Root cause

Live NodeAgentStatus/pillar logs on edge-dev show the reboot was a deliberate EVE-K cluster-mode transition, not a crash: zedagent logged cluster departing (Valid true->false) at 20:03:42.899, and nodeagent recorded RebootReason: BootReasonKubeTransition, Transition from cluster mode to single node at 20:03:44, with a clean, orderly shutdown of all services (no panic, no watchdog kill). The preceding test in this suite run, TestNativeKubeAppNetworking, left edge-dev in single-node eve-k cluster mode, but TestAppInstancePurge's own applied config (version 9) doesn't carry that cluster config forward on the reused device, so nodeagent correctly detected the mode mismatch and rebooted to apply it — tripping the harness's checkRebootCounts assertion of 0 expected reboots.

This is a known, recurring issue: live-ledger.md records the identical TestAppInstancePurge failure with the exact same BootReasonKubeTransition mechanism on 2026-10-30 (first occurrence) and again on 2026-11-01, both attributed to the same "cluster config not carried forward across back-to-back tests on a reused device" gap between TestNativeKubeAppNetworking and TestAppInstancePurge. Tonight's failure is a third consecutive reproduction of the same test-isolation/device-reuse gap.

It does not share a root cause with any other failure already logged in this run. Regarding PR #7020: its diff only touches evetest/broker (Proxmox VM provider task-polling, SDN-apply locking, orphan-tap cleanup) — entirely unrelated to cluster-mode/kube-transition logic — so this failure looks pre-existing/environmental (a test-harness device-reuse gap), not caused by this PR.

TestMgmtProxy

Failure

wait for the raw container pod to become Ready
Unexpected error:
    <*fmt.wrapError | 0x21ec62d6d820>: 
    failed to execute script over SSH for EVE device edge-dev1: Process exited with status 141 (stderr: )
    {
        msg: "failed to execute script over SSH for EVE device edge-dev1: Process exited with status 141 (stderr: )",
        err: <*ssh.ExitError | 0x21ec6262e2c0>{
            Waitmsg: {status: 141, signal: "", msg: "", lang: ""},
        },
    }
occurred
Claude's conclusion

PR #7020's diff has no hits for mgmtproxy/registry-mirror/proxy logic — consistent with the earlier finding that it only touches Proxmox broker VM provisioning/SDN-apply locking, unrelated to this test's failure path.

Root cause

This is the same recurring TestMgmtProxy bug seen repeatedly since the very first nightly run (2026-10-29) and in nearly every run since, including multiple times in this same run's prior ledger entries. Live evidence on edge-dev1 reproduces the identical mechanism exactly: the mgmtproxy-test-container pod is stuck in ImagePullBackOff, with kubelet failing Head "http://192.168.170.2:5000/...": dial tcp 192.168.170.2:5000: connect: no route to host (11 retries over ~3 minutes); ip route get 192.168.170.2 and ip route get 1.1.1.1 both resolve via eth1/cluster network (the test's deliberately-poisoned main routing table), and /run/mgmtproxy-containerd-env has only HTTPS_PROXY=http://127.0.0.1:5443 set, no HTTP_PROXY — so the plain-http:// registry-mirror pull bypasses mgmtproxy's loopback listener entirely. The SSH exit 141 is just kubectl wait's wrapper hitting SIGPIPE once its timeout expired against the never-Ready pod. This doesn't share a root cause with the earlier TestAppInstancePurge failure in this run (a cluster-mode-not-carried-forward reboot). PR #7020's diff touches only Proxmox broker VM provisioning/SDN-apply locking (confirmed no matches for mgmtproxy/proxy/registry-mirror code), so this is pre-existing/environmental test-infra gap, unrelated to the PR.

TestClusterVMApp

Failure
Timed out waiting for cluster nodes to become ready in cluster "test-cluster" (device "edge-dev2")
Claude's conclusion

Confirmed — PR #7020 only touches evetest/broker (Proxmox provider), entirely unrelated to pillar's kube cluster-transition/k3s join logic.

Root cause

This TestClusterVMApp failure (timeout waiting for cluster nodes in test-cluster) is the same known, recurring edge-dev3 join/reboot-retry "not authorized" bug already seen three times in live-ledger.md (2026-10-29/30, 2026-11-01, 2026-11-02) — this is now (at least) its fourth occurrence. Live evidence matches exactly: TestClusterToSingleConversion (the prior test in this run) converted edge-dev3 to a standalone node and passed cleanly at 21:12:52; TestClusterVMApp then re-applied cluster config (versions 6/7) to re-form the 3-node cluster, but edge-dev3 rebooted at 21:18:38 (NodeAgentStatus shows RebootReason: BootReasonKubeTransition, Reboot after retry cluster transition attempt 1, RestartCounter:2) and has been crash-looping every few minutes since, with /persist/kubelog/k3s.log repeatedly logging Skipping cluster configuration validation: not authorized → failed to bootstrap cluster data: not authorized. Live kubectl get nodes confirms only edge-dev1/edge-dev2 are Ready, with edge-dev3 permanently excluded, so the harness's 30-minute cluster-ready wait (checkpoint initial-config-applied through to the timeout at 21:43:17) correctly timed out waiting for a node that will never rejoin. This does not share a root cause with the earlier TestAppInstancePurge/TestMgmtProxy failures investigated in this run. PR #7020's diff touches only evetest/broker/provider/proxmox.go (Proxmox VM start/SDN-apply/orphan-cleanup locking) and unrelated vendor/deploy files — nothing in pillar's kube-transition or k3s join logic — so this is pre-existing and unrelated to the PR; it looks like an unfixed EVE-K node-rejoin bug after a standalone→cluster re-conversion.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestSecuritySuite: failure analysis

TestVaultKeyModeRecoveryOnZFS

Failure

a mode that opens the vault must not cost a second attempt
Expected
    <int>: 1
to equal
    <int>: 2
Claude's conclusion

No matches — confirms unrelated. I have enough to write the conclusion.

Root cause

This is a known, recurring test-expectation bug, not a new regression. Live evidence on edge-dev confirms the vault genuinely unsealed cleanly on the first try after reboot (VaultStatus shows UnlockMethod: tpm-local-sealed, no error) — the harness itself just reconnected over SSH, ran the assertion at vault_keymode_test.go:298, and failed comparing actual 1 vs. expected 2, i.e. the test's hardcoded expectation that "a mode that opens the vault" should cost 2 attempts is wrong; the device correctly needed only 1.

Per live-ledger.md, this identical assertion/line and identical 1 vs 2 mismatch has failed on TestVaultKeyModeRecoveryOnZFS/TestVaultKeyModeRecoveryOnExt4 in every nightly run since 2026-10-29 (runs #1–#7, i.e. 10-29, 10-30, 10-31, 11-01, 11-02, 11-03, and now 11-04) — every prior investigation reached the same conclusion: the test's expected attempt-count constant is stale, not a vaultmgr/pillar defect.

This run's PR #7020 diff touches only the evetest Proxmox broker/harness code (broker/provider/proxmox*.go, broker/main.go, deploy scripts) — nothing in pkg/pillar's vault/TPM code or in vault_keymode_test.go itself — so it's unrelated to this failure; this is the same long-standing test bug reproducing again, independent of any in-flight code change.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestAppsSuite: failure analysis

TestVGAPassthroughNoHostAccess

Failure
Host processes interfered with PCI devices while the VGA controller 0000:00:01.0 was passed through to the application:
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=10419 parent=runc:[1:CHILD]/10418 grandparent=runc:[0:PARENT]/10414
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=10419 parent=runc:[1:CHILD]/10418 grandparent=runc:[0:PARENT]/10414
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=10419 parent=runc:[1:CHILD]/10418 grandparent=runc:[0:PARENT]/10414
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=10419 parent=runc:[1:CHILD]/10418 grandparent=runc:[0:PARENT]/10414
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=10419 parent=runc:[1:CHILD]/10418 grandparent=runc:[0:PARENT]/10414
  [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=10419 parent=runc:[1:CHILD]/10418 grandparent=runc:[0:PARENT]/10414
Kernel messages mentioning 0000:00:01.0:
[    1.465493] [    T1] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint
[    1.469222] [    T1] pci 0000:00:01.0: BAR 0 [mem 0x80000000-0x80ffffff pref]
[    1.469839] [    T1] pci 0000:00:01.0: BAR 2 [mem 0x8264d000-0x8264dfff]
[    1.470494] [    T1] pci 0000:00:01.0: ROM [mem 0xffff0000-0xffffffff pref]
[    1.471169] [    T1] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.784633] [    T1] pci 0000:00:01.0: vgaarb: setting as boot VGA device
[    1.784633] [    T1] pci 0000:00:01.0: vgaarb: bridge control possible
[    1.784633] [    T1] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.968223] [    T1] pci 0000:00:01.0: Adding to iommu group 1
[  158.279425] [ T4546] vfio-pci 0000:00:01.0: vgaarb: deactivate vga console
[  158.279435] [ T4546] vfio-pci 0000:00:01.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  193.842751] [ T7579] vfio-pci 0000:00:01.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[  194.166639] [ T7579] vfio-pci 0000:00:01.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  491.860193] [T10442] vfio-pci 0000:00:01.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
Claude's conclusion

Confirmed the whitelist logic (isViolation checking strings.HasPrefix(a.Comm, "qemu-system-")) in evetest/tests/apps/pciaccess_helpers_test.go:407-412 is unchanged and still doesn't account for QEMU's vCPU thread renaming to CPU N/KVM. Live ps on the device confirms PID 10419 is vga-passthrough-app's own qemu-system-x86 process and TID 10442 (reported as "10419" comm-wise, matching pattern) is its CPU 0/KVM vCPU thread — not a host process.

Root cause: This is the same well-known false positive already documented in the ledger for this test. pciAccess.isViolation() only whitelists config-space writes from processes whose comm starts with "qemu-system-", but QEMU renames its vCPU thread to CPU N/KVM (while sharing the owning process's PID), so the test harness misclassifies the app's own vCPU thread writing to its own passed-through VGA device as "host interference." Live ps -eLo pid,tid,ppid,comm on edge-dev confirms PID 10419 is vga-passthrough-app's own qemu-system-x86_64 process and TID 10442 is its CPU 0/KVM thread — not an external host process — and the kernel's "Invalid PCI ROM header signature" messages are benign guest-side ROM re-probe noise during app restart. The ledger traces this exact bug back to the test's introduction on 2026-09-27 (commit c2539a9de), recurring repeatedly since (2026-09-27, 2026-10-29, 2026-10-30, and again just last night, 2026-11-03/04). It's unchanged and still unfixed. This run's PR #7020 only touches evetest/broker/Proxmox provider code (VM lifecycle, not the pciaccess tracer or guest-side logic), so this failure is unrelated to that diff — it's the same standing, known-flaky test harness bug.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterResilienceSuite: failure analysis

TestClusterPVCGCDeletesOnlyOrphans

Failure

Timed out after 900.001s.
The function passed to Eventually failed at /evetest/tests/cluster/pvcgc_test.go:115 with:
node "edge-dev2" has no created VolumeStatus for volume 7a4bcd20-f7b3-41a7-bc43-34b2a5a9ad58
Expected
    <bool>: false
to be true
Claude's conclusion

Root cause: This is the identical tie-breaker/Longhorn gating bug diagnosed three times already tonight for this same test — now a fourth occurrence. Live evidence on edge-dev2: kubectl get nodes shows it SchedulingDisabled (the tie-breaker/witness node), its VolumeStatus for 7a4bcd20-f7b3-41a7-bc43-34b2a5a9ad58 (pvc-gc-persistent) is permanently stuck at State:109 (CREATING_VOLUME), and volumemgr is continuously logging WaitForLonghornReady: not ready yet: longhorn daemonset:longhorn-manager missing on this node every ~5s right now with no progress — because Longhorn's daemonsets are deliberately never scheduled on the tie-breaker node, so kubeapi.WaitForLonghornReady can never succeed there, and volumehandlers/csihandler.go's PrepareVolume calls it unconditionally instead of checking EdgeNodeClusterConfig.IsTieBreakerNode as documented in longhorninfo.go:720-721.

Per the ledger, this is a known/recurring issue first seen tonight (2026-11-04, run #7) — this is the 4th occurrence in this same run alone (after volumes 18355a22-..., efb4e4cd-..., e43bad25-...), confirming it's a standing, deterministic code gap rather than a flake. The PR under test (#7020) only touches evetest/broker (Proxmox VM-provider locking/task-wait logic) and evetest/deploy/proxmox — it never touches pkg/pillar/kubeapi or pkg/pillar/volumehandlers — so this failure is unrelated to the PR's changes.

TestVMAppPurgeDuringFailover

Failure

Timed out after 900.001s.
The function passed to Eventually failed at /evetest/tests/cluster/purge_during_failover_test.go:169 with:
expected exactly one VMIRS for the app, found []
Expected
    <[]string | len:0, cap:0>: nil
to have length 1
Claude's conclusion

Confirmed, live and ongoing right now. This is identical to the earlier-investigated failure in this same run and the long-standing known bug. The PR diff only touches evetest/broker and evetest/deploy/proxmox (infra), not pillar code, so it's unrelated.

Root cause

This is the same getKubeAppActivateStatus zedmanager/zedkube "purge-after-failover" deadlock already diagnosed earlier in this very run (live-investigation.md, same test, same failure line purge_during_failover_test.go:169, same "found []" signature). Live confirmation just now on edge-dev2: AppInstanceStatus for failover-purge-app (d2cc3519-...) shows PurgeInprogress:4, PurgeStartedAt:2026-11-04T22:58:56Z, Activated:false, IsDesignatedNodeID:false; zedmanager (pkg/pillar/cmd/zedmanager/zedmanager.go:2059) is logging an unbroken stream of getKubeAppActivateStatus(...): not designated, node e73f8ed8-..., onTheDevice false, statusRunning false every ~17-20s through the present moment (23:13–23:14), and kubectl -n eve-kube-app get vmirs,vmi,vm returns "No resources found", matching the test's "found []". evetest status confirms edge-dev1 (the designated node) is EVE_DEVICE_STATE_SUSPECT, deliberately downed for this failover scenario — so once the purge tore down the old VMIRS, nothing in the cluster can originate a replacement.

Per live-ledger.md, this is a known, recurring issue going back to 2026-10-29, recurring again on 2026-10-30, 2026-10-31, and (per the earlier conclusion in this run's own live-investigation.md) again tonight prior to this occurrence — this is effectively a second confirmation within the same run of the same standing, unfixed bug. The PR under test (#7020) only touches evetest/broker and evetest/deploy/proxmox (Proxmox VM-provider locking/SDN-apply logic), not pkg/pillar/cmd/zedmanager or anything in the purge/activation path, so this failure is unrelated to the PR's changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s) testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants