Repository navigation
evetest: Proxmox broker reliability fixes - #7020
Conversation
The Proxmox broker crashed with "panic: send on closed channel" inside go-proxmox's TermWebSocket after a VM's serial console connection broke. The function's close callback closes its result channels one second after signalling its goroutines to stop. If the connection is already broken, the writer goroutine fails to send the websocket close frame and blocks reporting that on the unbuffered error channel, which nobody reads any more. The channel is then closed under it and the panic happens in a library goroutine, so it cannot be recovered and takes down the broker together with every session it serves. The latest go-proxmox release has the same code. Talk the termproxy protocol directly with gorilla/websocket instead, reusing only the library's authenticated API session. There are no channels: Read and Write go straight to the websocket, writes are serialized, every write has a deadline, a failed keepalive closes the console so a silently dead connection cannot block Read forever, and Close only uses methods gorilla allows concurrently with everything else. The tests run the client against a fake termproxy server, including a connection that breaks underneath an active console. That is the scenario that made the old code panic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
PVE answers a lookup of a VM that does not exist with an HTTP 500, the same status it uses for any other server-side failure, such as failing to fork an API worker on a host that is out of memory. Two code paths misread such failures: - When the create call failed, SetupDevice checked whether the VM existed anyway and quietly assumed nothing was created if that lookup failed. On a host out of memory, PVE had already written a placeholder config holding "lock: create" before failing to fork the create worker, and that VM was leaked. Even when found, it could not have been deleted, since a plain delete refuses a locked VM. - Teardown treated any lookup error other than a transport error as proof that the VM was gone, and leaked the VM on a 500. Add removeVM, which treats only PVE's "Configuration file ... does not exist" as a missing VM, stops the VM if it is running (paused included) and deletes it, overriding any lock left on it. Teardown and both failed-setup paths use it. On a failed create the cleanup runs before vmCreateMutex is released: if nothing was created, the VMID is free again, and removing it after another device was given that ID would delete that device's VM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
When the Proxmox broker crashed, every session it served lost its teardown, and nothing removed the leftovers afterwards: the VMs kept running and holding host memory, and their xconnect VNets and uploaded firmware volumes stayed behind. One crash left 22 VMs, 27 VNets and 28 firmware volumes on the host. Client sessions do not survive a broker restart, so at startup, before serving any session, remove everything the provider creates: VMs named with the "ev-" prefix, xconnect VNets of the evetest SDN zone, and "ev-" volumes in the import storage. The uplink VNet is installer-owned and stays. Also remove the nameless "lock: create" placeholder VMs that PVE leaves behind when it fails to start a create worker, unless a task is still running for them, so that a VM being created from the PVE UI is spared. This relies on running a single broker per Proxmox host, as the broker installer sets up. The sweep is best-effort: failures are logged and the broker starts anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
The broker service had Restart=no, so a single crash took the broker down until someone restarted it by hand, and every test run in the meantime failed to connect. Restart it on failure instead. The broker now removes what the crashed run left behind at startup, so a restarted broker comes back clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
The qemu provider starts device VMs without KVM async page faults (-kvm-asyncpf,-kvm-asyncpf-int): on a host that overcommits memory, a guest task faulting on a page the host swapped out can otherwise wait forever for KVM's "page ready" wakeup. Proxmox VMs were still created with plain "cpu: host", and a nightly run hit exactly that hang while the Proxmox host was swapping: an app's qemu inside EVE sat in kvm_async_pf_task_wait until the test timed out. PVE's "cpu" option only accepts a fixed set of flags, so disable the two properties as -global defaults of the host CPU model instead. On PVE's QEMU 11.0 this leaves the flags of PVE's own -cpu option in effect (checked over QMP: kvm-asyncpf and kvm-asyncpf-int read false, kvm-pv-eoi from -cpu still true). Async page faults are x86-only, so arm64 VMs are left alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
waitTask waited with go-proxmox's Task.Wait, which returns nil as soon
as a task stops running, whether it succeeded or failed. A VM start
that failed was therefore reported as successful. In the nightly runs
this happened to seven SDN VMs whose QEMU exited at start ("start
failed: QEMU exited with code 1"): the broker logged that the SDN
device was powered on, then dialed its gRPC tunnel at an IP nobody
held for five minutes until the test failed with "no route to host".
The same blind spot hid failed VM creates, uploads, deletes and SDN
applies. Task.Wait also gave up at the first task status poll that
failed, so one lost response (e.g. a TLS handshake timeout) aborted a
power-off that may well have succeeded.
Poll tasks with our own loop instead: a task that PVE reports as
failed is an error carrying its exit status, up to five failed status
polls in a row are tolerated, and the overall deadline stays.
Reporting failures exposes cases that only worked because failures
were ignored, e.g. starting a VM that already runs. Power operations
now go through setVMPowerState, which reads the VM first and skips the
operation when the VM already is in the target state, and retries the
operation otherwise. The same check makes a retry after an attempt
whose response was lost count as done instead of repeating it. A
failed graceful shutdown is not retried: it means the guest ignored the
ACPI request. VM lookups by device name are retried too, except for a
VM that does not exist; until now a single truncated API response
("unexpected end of JSON input") failed a power-on outright.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Applying SDN changes restarts the evetest zone's dnsmasq. A VM that starts meanwhile fails: PVE's network script reloads that dnsmasq to add the VM's DHCP mapping, gets "dnsmasq@evetest.service is not active, cannot reload", and QEMU exits. Concurrent sessions do this to each other: one session's SDN apply for a new xconnect network killed the SDN VM start of another, seven times across the nightly runs on both Proxmox hosts. Hold a read lock for each VM start and the write lock for each SDN apply. With failed tasks now reported, such a start would also be retried, but there's no reason to lose an attempt to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7020 +/- ##
==========================================
+ Coverage 28.76% 29.26% +0.49%
==========================================
Files 544 555 +11
Lines 98795 100390 +1595
==========================================
+ Hits 28421 29377 +956
- Misses 68086 68770 +684
+ Partials 2288 2243 -45 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
When a VM's QEMU process is killed (e.g. by the OOM killer), the kernel can leak a reference to one of its tap devices. The tap then stays behind with no process holding it, and every later VM that gets the same VM ID fails to start with "could not configure /dev/net/tun (tapNxM): Device or resource busy". Remove any tap of the VM ID in the hookscript's pre-start phase, when no QEMU runs for it. With a leaked reference, "ip link delete" frees the name but then blocks waiting for the reference, so it runs detached and the hook waits only for the name to disappear. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
|
Marking as ready for review and merge. |
TestKonvertSuite: failure analysisTestKvmToKRepartitionGeometryFailureClaude's conclusionConfirmed — the PR exclusively modifies Proxmox VM lifecycle/broker internals (taps, SDN-apply races, console logging, retry logic), none of which touches image pull/registry or Have everything needed to write the root cause now. Root cause
This is the same known, recurring environment/test-config gap already logged repeatedly in The PR (#7020) under test only touches TestBackupCorruptRestoreFailureClaude's conclusionConfirmed: exact same failure mode and stack trace ( Root cause: ShrinkFailureClaude's conclusionAlready confirmed in both prior investigations in this same run — PR #7020 only touches Proxmox broker internals, unrelated to image resolution. No need to re-read the full diff. Root causeThis is the identical root cause already diagnosed twice earlier in this same run ( Per PR #7020 only touches Proxmox VM lifecycle/broker internals (stale-tap cleanup, SDN-apply VM-start guarding, task/retry handling, console logging) — nothing related to image-pull/registry resolution — and this failure occurred entirely in setup before any of that code ran, so it is environmental/pre-existing and unrelated to the PR. PersistTooFullFailureClaude's conclusionI have everything needed to write the root cause now. Root cause
This is a known, long-recurring environment/test-config gap — the ledger shows PR #7020 only touches ZFSPersistFailureClaude's conclusionConfirmed identical signature: same stack trace ( Root cause
|
TestNodeClusterSuite: failure analysisTestAppInstancePurgeFailureClaude's conclusionConfirmed via ledger: this exact failure has happened before, with identical mechanism. Root causeLive This is a known, recurring issue: It does not share a root cause with any other failure already logged in this run. Regarding PR #7020: its diff only touches TestMgmtProxyFailureClaude's conclusionPR #7020's diff has no hits for mgmtproxy/registry-mirror/proxy logic — consistent with the earlier finding that it only touches Proxmox broker VM provisioning/SDN-apply locking, unrelated to this test's failure path. Root causeThis is the same recurring TestClusterVMAppFailureClaude's conclusionConfirmed — PR #7020 only touches Root causeThis |
TestSecuritySuite: failure analysisTestVaultKeyModeRecoveryOnZFSFailureClaude's conclusionNo matches — confirms unrelated. I have enough to write the conclusion. Root causeThis is a known, recurring test-expectation bug, not a new regression. Live evidence on Per This run's PR #7020 diff touches only the |
TestAppsSuite: failure analysisTestVGAPassthroughNoHostAccessFailureClaude's conclusionConfirmed the whitelist logic ( Root cause: This is the same well-known false positive already documented in the ledger for this test. |
TestNodeClusterResilienceSuite: failure analysisTestClusterPVCGCDeletesOnlyOrphansFailureClaude's conclusionRoot cause: This is the identical tie-breaker/Longhorn gating bug diagnosed three times already tonight for this same test — now a fourth occurrence. Live evidence on Per the ledger, this is a known/recurring issue first seen tonight (2026-11-04, run #7) — this is the 4th occurrence in this same run alone (after volumes TestVMAppPurgeDuringFailoverFailureClaude's conclusionConfirmed, live and ongoing right now. This is identical to the earlier-investigated failure in this same run and the long-standing known bug. The PR diff only touches evetest/broker and evetest/deploy/proxmox (infra), not pillar code, so it's unrelated. Root causeThis is the same Per |
Description
Reliability fixes for the evetest broker's Proxmox device provider, found
while triaging the evetest nightly runs. One commit per fix:
panic: send on closed channelinside go-proxmox's
TermWebSocketwhen a VM's serial consoleconnection broke (the latest go-proxmox release has the same code).
Replaced it with our own termproxy client on top of gorilla/websocket.
VM and e.g. a failed worker fork with HTTP 500. Only PVE's "Configuration
file ... does not exist" now counts as a missing VM. Teardown and failed
creates no longer leak VMs on other errors, and can remove a placeholder VM
left locked by a create that PVE could not start.
xconnect VNets and import-storage volumes a previous run left behind (one
crash left 22 VMs running). This relies on one broker per Proxmox host, as
the installer sets up.
Restart=on-failure.provider already does. A guest hung on an async page fault while the host
was swapping.
Task.Waitreturns nil for a task that failed, and gives up at the first status poll
that fails.
waitTasknow reports failed tasks and tolerates transient pollerrors. Power operations skip a VM already in the target state and retry
otherwise; VM lookups are retried too. This caused all 7 "SDN unreachable
at setup" failures in the nightly runs: the SDN VM had failed to start, but
the broker reported it as powered on.
and a VM starting meanwhile fails. Starts and SDN applies are now
serialized.
the VM's taps behind, and every later VM with the same VM ID failed to start
with "Device or resource busy". The hookscript now removes the VM's stale
taps at
pre-start.How to test and validate this PR
Covered by unit tests under
evetest/broker/provider/(fake termproxy server,fake Proxmox task and VM endpoints):
go test ./broker/provider/fromevetest/.End-to-end validation is the evetest nightly run against the Proxmox brokers:
ev-*VMs on the Proxmox hosts afterwards(
qm list).of the uplink IPs".
status instead of reporting the VM as powered on.
Changelog notes
No user-facing changes.
PR Backports
Checklist
And the last but not least:
check them.
No documentation changes are needed. The fixes run on the amd64 Proxmox
brokers, but are not yet proven by a nightly run (hence the draft). The
Proxmox hosts are amd64 only, so arm64 is not tested.
🤖 Generated with Claude Code