Skip to content

evetest: fix recurring test and harness failures - #7024

Merged
eriknordmark merged 16 commits into
lf-edge:masterfrom
milan-zededa:evetest-harness-test-fixes
Nov 4, 2026
Merged

eriknordmark merged 16 commits into
lf-edge:masterfrom
milan-zededa:evetest-harness-test-fixes

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes for recurring evetest nightly failures that are caused by the tests or
the test framework, not by EVE. One commit per fix:

  • Konvert tests pinned to a release: pull the initial release from
    lfedge/eve instead of the run's CI repo, which has no release tags.
  • Cluster mode inherited through device reuse: the harness expects EVE's
    reboot when a test drops the cluster config a reused device inherited from
    the previous test.
  • Registry mirrors inherited through device reuse: a device whose
    registry mirrors the next test does not want is not reused (broke
    TestMgmtProxy).
  • Stale cluster readiness: cluster readiness waits ignore reports for
    another cluster, and the cluster tests retry kubectl apply on transient
    API server errors.
  • VGA passthrough test: QEMU's vCPU threads (CPU <n>/KVM) are no longer
    flagged as host processes touching the passed-through device.
  • Vault key-mode tests: count only newlogd's keep stream, since the upload
    copy of a log record is deleted once sent and made the count drop.
  • Network instance watches: don't wait for a message that an earlier wait
    already consumed (ACL, VLAN, STP and routing tests).
  • TestNetworkConfigFallback: lower timer.port.testinterval, which a DPC
    retest can depend on.
  • Reused clusters: a test reusing the devices of a running cluster keeps
    that cluster's ID and join token instead of generating new ones. EVE never
    re-reported a new ID (broke TestThreeNodesPortScan), and a node rejoining
    the cluster with a new token was rejected as "not authorized" (broke
    TestClusterVMApp).
  • Broker power operations: the harness gives powering a VM on or off
    and rebooting it 5 minutes instead of 1, which a slow Proxmox task with
    the broker's retries didn't fit in.
  • Waiting for device certificates: keep waiting while Adam answers 400,
    which it does until the device posts its certificates (broke
    TestInfoRetriedAfterServerError setup on a slow host).
  • TestPNAC: lower timer.port.testbetterinterval, which the return to a
    DPC that failed its first test depends on.
  • TestTieBreakerCluster: create its devices from scratch, since a reused
    cluster turned the test into converting a running cluster to one with a
    tie-breaker.
  • CI downloads: retry go install, the evetest CLI build and the EVE
    image artifact download, whose one-off failures failed whole suites.
  • evetest image build: download Go modules once instead of twice, with
    retries. Suites often failed at image build with a proxy.golang.org
    "TLS handshake timeout".
  • Heavy suites on strong runners only: a test or suite can be tagged
    with evetest.WithTags(evetest.TagHeavyTest) in its Init call. The
    tags are shown by make list-tests and make list-test-suites, and the
    nightly and PR workflows run heavy suites (the EVE-k focused ones and
    TestApplicationConnectivitySuite) only on runners labelled
    evetest-runner. All other suites also run on runners labelled
    evetest-runner-light. Nightly runs showed the suites taking 2 to 4
    times longer on the weaker farfalle host, with the cluster suites failing
    there. gh-runner-installer.sh accepts several --gh-label values.

How to test and validate this PR

Run the evetest suites on this PR (/evetest all). The affected tests are
TestKvmToKRepartitionGeometry, TestBackupCorruptRestore, Shrink,
PersistTooFull, ZFSPersist, TestAppInstancePurge, TestMgmtProxy,
TestNativeKubeAppNetworking, TestVGAPassthroughNoHostAccess,
TestVaultKeyModeRecoveryOnExt4/OnZFS, TestLocalNetInstanceACLs,
TestSwitchNetInstanceACLs, TestAccessVLANs, TestSwitchNIWithMultiplePorts,
TestLocalNIWithMultiplePorts, TestNetworkConfigFallback,
TestPNACWithoutProxy/WithProxy, TestThreeNodesPortScan,
TestBootstrapWithMgmtVLAN, TestInfoRetriedAfterServerError,
TestTieBreakerCluster and TestClusterVMApp. The harness changes are also covered by
unit tests in evetest/ (go test .).

Changelog notes

No user-facing changes.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't check them.

🤖 Generated with Claude Code

@milan-zededa milan-zededa added stable Should be backported to stable release(s) testing New test or test framework changes labels Nov 3, 2026
@milan-zededa

This comment was marked as resolved.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@milan-zededa
milan-zededa force-pushed the evetest-harness-test-fixes branch from dec3be1 to ab52ed3 Compare November 3, 2026 21:54
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@lf-edge lf-edge deleted a comment from github-actions Bot Nov 4, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Nov 4, 2026
@milan-zededa
milan-zededa force-pushed the evetest-harness-test-fixes branch 2 times, most recently from 6ed598c to e4abe9b Compare November 4, 2026 09:12
@milan-zededa

This comment was marked as resolved.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterSuite: failure analysis

TestTieBreakerCluster

Failure

Timed out after 1200.001s.
Expected success, but got an error:
    <*errors.errorString | 0x2797e332bee0>: 
    tie-breaker phase completed on every node: query returned nothing
    {
        s: "tie-breaker phase completed on every node: query returned nothing",
    }
Claude's conclusion
Root cause

TestTieBreakerCluster never completes its tie-breaker phase because kube-init's tiebreaker.ConfigApply on edge-dev3 has been stuck retrying the very first mutating step (kubevirtConfig) every ~15s since 11:21:26, failing each time with failed calling webhook "kubevirt-update-validator.kubevirt.io": ... dial tcp 10.43.55.115:443: connect: network is unreachable (confirmed live in /persist/kubelog/*.log on edge-dev3, 60+ consecutive occurrences through 11:31:21). Since the webhook call never succeeds, ConfigApply never reaches StatusSet, so no node ever gets the tie-breaker-config-applied=1 label — which is exactly what the test's kubectl get nodes -l tie-breaker-config-applied=1 query is polling for, explaining the "query returned nothing" timeout at 1200s.

Live inspection shows why the webhook is unreachable: all three nodes' kubevirt-operator-webhook target pods (virt-operator-*) are running on edge-dev1, but edge-dev3's (and even edge-dev1's own) main routing table has no routes to the other nodes' pod CIDRs (10.42.0.0/24/10.42.1.0/24/10.42.2.0/24) and no flannel/VXLAN interface exists anywhere — only each node's local cni0 subnet and the 10.43.0.0/16 service CIDR (routed via eth1) are reachable. This is a cross-node CNI/pod-routing gap in this HA three-node + SeparateClusterPort network-model setup, not a tie-breaker logic bug per se, though it manifests through the tie-breaker's first webhook-dependent patch.

This exact failure signature doesn't appear anywhere in live-ledger.md — it's a new occurrence, not a known/recurring issue. The two prior TestTieBreakerCluster ledger entries were unrelated (a device onboarding stall and a Proxmox VM-power-on JSON error). The tie-breaker feature itself (pkg/kube/kube-init/tiebreaker/tiebreaker.go) is very recently added (ee13ef0c1), consistent with this being the first time this cross-node webhook path has been exercised.

PR #7024's diff is entirely confined to evetest/ test-harness code (cluster-identity reuse across tests, broker power-on/reboot timeout bumps, cluster-report matching) — it touches nothing in pkg/kube/kube-init or any CNI/networking code, so it's very unlikely to be the cause of this failure. This looks like a pre-existing/environmental infrastructure bug (missing cross-node pod-CIDR routes) surfaced by this still-new test, unrelated to the PR under test.

TestClusterVMApp

Failure
Timed out waiting for cluster nodes to become ready in cluster "test-cluster" (device "edge-dev3")
Claude's conclusion

All confined to evetest/ harness code — none touches pkg/kube, zedkube, or nodeagent, so PR #7024 is not the cause.

Root cause

This is the same recurring TestClusterVMApp / edge-dev3 failure that's been seen repeatedly in live-ledger.md (2026-10-29, 2026-10-30 ×2, 2026-10-31, and earlier in the current run). Live evidence matches exactly: edge-dev3's nodeagent status shows RebootReason: BootReasonKubeTransition, Reboot after retry cluster transition attempt 1 (RestartCounter:2, reboot at 2026-11-04T12:04:54Z), and /persist/kubelog/k3s.log has been crash-looping every few minutes since then (still failing at 12:26:12) with Skipping cluster configuration validation: not authorized → failed to bootstrap cluster data: not authorized, so k3s never forms a working API server on that node. kubectl get nodes against edge-dev1 confirms only edge-dev1/edge-dev2 are Ready, with edge-dev3 permanently excluded — exactly matching the test's "Timed out waiting for cluster nodes to become ready ... (device edge-dev3)" symptom.

This does not share a root cause with this run's earlier TestTieBreakerCluster failure (that was a cross-node CNI/pod-routing gap blocking a kubevirt webhook call, unrelated to node-join). It is a known, recurring issue — the identical "stale cluster identity rejected after kube-transition reboot-retry" mechanism has now reproduced on at least 4-5 prior nights since 2026-10-29. PR #7024's diff is entirely confined to evetest/ test-harness files and touches nothing in pkg/kube, zedkube, or nodeagent, so it's very unlikely to be the cause — this looks like a pre-existing, unfixed EVE-K node-rejoin/reboot-retry bug, not something introduced by this PR.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestApplicationConnectivitySuite: failure analysis

TestSwitchNIDeletedWithAppsAttached

Failure
timed out after 5m0.001006885s (excluding download) waiting for app "03732aef-58ae-496c-bbc3-fb44883f171b" (app-1) on device "edge-dev" to reach RUNNING state (last state: RESOLVING_TAG)
Claude's conclusion

I have enough to write the final root-cause summary.

Root cause

Live evidence: app-1 and app-2 are both wedged in RESOLVING_TAG (ContentTreeStatus state 101, ContentSha256 empty) for the image lfedge/evetest-ubuntu-ctr:1.2 served by the SDN's local registry at 192.168.170.2:5000. Watching the device live, the downloader's objectMetadata/resolveTagsToHash manifest-resolution call for both apps only failed just now, at 13:20:36 — a full 10 minutes after it started at ~13:10:46 — with "no response from the transport for lfedge/evetest-ubuntu-ctr:1.2 in 10m0.0s, cancelling", well past the test's 5-minute deadline that had already fired at 13:15:46. The registry itself is healthy (curl http://192.168.170.2:5000/v2/ → 200 right now), so this is a hung/stalled tag-resolution request against the registry rather than a registry outage.

This is the same signature and mechanism already recorded in live-ledger.md as a known, recurring issue dating back to 2026-10-31 (TestFlowLog, TestSwitchNetInstanceACLsWithFlowLog), which recurred again on 2026-11-03 (TestFlowLog) and 2026-11-03 (TestLocalNetInstanceACLs's acl-app-1) — always a runResolveHandler/manifest-resolve goroutine in the downloader hanging well past the 5-minute test timeout against this same SDN local registry. Tonight's instance is slightly broader than prior occurrences (both sibling apps hung identically rather than one succeeding quickly), but it's the same class of bug, not a new mechanism.

PR #7024's diff only touches evetest/* test-harness files (cluster-reuse/reboot-expectation logic for device/cluster config), nothing in pkg/pillar downloader/volumemgr/pubsub code — so this failure is unrelated to the PR under test; it's a pre-existing, recurring EVE/pillar-side infrastructure bug in the resolve-tag path against the SDN registry.

@milan-zededa
milan-zededa force-pushed the evetest-harness-test-fixes branch from 54eeeda to 9af53fa Compare November 4, 2026 13:59
@milan-zededa
milan-zededa force-pushed the evetest-harness-test-fixes branch from 9af53fa to 1c7c33c Compare November 4, 2026 14:17
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

TestNodeClusterResilienceSuite: failure analysis

TestDNIDandBackupDNID

Failure
timed out after 6m0.000762014s (excluding download) waiting for app "7db4d72c-99f6-44cc-b491-8596378fa3a1" (vm2) on device "edge-dev1" to reach RUNNING state (last state: HALTING)
Claude's conclusion
Root cause

Live evidence on edge-dev1: app vm2's AppInstanceStatus is stuck in HALTING with appErr: "failed calling webhook "virtualmachinereplicaset-validator.kubevirt.io": ... dial tcp 10.43.51.163:443: connect: connection refused" (virt-api unreachable). This happened right as edge-dev3 — deliberately taken down by this test's backup-DNID failover scenario, and currently confirmed fully unreachable (SSH times out / "no route to host") — went NodeNotReady; zedkube's own logs show it then force-deleted all KubeVirt/Longhorn control-plane pods that had been running on edge-dev3 (DeleteControlPlanePodsOnNode: ... IS declared unreachable long enough to force delete all kubevirt pods, deleting virt-api/virt-controller/virt-operator). The replacement control-plane pods rescheduled onto edge-dev1 then sat in ContainerCreating for well over a minute, repeatedly failing CNI sandbox setup with "no routing classification marker yet for pod" — EVE's own eve-bridge CNI plugin (pkg/kube/eve-bridge/eve-bridge.go:238) deliberately rejects pod network setup until zedkube has classified the pod, and zedkube hadn't caught up amid the simultaneous mass pod churn. With virt-api unavailable device-wide for that stretch, vm2's VMIRS webhook call failed and it never got past HALTING before the test's 6-minute deadline (the pods did recover ~2 minutes later, confirming this was transient, just outside the test's window).

This doesn't match either prior TestDNIDandBackupDNID ledger entry — 2026-11-01's was a broker→Proxmox TLS-handshake-timeout infra flake during PowerOff, and 2026-10-31's was a distinct zedmanager status-reporting gap (handleENClusterAppStatusImpl not deriving RUNNING for a VMI scheduled off the DNID node) — so this specific virt-api-outage/CNI-classification-race signature is a new failure mode for this test, not a previously recorded recurring issue, though it's the same broad "DNID/backup-DNID failover resilience" theme as those and the separately-tracked TestVMAppPurgeDuringFailover bug. PR #7024's diff touches only evetest/ harness code (Dockerfile build caching, cluster-ID/token reuse for reused devices, test files) with nothing in pkg/pillar, zedkube, kubeapi, or the eve-bridge CNI plugin, so this failure looks pre-existing/environmental — a device-side timing race during node failover — rather than caused by this PR.

TestVMAppPurgeDuringFailover

Failure

Timed out after 900.000s.
The function passed to Eventually failed at /evetest/tests/cluster/purge_during_failover_test.go:169 with:
expected exactly one VMIRS for the app, found []
Expected
    <[]string | len:0, cap:0>: nil
to have length 1
Claude's conclusion

The diff touches only evetest/ harness code, nothing in pkg/pillar/zedmanager/zedkube — confirms unrelated to the PR. I have enough to write the conclusion.

Root cause

This is the same already-known, recurring zedmanager/zedkube "purge-during-failover" deadlock in getKubeAppActivateStatus (pkg/pillar/cmd/zedmanager/zedmanager.go:2059), now reproducing for a fourth consecutive time tonight (the ledger already recorded this test failing with the identical signature twice earlier in this same run, run #7, and it's been a known/recurring issue since 2026-10-29, with repeats on 10-30 and 10-31). Live evidence on edge-dev2 confirms the exact same fingerprint: app failover-purge-app (uuid 86a178a0-...) has PurgeInprogress:4, Activated:false, IsDesignatedNodeID:false, and zedmanager is logging a continuous getKubeAppActivateStatus(86a178a0-...): not designated, node 0d5f4222-..., onTheDevice false, statusRunning false loop; kubectl -n eve-kube-app get vmirs,vmi,vm returns "No resources found" cluster-wide, matching the test's found []. edge-dev1 (the designated node) is deliberately EVE_DEVICE_STATE_SUSPECT/down for this failover scenario — once the purge tore down the old VMIRS, nothing else in the cluster is eligible to originate a replacement, so the app is permanently stuck. PR #7024's diff touches only evetest/ harness files (no changes to pkg/pillar, zedmanager, or zedkube), so this is unrelated to the PR and is purely the standing, unfixed device-side bug.

@milan-zededa
milan-zededa force-pushed the evetest-harness-test-fixes branch from 867e697 to 7e95422 Compare November 4, 2026 17:46
@milan-zededa
milan-zededa marked this pull request as ready for review November 4, 2026 17:48
milan-zededa and others added 16 commits November 4, 2026 18:59
The konvert tests that start the device on a release (10.1.0 by
default) left INITIAL_EVE_REPO empty, so the harness pulled the release
from the run's own repo. In CI that repo holds only the build under
test (evebuild/nightly in the nightly runs, evebuild/pr in PR runs),
so all five of these tests failed at setup in every run with
`EVE image "evebuild/nightly:10.1.0-kvm-amd64" is not available`.

Default the initial repo to lfedge/eve for these tests, as the upgrade
tests already do. The flavor-switch tests keep starting from the build
under test, so their default stays the repo under test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Device reuse keeps the cluster block of the device's config: the reset
policies clear apps and network instances, but leaving cluster mode
takes a reboot, which a reset cannot hide. So when a test that does not
use cluster mode reuses a device a cluster test left in cluster mode,
its first config drops the cluster block and EVE correctly reboots to
convert back to a single node (BootReasonKubeTransition, "Transition
from cluster mode to single node"). Nothing expected that reboot, and
the test failed its reboot audit: TestAppInstancePurge after
TestNativeKubeAppNetworking, in two nightly runs.

Mark a reused device whose config still holds a cluster block, and
expect the reboot in ApplyConfig when the test's first config drops it.
Refusing to reuse such devices would also stop consecutive cluster
tests from reusing their nodes. A cluster block that the test applied
itself stays the test's to account for, as TestClusterToSingleConversion
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
TestMgmtProxy asks for direct registry pulls (RequireDirectRegistryPulls)
because its pod must pull through EVE's management proxy. That
requirement only skips writing the containerd mirror config while a
device is set up, though, and the devices reused from
TestThreeNodesCluster already pull through the evetest registry mirror.
The pull then bypassed the proxy and failed with "no route to host",
in five nightly runs.

Record on each device which mirror setup it got (none, dual-stack or
IPv6-only), and don't reuse a device whose mirrors differ from what the
next test wants. A device without mirrors is still reused: lacking
mirrors only makes pulls slower.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
TestNativeKubeAppNetworking failed in three nightly runs: kubectl apply
of its pod failed with "failed to download openapi: the server is
currently unable to handle the request" right after the test had
waited for its single-node cluster to become ready. Two causes:

- WaitForClusterNodeIsReady (and EdgeCluster.WaitUntilNodesAreReady)
  accept the newest report already received. On a device reused from
  the previous test, that report comes from before the test's cluster
  config, so the wait returned at once, and k3s then restarted for the
  single-to-cluster transition under the test's kubectl call.
- Even a fresh report only says that the node is Ready. Seconds later,
  the API server on a single 4-vCPU node can still be too busy
  starting kubevirt, CDI and Longhorn to serve an apply.

Accept only reports for the cluster the device is configured for: EVE
reports the configured cluster's UUID, or the nil UUID without a
cluster config, so a report naming anything else predates the config.
Reports without a cluster ID, from EVE versions that send none, are
still accepted. In the cluster tests, retry kubectl apply on transient
API server errors for up to two minutes; any other error still fails
at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
TestVGAPassthroughNoHostAccess failed in every nightly run with
"Host processes interfered with PCI devices ...: [app-restart]
pci_user_write_config_word comm=CPU 0/KVM pid=<qemu>". The tracer's
comm is the name of the calling thread, and QEMU names its vCPU threads
"CPU <n>/KVM". The guest's own config-space writes to the
passed-through device reach the kernel from those threads, but
isViolation allowed only threads named "qemu-system-*", so every write
of the guest was reported as host interference.

Allow QEMU's vCPU threads too. Only a qemu holding the device's VFIO
file can write its config space this way, so no access by another
process is let through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
TestVaultKeyModeRecoveryOnExt4/OnZFS failed in five nightly runs with
assertions like "the correction must survive the reboot: Expected
<int>: 1 to equal <int>: 2": a log line count went down between the
baseline and the final check. newlogd writes every device log record
twice, to a keep stream and an upload stream, and newlogCount counted
both. loguploader deletes the upload copy once it is sent, so any
upload between the two counts lowered the count.

Count only the keep stream: its plaintext chunks in collect/ and its
gzipped chunks in keepSentQueue/, dropping the duplicates a chunk has
while it is being compressed. Read the plaintext chunks with cat:
pillar's zcat is BusyBox's, whose -f does not pass plaintext through,
so the records still in collect/ were never counted at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
TestLocalNetInstanceACLs failed waiting 5 minutes for a network
instance info message reporting both app VIFs with IP addresses,
although the network instance had reported exactly that. The test first
waited for the NI to come ONLINE, then for the VIF IPs, on the same
watch. EVE publishes NI info only when the NI's status changes, and
the one message showing the NI ONLINE already held both IPs: the first
wait consumed it, and no further message came for the second.

Add expectNIInfo, which first checks the message an earlier wait
already received, and use it for the second wait of each such pair: in
the two ACL tests, and in the VLAN and STP tests. The routing failover
test was worse: it opened a new watch for each of the two routes it
waits for, so if both routes moved in one message, the second watch
never saw it. Wait for both routes with one watch and one condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
TestNetworkConfigFallback's Phase 3 waits for the fixed higher-priority
DPC to be retested and become active again, within a budget that
assumes the retest comes from the 1-minute testbetterinterval timer.
But when a DPC verification finds nothing testable yet, because every
entry is still within its failure cooldown, nim re-arms only the
regular testinterval timer, which the test left at its 5-minute
default. That retest came 338s after the failure, past the 230s budget.

Lower timer.port.testinterval to its 1-minute minimum too, so the
retest comes within one tick either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Every evetest CI job builds the evetest image when its sources are not
published yet, and each build downloaded all Go modules twice: in the
build stage and again in the final stage. With all suite jobs starting
together, proxy.golang.org often timed out ("TLS handshake timeout")
and the suite failed before running any test.

Copy the module cache from the build stage into the final image instead
of downloading it again, and retry the remaining download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
When a stress cycle disables PNAC, nim can test the new DPC before the
port has an IP from the pre-auth VLAN. The test fails, nim falls back
to the previous DPC and retries the new one only after
timer.port.testbetterinterval, 10 minutes by default, while the test
waits 5 minutes for the port to reach the pre-auth VLAN.

Lower the interval to its 60s minimum, as TestNetworkConfigFallback does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Adam answers 400 for a known device until the device posts its
certificates, because its file backend reports the missing certs file
as a generic error. The harness counted every 400 as a failure and gave
up on the ECDH certificate after 11 polls (about 55s), well before
deviceAttestTimeout. On a slow host the device posted its certificates
73s after onboarding, and test setup failed.

Treat 400 as "not published yet", and raise deviceAttestTimeout from 2
to 5 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The harness gave the broker 1 minute to power an EVE VM on or off or to
reboot it. On Proxmox each is a task that can wait about a minute on a
busy host: a VM start waited 61s for the storage lock held by a parallel
VM create, and the harness cancelled it 1s before the VM came up. The
broker also retries a failed task up to 3 times, which never fit in 1
minute.

Raise the three timeouts to 5 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Every cluster test generated a new cluster ID and join token, even when
it reused the devices of a cluster the previous test had formed. That
cluster keeps running, but the devices kept reporting its old ID: EVE
re-reports cluster info only when the cluster's state changes, not on
an ID change alone. The cluster readiness wait never accepted those
reports, and TestThreeNodesPortScan timed out after
TestThreeNodesCluster. The token mattered too: after
TestClusterToSingleConversion took edge-dev3 out of the cluster,
TestClusterVMApp configured a new token, which the running k3s of
edge-dev1 and edge-dev2 kept ignoring, so it rejected edge-dev3 joining
with it as "not authorized".

When the nodes of a new cluster config are reused devices that still
run one cluster, and no other device is in it, keep that cluster's ID
and token, as a real controller would. Nodes without a cluster config
join it with the same token. The token is kept in plaintext on the
harness's device config for this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
When TestTieBreakerCluster reused the devices of the previous cluster
test, it got their formed cluster and only made one node the
tie-breaker. That conversion should work but is unlikely in practice,
and it currently breaks cross-node pod networking: kube-init never
reached the end of its tie-breaker setup, failing on an unreachable
kubevirt webhook, in both runs that reused a cluster. On fresh devices
the test passed.

The test is about forming a cluster with a tie-breaker, so create its
devices from scratch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Suites failed before running any test when a download in the suite's
setup failed once: "go install gotestfmt" timed out on
proxy.golang.org, and downloading the EVE image artifacts failed
despite the short retries of download-artifact itself.

Retry go install and the evetest CLI build (which downloads Go modules)
up to 3 times through a small retry.sh, and download the EVE image
artifacts once more after a minute if the first download fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Some evetest hosts have much slower storage and CPU, and the long
cluster suites take 2-3x longer there and tend to time out.

Add evetest.WithTags, an Init option that tags a test or a suite, and
the TagHeavyTest tag, which declares that it needs a powerful host. The
option has no effect at run time: list-tests reads it from the source,
shows the tags, and "make list-test-suites" prints those of each suite
after its name. The nightly and PR workflows run heavy suites only on
runners labelled evetest-runner, while all other suites may also use
runners labelled only evetest-runner-light.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa force-pushed the evetest-harness-test-fixes branch from 7e95422 to edf3067 Compare November 4, 2026 17:59

@rucoder rucoder left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

evetest: skip decisions for this PR

This PR only touches the evetest test harness/CI itself (device reuse,
cluster readiness/identity, broker timeouts, device-cert waiting, a few
test files, and CI download/runner-tagging plumbing) — no pkg/pillar or
other EVE source changed. The harness changes are generic (onboarding,
power ops, device reuse) and are incidentally exercised by every suite, so
I kept the tests that most directly validate each fix and pruned suites
and subtests that are either unrelated to any of the changed areas, or
that would only exercise the same changed path redundantly.

Skipped whole suites (unrelated to anything in this PR):
TestDatastoreSuite, TestDiagSuite, TestHardwareSuite,
TestPcibackErrorSuite, TestLPSSuite, TestStorageSuite,
TestTelemetrySuite, TestPortScanSuite.

TestApplicationConnectivitySuite — kept the tests that exercise the
"don't wait for network-instance info an earlier wait already consumed"
fix (TestLocalNetInstanceACLs, TestSwitchNetInstanceACLs,
TestAccessVLANs, TestSwitchNIWithMultiplePorts,
TestLocalNIWithMultiplePorts). Skipped the suite's other subtests
(unrelated features: TestLocalNI, TestSwitchNI, TestNIReplace,
TestMoveAppBetweenNIs, TestAirGapSwitchNI, TestLimitedIPSpace,
TestFlowLog, TestApplicationIPv6Connectivity,
TestApplicationNTPConfig, TestPropagatedRoutes,
TestApplicationGateway, TestPortForwarding,
TestNetworkAdapterPassthrough, TestNetworkAdapterPassthroughChange,
TestStagedNICChange, TestNICCountChange,
TestNICCountChangeOrderedInterface, TestSwitchNIPortConfigRace,
TestLocalNIUplinkFlap, TestSwitchNIDeletedWithAppsAttached), plus the
WithFlowLog ACL variants (TestLocalNetInstanceACLsWithFlowLog,
TestSwitchNetInstanceACLsWithFlowLog), which exercise the exact same
fixed wait as their non-flowlog counterparts already kept.

TestAppsSuite — kept TestVGAPassthroughNoHostAccess (the fix to
stop flagging QEMU's vCPU threads). Skipped the suite's other,
feature-unrelated subtests (TestPurgeNeverActivatedApp, TestVNC,
TestAppInstanceMetadata, TestAppUserData, TestAppLogs,
TestCgroupTree, TestMemoryPressureEvents, TestAppRestart,
TestAppTeardownWithRemovedAdapter, TestDeviceShutdownAndRecovery,
TestHaltAfterImmediateDeactivate, TestHaltUnresponsiveGuest,
TestWorkerPoolSaturation, TestVMAppPurgeReplacesVMIRS,
TestVMAppPurgeAfterPowerCycle, TestLotsOfApps,
TestLayerDownloadErrorsBounded, TestContainerdCPUs).

TestDeviceConnectivitySuite — kept TestPNACWithoutProxy/
TestPNACWithProxy and TestNetworkConfigFallback (the
testbetterinterval/testinterval fixes). Skipped the suite's other,
unrelated connectivity subtests (TestDHCPIPv4Only, TestStaticIPv4Only,
TestDNSFunctionality, TestPortFailover, TestIntermittentConnectivity,
TestMgmtTrafficRoutedViaApp, TestDeviceIPv6Connectivity,
TestDeviceNTPConfig, TestActiveBackupBond, TestLACPBond,
TestVLANSubinterfaces, TestVLANSubinterfacesOnTopOfLAGs,
TestCellularConnectivity, TestWifiConnectivity).

TestSecuritySuite — kept TestVaultKeyModeRecoveryOnExt4/OnZFS
(the newlogd keep-stream counting fix). Skipped the suite's other,
unrelated subtests (TestAppArmorEnabled, TestVCom, TestAppVTPM and
all its variants, TestControllerSigningCertChange,
TestControllerEncryptCertChange).

TestKonvertSuite — kept TestKvmToKRepartitionGeometry,
TestBackupCorruptRestore, Shrink, PersistTooFull and ZFSPersist,
which cover the INITIAL_EVE_REPO default fix. The suite's remaining
repartition-family tests (TestPersistWipeRestore, Grow,
GrowLocalNI, TestKvmToKRepartitionTopology,
TestKvmToKVolumeMigration, TestFirstBootEVEKAppVolume,
TestKvmToKZFSVaultMigration, TestKvmToKZFSVaultPowerCutMidSwap) rely
on the exact same default and would only duplicate that coverage, so
they're skipped. The flavor-switch tests (TestKvmToKUpgrade,
TestKvmToKContentTree, TestKvmToKAppRecreate) already defaulted
correctly and are untouched by this PR, so they're skipped too.

TestBootstrapSuite — kept TestBootstrapWithMgmtVLAN, which the PR
description calls out directly. The device-cert/attestation-timeout fix
it validates is generic to onboarding and applies identically regardless
of bootstrap flavor, so the suite's other bootstrap-variant groups
(TestBootstrapWithLastResort, TestBootstrapWithStaticIP,
TestBootstrapWithProxy, TestBootstrapWithLACPBond,
TestBootstrapWithNetworkBoot) would only onboard a device the same way
again and are skipped.

Everything else (TestControllerFaultsSuite including
TestInfoRetriedAfterServerError, TestNodeClusterSuite including
TestAppInstancePurge/TestMgmtProxy/TestNativeKubeAppNetworking/
TestThreeNodesPortScan/TestTieBreakerCluster/TestClusterVMApp,
TestNodeClusterResilienceSuite, TestKubeRegistrationSuite, and
TestUpgradeSuite) is kept as-is.

An engineer can comment /evetest all to force the full suite to run
instead, or /evetest <name> <name> ... to run only specific
tests/suites.

@eriknordmark
eriknordmark merged commit 27a85cc into lf-edge:master Nov 4, 2026
67 of 76 checks passed
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestControllerFaultsSuite: failure analysis

TestControllerFaultsSuite

Failure
failed to connect to the broker at 10.208.13.244:50221: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.208.13.244:50221: connect: no route to host"
Claude's conclusion

This confirms nothing in the PR touches the broker-dial/network path itself — only build caching and timeout-constant changes. I have enough evidence now.

Root cause

This suite failed at the very first step of evetest.Init (harness.go:923), 3 seconds after the harness started up and before Adam, any EVE device, or SDN were even created — evetest status confirms "No EVE devices found" and adam-db/ has no device/onboard entries. The error is a pure network-layer failure (connect: no route to host) dialing the Proxmox broker VM at 10.208.13.244:50221, i.e. the broker host itself was unreachable from this runner at that moment, not an application-level timeout.

The ledger shows this same broker endpoint (10.208.13.244:50221) failing repeatedly in recent runs, but always as i/o timeout or connect: network is unreachable cascading from a specific 2026-11-02 incident (a wedged qemu guest on the broker host hanging in kvm_async_pf_task_wait_schedule), and separately as a TestRadioSilence/TestProfile broker-timeout on 2026-11-03. Tonight's no route to host variant at this exact port doesn't match any prior ledger entry, so this is a new failure signature for an already-flaky broker endpoint — recurring symptom class, not an identical repeat.

The PR (#7024) only touches evetest Docker build caching/retries, CI workflow retry logic, and broker RPC timeout constants (power on/off/reboot raised from 1→5 min) — none of which affects the initial gRPC dial to the broker, and the failure happened before any of that code could even execute. This looks environmental (broker host/network transiently unreachable), unrelated to the PR.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestDeviceConnectivitySuite: failure analysis

TestDeviceConnectivitySuite

Failure
failed to connect to the broker at 10.208.13.244:50221: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.208.13.244:50221: connect: no route to host"
Claude's conclusion
Root cause

Live evidence shows this failed at the very first step of the suite — evetest.Init()'s initial brokerClient.Connect() call, 3 seconds after the harness started, before any EVE device or SDN was ever created (evetest status now reports "No EVE devices found", evetest sdn status reports "SDN client is not initialized"). /proc/net/arp on this runner shows the default gateway (10.208.13.254) has a resolved, complete ARP entry, but the broker IP 10.208.13.244 has a stale/incomplete ARP entry — i.e. this specific runner host currently cannot reach that broker at the network layer, even though its own NIC (eth0) is up with carrier.

PR #7024 is directly implicated: it adds a new evetest-runner-light GitHub Actions runner label and changes evetest-pr.yml's runs-on so that any suite not tagged evetest.TagHeavyTest (confirmed from the diff that TestDeviceConnectivitySuite was not tagged heavy, unlike TestApplicationConnectivitySuite which was) now runs on evetest-runner-light instead of unconditionally on evetest-runner. Per the PR's own README update, "light" runners are meant for separate, weaker hosts, each with their own EVETEST_BROKER_ADDRESS. This is consistent with this suite landing, for the first time, on a different physical/VM host than usual — one whose network isn't configured to reach 10.208.13.244.

The ledger does show 10.208.13.244:50221 unreachability as a recurring symptom (2026-11-01 through 2026-11-03), but in every prior case it was a downstream teardown failure after some other test had already broken the device/broker connection mid-run (qemu hangs, Proxmox TLS stalls, etc.) — never a failure at suite-start Init() before any device existed, as seen here. So this is a new failure mode, not a repeat of the known cascading-outage pattern, and it plausibly stems directly from this PR's runner-routing change sending the suite to a light runner that lacks broker connectivity.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestUpgradeSuite: failure analysis

TestUpgradeSuite

Failure
failed to connect to the broker at 10.208.13.244:50221: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.208.13.244:50221: connect: no route to host"
Claude's conclusion
Root cause assessment

The failure happened instantly (3 seconds into Init(), before any device/SDN was created — evetest status confirms "No EVE devices found") trying to make the very first gRPC call to the broker at 10.208.13.244:50221, with connect: no route to host — a routing-layer failure, not a slow/crashed service. This is distinct from every prior ledger occurrence of this exact broker IP:port: those always involved the endpoint going unreachable mid-run, after a device had already been created and something else (a wedged qemu guest, a dead SDN tunnel) made the broker/agent stop responding (e.g. run #6 2026-11-03, run #4 2026-11-01's TestProfile/TestRadioSilence). Here there's no prior test in this run and no device was ever provisioned, so it can't be the usual cascading-device-hang pattern — this looks like the test's own host/runner never had a network path to the broker at all.

That points directly at this PR (#7024): it adds a new evetest-runner-light self-hosted runner label (.github/actionlint.yaml, evetest-nightly.yml, evetest-pr.yml) and changes suite scheduling so that any suite not tagged evetest.WithTags(evetest.TagHeavyTest) now runs on evetest-runner-light instead of the previously-used evetest-runner. TestUpgradeSuite's evetest.Init(test) call carries no tags, so under this PR it would be routed onto the new, presumably not-yet-network-configured evetest-runner-light pool — plausibly explaining a brand-new "no route to host" to the broker from a host that simply lacks the route/firewall rule the established evetest-runner machines have. The rest of the PR's evetest code changes (harness.go timeouts, setup.go registry-mirror logic) don't touch the broker-connect path (harness.go:901-923) at all, so I don't see another PR-driven mechanism. I don't have direct visibility into which runner label actually executed this job to fully confirm it, but given the timing and nature of the failure, the new runner-routing change is the most plausible culprit — this is not a previously-known/recurring issue per the ledger.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestSecuritySuite: failure analysis

TestSecuritySuite

Failure
failed to connect to the broker at 10.208.13.244:50221: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.208.13.244:50221: connect: no route to host"
Claude's conclusion

I have enough evidence already from evetest status/sdn status/eve info confirming the broker is unreachable and nothing was ever provisioned, matching the pattern already logged twice tonight for other suites. I'll skip the extra ping and finalize the analysis.

Root cause

TestSecuritySuite failed 4 seconds into harness Init() (harness.go:923), before any EVE device or SDN VM was ever created — evetest status shows "No EVE devices found" and evetest sdn status/evetest eve info both report nothing initialized. The error (dial tcp 10.208.13.244:50221: connect: no route to host) is an immediate network-layer rejection on the very first gRPC dial to the Proxmox/VM-provisioning broker, not a stall or timeout.

This is not a new or isolated event: it's the third occurrence tonight in this same run (#8), byte-identical to the already-logged TestApplicationConnectivitySuite and TestDiagSuite failures in live-ledger.md — same broker endpoint 10.208.13.244:50221, same "no route to host" signature, same pre-provisioning timing. The ledger traces this broker-unreachable pattern back further to the broader "SDN/broker tunnel becomes unreachable" infrastructure instability first flagged on 2026-10-29 and recurring repeatedly since, so this is a known, recurring environmental issue rather than a new one.

PR #7024's diff touches evetest/harness.go, but only to lengthen broker operation timeouts (power on/off, reboot, device-attest) for slow Proxmox tasks — it doesn't touch the initial Init() connection/dial path where this failure occurs, and "no route to host" is an immediate connection refusal, not a timeout being exceeded. This failure is unrelated to the PR and is Proxmox/broker infrastructure outage, consistent with tonight's run-wide pattern.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestBootstrapSuite: failure analysis

TestBootstrapSuite

Failure
failed to connect to the broker at 10.208.13.244:50221: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.208.13.244:50221: connect: no route to host"
Claude's conclusion

That's the exact line that failed, and the PR diff doesn't touch this code path at all, confirming the failure is unrelated to the PR.

Root cause

TestBootstrapSuite failed in Init() itself (harness.go:923), ~4 seconds after harness startup, trying to dial the infrastructure broker at 10.208.13.244:50221 — before any EVE device or SDN was ever provisioned (evetest status/sdn status/eve info all confirm "no EVE devices" / "SDN client is not initialized"). The error (no route to host) is an immediate network-layer rejection, not a slow service.

This shares the exact same root cause as two other failures already recorded earlier in tonight's run (TestApplicationConnectivitySuite and TestDiagSuite), both of which hit the identical broker endpoint 10.208.13.244:50221 with the identical no route to host error at Init(), before any device/SDN existed — this is one single broker-unreachable outage affecting multiple suites tonight, not three separate incidents. Per the ledger, broker/SDN unreachability under this general "no route to host" signature is a known, recurring class of infrastructure instability dating back to 2026-10-29 (recurring nightly since), though tonight's specific pattern — the broker's own control port being unreachable at harness startup, rather than the SDN VM's gRPC tunnel failing later — most closely matches the 2026-11-03 (run #6) cascade where the same broker IP went from stalled to unreachable, now escalated to immediate no route to host.

I checked PR #7024's diff: it only touches CI-script retry logic (artifact download retries, retry.sh wrapper) and broker RPC timeout constants for power-on/off/reboot operations plus harness device-state bookkeeping — it never touches the broker-dial/connection code at harness.go:923, and the failure happens before any device/power operation runs. This failure is environmental/infrastructure (broker host unreachable), not caused by this PR.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

TestAppsSuite: failure analysis

TestAppsSuite

Failure
failed to connect to the broker at 10.208.13.244:50221: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.208.13.244:50221: connect: no route to host"
Claude's conclusion
Root cause

TestAppsSuite died in evetest.Init() (harness.go:923) on the very first gRPC dial to the Proxmox broker at 10.208.13.244:50221, before any EVE device or SDN was provisioned — live evetest status confirms "No EVE devices found" and evetest sdn status reports "SDN client is not initialized", so there's no device/guest state to inspect; the broker host itself is simply unreachable (no route to host) from this runner right now.

This is not an isolated incident tonight: the same exact endpoint and error already hit TestApplicationConnectivitySuite, TestDiagSuite, and TestKubeRegistrationSuite earlier in this same run (#8, per live-ledger.md), all failing identically at Init before provisioning anything — so TestAppsSuite shares the same root cause as those three. Notably TestKubeRegistrationSuite is a heavy-tagged suite (confirmed WithTags(evetest.TagHeavyTest) in evetest/tests/kuberegistration/testsuite_test.go), so this isn't confined to one runner pool — it looks like a genuine, broad Proxmox-broker outage affecting the whole fleet tonight, not a single-runner-label issue.

Per the ledger, this broker endpoint (10.208.13.244:50221) is a known recurring trouble spot, first flagged 2026-11-03 (run #6, where it degraded mid-run from i/o timeout to network is unreachable after a wedged qemu guest), and the "SDN/broker gRPC no route to host" pattern has recurred across multiple nightly runs since (e.g. 2026-11-01, 2026-11-04). Tonight's run #8 shows an escalation of that same instability: multiple suites now fail immediately on the first connection attempt rather than losing an already-established connection mid-run.

PR #7024's diff only touches evetest/CI code (timeout constants in harness.go, the new heavy tag mechanism, runner-pool routing in the nightly/PR workflows, the Proxmox gh-runner-installer.sh multi-label support) — none of it touches the broker dial/connect logic in harness.go, and the fact that a heavy-tagged suite (still on the original evetest-runner pool) hit the identical failure rules out the new evetest-runner-light pool as the cause. This looks like pre-existing Proxmox-broker infrastructure flakiness, unrelated to this PR's changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s) testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants