Repository navigation
evetest: fix recurring test and harness failures - #7024
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dec3be1 to
ab52ed3
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
6ed598c to
e4abe9b
Compare
This comment was marked as resolved.
This comment was marked as resolved.
TestNodeClusterSuite: failure analysisTestTieBreakerClusterFailureClaude's conclusionRoot cause
Live inspection shows why the webhook is unreachable: all three nodes' This exact failure signature doesn't appear anywhere in PR #7024's diff is entirely confined to TestClusterVMAppFailureClaude's conclusionAll confined to Root causeThis is the same recurring This does not share a root cause with this run's earlier |
TestApplicationConnectivitySuite: failure analysisTestSwitchNIDeletedWithAppsAttachedFailureClaude's conclusionI have enough to write the final root-cause summary. Root causeLive evidence: This is the same signature and mechanism already recorded in PR #7024's diff only touches |
54eeeda to
9af53fa
Compare
9af53fa to
1c7c33c
Compare
TestNodeClusterResilienceSuite: failure analysisTestDNIDandBackupDNIDFailureClaude's conclusionRoot causeLive evidence on This doesn't match either prior TestVMAppPurgeDuringFailoverFailureClaude's conclusionThe diff touches only Root causeThis is the same already-known, recurring |
1c7c33c to
8fad23a
Compare
867e697 to
7e95422
Compare
The konvert tests that start the device on a release (10.1.0 by default) left INITIAL_EVE_REPO empty, so the harness pulled the release from the run's own repo. In CI that repo holds only the build under test (evebuild/nightly in the nightly runs, evebuild/pr in PR runs), so all five of these tests failed at setup in every run with `EVE image "evebuild/nightly:10.1.0-kvm-amd64" is not available`. Default the initial repo to lfedge/eve for these tests, as the upgrade tests already do. The flavor-switch tests keep starting from the build under test, so their default stays the repo under test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Device reuse keeps the cluster block of the device's config: the reset policies clear apps and network instances, but leaving cluster mode takes a reboot, which a reset cannot hide. So when a test that does not use cluster mode reuses a device a cluster test left in cluster mode, its first config drops the cluster block and EVE correctly reboots to convert back to a single node (BootReasonKubeTransition, "Transition from cluster mode to single node"). Nothing expected that reboot, and the test failed its reboot audit: TestAppInstancePurge after TestNativeKubeAppNetworking, in two nightly runs. Mark a reused device whose config still holds a cluster block, and expect the reboot in ApplyConfig when the test's first config drops it. Refusing to reuse such devices would also stop consecutive cluster tests from reusing their nodes. A cluster block that the test applied itself stays the test's to account for, as TestClusterToSingleConversion already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
TestMgmtProxy asks for direct registry pulls (RequireDirectRegistryPulls) because its pod must pull through EVE's management proxy. That requirement only skips writing the containerd mirror config while a device is set up, though, and the devices reused from TestThreeNodesCluster already pull through the evetest registry mirror. The pull then bypassed the proxy and failed with "no route to host", in five nightly runs. Record on each device which mirror setup it got (none, dual-stack or IPv6-only), and don't reuse a device whose mirrors differ from what the next test wants. A device without mirrors is still reused: lacking mirrors only makes pulls slower. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
TestNativeKubeAppNetworking failed in three nightly runs: kubectl apply of its pod failed with "failed to download openapi: the server is currently unable to handle the request" right after the test had waited for its single-node cluster to become ready. Two causes: - WaitForClusterNodeIsReady (and EdgeCluster.WaitUntilNodesAreReady) accept the newest report already received. On a device reused from the previous test, that report comes from before the test's cluster config, so the wait returned at once, and k3s then restarted for the single-to-cluster transition under the test's kubectl call. - Even a fresh report only says that the node is Ready. Seconds later, the API server on a single 4-vCPU node can still be too busy starting kubevirt, CDI and Longhorn to serve an apply. Accept only reports for the cluster the device is configured for: EVE reports the configured cluster's UUID, or the nil UUID without a cluster config, so a report naming anything else predates the config. Reports without a cluster ID, from EVE versions that send none, are still accepted. In the cluster tests, retry kubectl apply on transient API server errors for up to two minutes; any other error still fails at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
TestVGAPassthroughNoHostAccess failed in every nightly run with "Host processes interfered with PCI devices ...: [app-restart] pci_user_write_config_word comm=CPU 0/KVM pid=<qemu>". The tracer's comm is the name of the calling thread, and QEMU names its vCPU threads "CPU <n>/KVM". The guest's own config-space writes to the passed-through device reach the kernel from those threads, but isViolation allowed only threads named "qemu-system-*", so every write of the guest was reported as host interference. Allow QEMU's vCPU threads too. Only a qemu holding the device's VFIO file can write its config space this way, so no access by another process is let through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
TestVaultKeyModeRecoveryOnExt4/OnZFS failed in five nightly runs with assertions like "the correction must survive the reboot: Expected <int>: 1 to equal <int>: 2": a log line count went down between the baseline and the final check. newlogd writes every device log record twice, to a keep stream and an upload stream, and newlogCount counted both. loguploader deletes the upload copy once it is sent, so any upload between the two counts lowered the count. Count only the keep stream: its plaintext chunks in collect/ and its gzipped chunks in keepSentQueue/, dropping the duplicates a chunk has while it is being compressed. Read the plaintext chunks with cat: pillar's zcat is BusyBox's, whose -f does not pass plaintext through, so the records still in collect/ were never counted at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
TestLocalNetInstanceACLs failed waiting 5 minutes for a network instance info message reporting both app VIFs with IP addresses, although the network instance had reported exactly that. The test first waited for the NI to come ONLINE, then for the VIF IPs, on the same watch. EVE publishes NI info only when the NI's status changes, and the one message showing the NI ONLINE already held both IPs: the first wait consumed it, and no further message came for the second. Add expectNIInfo, which first checks the message an earlier wait already received, and use it for the second wait of each such pair: in the two ACL tests, and in the VLAN and STP tests. The routing failover test was worse: it opened a new watch for each of the two routes it waits for, so if both routes moved in one message, the second watch never saw it. Wait for both routes with one watch and one condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
TestNetworkConfigFallback's Phase 3 waits for the fixed higher-priority DPC to be retested and become active again, within a budget that assumes the retest comes from the 1-minute testbetterinterval timer. But when a DPC verification finds nothing testable yet, because every entry is still within its failure cooldown, nim re-arms only the regular testinterval timer, which the test left at its 5-minute default. That retest came 338s after the failure, past the 230s budget. Lower timer.port.testinterval to its 1-minute minimum too, so the retest comes within one tick either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Every evetest CI job builds the evetest image when its sources are not
published yet, and each build downloaded all Go modules twice: in the
build stage and again in the final stage. With all suite jobs starting
together, proxy.golang.org often timed out ("TLS handshake timeout")
and the suite failed before running any test.
Copy the module cache from the build stage into the final image instead
of downloading it again, and retry the remaining download.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
When a stress cycle disables PNAC, nim can test the new DPC before the port has an IP from the pre-auth VLAN. The test fails, nim falls back to the previous DPC and retries the new one only after timer.port.testbetterinterval, 10 minutes by default, while the test waits 5 minutes for the port to reach the pre-auth VLAN. Lower the interval to its 60s minimum, as TestNetworkConfigFallback does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Adam answers 400 for a known device until the device posts its certificates, because its file backend reports the missing certs file as a generic error. The harness counted every 400 as a failure and gave up on the ECDH certificate after 11 polls (about 55s), well before deviceAttestTimeout. On a slow host the device posted its certificates 73s after onboarding, and test setup failed. Treat 400 as "not published yet", and raise deviceAttestTimeout from 2 to 5 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
The harness gave the broker 1 minute to power an EVE VM on or off or to reboot it. On Proxmox each is a task that can wait about a minute on a busy host: a VM start waited 61s for the storage lock held by a parallel VM create, and the harness cancelled it 1s before the VM came up. The broker also retries a failed task up to 3 times, which never fit in 1 minute. Raise the three timeouts to 5 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Every cluster test generated a new cluster ID and join token, even when it reused the devices of a cluster the previous test had formed. That cluster keeps running, but the devices kept reporting its old ID: EVE re-reports cluster info only when the cluster's state changes, not on an ID change alone. The cluster readiness wait never accepted those reports, and TestThreeNodesPortScan timed out after TestThreeNodesCluster. The token mattered too: after TestClusterToSingleConversion took edge-dev3 out of the cluster, TestClusterVMApp configured a new token, which the running k3s of edge-dev1 and edge-dev2 kept ignoring, so it rejected edge-dev3 joining with it as "not authorized". When the nodes of a new cluster config are reused devices that still run one cluster, and no other device is in it, keep that cluster's ID and token, as a real controller would. Nodes without a cluster config join it with the same token. The token is kept in plaintext on the harness's device config for this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
When TestTieBreakerCluster reused the devices of the previous cluster test, it got their formed cluster and only made one node the tie-breaker. That conversion should work but is unlikely in practice, and it currently breaks cross-node pod networking: kube-init never reached the end of its tie-breaker setup, failing on an unreachable kubevirt webhook, in both runs that reused a cluster. On fresh devices the test passed. The test is about forming a cluster with a tie-breaker, so create its devices from scratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Suites failed before running any test when a download in the suite's setup failed once: "go install gotestfmt" timed out on proxy.golang.org, and downloading the EVE image artifacts failed despite the short retries of download-artifact itself. Retry go install and the evetest CLI build (which downloads Go modules) up to 3 times through a small retry.sh, and download the EVE image artifacts once more after a minute if the first download fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Some evetest hosts have much slower storage and CPU, and the long cluster suites take 2-3x longer there and tend to time out. Add evetest.WithTags, an Init option that tags a test or a suite, and the TagHeavyTest tag, which declares that it needs a powerful host. The option has no effect at run time: list-tests reads it from the source, shows the tags, and "make list-test-suites" prints those of each suite after its name. The nightly and PR workflows run heavy suites only on runners labelled evetest-runner, while all other suites may also use runners labelled only evetest-runner-light. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
7e95422 to
edf3067
Compare
evetest: skip decisions for this PRThis PR only touches the Skipped whole suites (unrelated to anything in this PR):
Everything else ( An engineer can comment |
TestControllerFaultsSuite: failure analysisTestControllerFaultsSuiteFailureClaude's conclusionThis confirms nothing in the PR touches the broker-dial/network path itself — only build caching and timeout-constant changes. I have enough evidence now. Root causeThis suite failed at the very first step of The ledger shows this same broker endpoint ( The PR (#7024) only touches evetest Docker build caching/retries, CI workflow retry logic, and broker RPC timeout constants (power on/off/reboot raised from 1→5 min) — none of which affects the initial gRPC dial to the broker, and the failure happened before any of that code could even execute. This looks environmental (broker host/network transiently unreachable), unrelated to the PR. |
TestDeviceConnectivitySuite: failure analysisTestDeviceConnectivitySuiteFailureClaude's conclusionRoot causeLive evidence shows this failed at the very first step of the suite — PR #7024 is directly implicated: it adds a new The ledger does show |
TestUpgradeSuite: failure analysisTestUpgradeSuiteFailureClaude's conclusionRoot cause assessmentThe failure happened instantly (3 seconds into That points directly at this PR (#7024): it adds a new |
TestSecuritySuite: failure analysisTestSecuritySuiteFailureClaude's conclusionI have enough evidence already from Root causeTestSecuritySuite failed 4 seconds into harness This is not a new or isolated event: it's the third occurrence tonight in this same run (#8), byte-identical to the already-logged PR #7024's diff touches |
TestBootstrapSuite: failure analysisTestBootstrapSuiteFailureClaude's conclusionThat's the exact line that failed, and the PR diff doesn't touch this code path at all, confirming the failure is unrelated to the PR. Root cause
This shares the exact same root cause as two other failures already recorded earlier in tonight's run ( I checked PR #7024's diff: it only touches CI-script retry logic (artifact download retries, |
TestAppsSuite: failure analysisTestAppsSuiteFailureClaude's conclusionRoot cause
This is not an isolated incident tonight: the same exact endpoint and error already hit Per the ledger, this broker endpoint ( PR #7024's diff only touches evetest/CI code (timeout constants in |
Description
Fixes for recurring evetest nightly failures that are caused by the tests or
the test framework, not by EVE. One commit per fix:
lfedge/eveinstead of the run's CI repo, which has no release tags.reboot when a test drops the cluster config a reused device inherited from
the previous test.
registry mirrors the next test does not want is not reused (broke
TestMgmtProxy).
another cluster, and the cluster tests retry
kubectl applyon transientAPI server errors.
CPU <n>/KVM) are no longerflagged as host processes touching the passed-through device.
copy of a log record is deleted once sent and made the count drop.
already consumed (ACL, VLAN, STP and routing tests).
timer.port.testinterval, which a DPCretest can depend on.
that cluster's ID and join token instead of generating new ones. EVE never
re-reported a new ID (broke TestThreeNodesPortScan), and a node rejoining
the cluster with a new token was rejected as "not authorized" (broke
TestClusterVMApp).
and rebooting it 5 minutes instead of 1, which a slow Proxmox task with
the broker's retries didn't fit in.
which it does until the device posts its certificates (broke
TestInfoRetriedAfterServerError setup on a slow host).
timer.port.testbetterinterval, which the return to aDPC that failed its first test depends on.
cluster turned the test into converting a running cluster to one with a
tie-breaker.
go install, the evetest CLI build and the EVEimage artifact download, whose one-off failures failed whole suites.
retries. Suites often failed at image build with a proxy.golang.org
"TLS handshake timeout".
with
evetest.WithTags(evetest.TagHeavyTest)in itsInitcall. Thetags are shown by
make list-testsandmake list-test-suites, and thenightly and PR workflows run heavy suites (the EVE-k focused ones and
TestApplicationConnectivitySuite) only on runners labelled
evetest-runner. All other suites also run on runners labelledevetest-runner-light. Nightly runs showed the suites taking 2 to 4times longer on the weaker farfalle host, with the cluster suites failing
there.
gh-runner-installer.shaccepts several--gh-labelvalues.How to test and validate this PR
Run the evetest suites on this PR (
/evetest all). The affected tests areTestKvmToKRepartitionGeometry, TestBackupCorruptRestore, Shrink,
PersistTooFull, ZFSPersist, TestAppInstancePurge, TestMgmtProxy,
TestNativeKubeAppNetworking, TestVGAPassthroughNoHostAccess,
TestVaultKeyModeRecoveryOnExt4/OnZFS, TestLocalNetInstanceACLs,
TestSwitchNetInstanceACLs, TestAccessVLANs, TestSwitchNIWithMultiplePorts,
TestLocalNIWithMultiplePorts, TestNetworkConfigFallback,
TestPNACWithoutProxy/WithProxy, TestThreeNodesPortScan,
TestBootstrapWithMgmtVLAN, TestInfoRetriedAfterServerError,
TestTieBreakerCluster and TestClusterVMApp. The harness changes are also covered by
unit tests in
evetest/(go test .).Changelog notes
No user-facing changes.
PR Backports
Checklist
🤖 Generated with Claude Code