Repository navigation
evetest: run several tests in parallel with a dashboard UI - #6783
Conversation
07cb995 to
95b0c2b
Compare
|
Claude finished @milan-zededa's task in 3m 57s —— View job Review completeI reviewed the orchestrator, instance registry, CLI, and the three UI backends. This is a well-structured, self-contained layer on top of existing evetest; the test framework, tests and CI are untouched. Code quality is high (atomic registry writes, careful locking, thoughtful ownership/pid-reuse handling). I posted two minor inline findings:
Nothing blocking. A couple of smaller observations that I did not file as inline comments:
|
e6520c5 to
4b61366
Compare
|
Addressed the remaining review observations:
|
4b61366 to
d5e1fff
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6783 +/- ##
==========================================
+ Coverage 28.09% 28.39% +0.29%
==========================================
Files 534 545 +11
Lines 97719 99456 +1737
==========================================
+ Hits 27453 28236 +783
- Misses 68029 68795 +766
- Partials 2237 2425 +188 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@milan-zededa I'm testing it. I found an issue with the same ip address of nodes but i used manual ssh. I need to recheck it |
9af8069 to
e6c9cd5
Compare
|
@milan-zededa could you rebase onto master? The |
EVETEST_SKIP takes a comma-separated list of test/suite names to skip from evetest.Init. A suite is skipped by its own function name; a suite subtest variant (e.g. TestBootstrapWithAutoDiscoveredProxy) can be skipped either by its own variant name or by the name of the underlying test function shared by all of its variants (e.g. TestBootstrapWithProxy), which skips every variant at once. Signed-off-by: Milan Lenco <milan@zededa.com>
"make list-test-suites" (list-tests -suites-only) prints a plain list of test suite names, one per line, for scripting -- e.g. generating a CI job matrix without needing a host-side Go toolchain. Signed-off-by: Milan Lenco <milan@zededa.com>
Every evetest container now registers itself in ~/.evetest/instances/ while running. The CLI uses the registry to pick the instance to talk to, in this order: --instance/-i, EVETEST_INSTANCE, an explicit EVETEST_API_PORT/EVETEST_API_ADDRESS, the instance chosen with "evetest use" in the current terminal, the only running instance, and finally an interactive menu when several are running. The choice made by "evetest use" is remembered per terminal session (keyed by the session leader and its start time), not globally. A registered instance is considered gone only when connecting to it is refused, so that a busy host running several instances does not make a running one disappear from the CLI's view. New commands: instances (ps), use, logs and stop. The CLI connects to an instance only for commands which need one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
"make evetest NAME=a,b,c" runs the tests as parallel instances, at most
EVETEST_PARALLEL at a time (default 1). Without NAME all test suites are
run. A single name behaves as before.
The Makefile starts an orchestrator container ("evetest run", named
evetest-orchestrator-<run-id>) on the host network with the Docker socket
mounted. It allocates a free API port per test and starts one evetest
container per test, reusing the docker options the Makefile assembles for
a single run. Results are summarized at the end and make fails if any
test failed. EVETEST_FAIL_FAST cancels the remaining tests on the first
failure. A first Ctrl-C stops the instances gracefully, a second one
kills them.
A test stopped from outside ("evetest stop", docker stop, Ctrl-C) is
reported as cancelled, not passed: the container entrypoint now exits
with 128 plus the signal number instead of 0, and the orchestrator reports
such a job as cancelled, which does not trigger fail-fast.
EVETEST_UI selects the presentation: tmux (default on a terminal, one
pane per instance, a failed test's pane moves to its own window), prefixed
(interleaved output with a per-test prefix) and plain (status changes and
the tail of failed tests, default without a terminal). The orchestrator
gets a terminal for the tmux UI. The full output of every test is written
to ~/.evetest/runs/<run-id>/.
The EVE version under test is shown in the tmux status line and
orchestrator window, and printed at the start of the plain and prefixed
output. The Makefile passes it to the orchestrator container.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
EVETEST_UI=dashboard shows a multi-test run on a single screen: a table with one row per test (state, time, last line of output) and a full- screen scrollable output view of the selected test. A running test reports whether it is paused at a checkpoint or after a failure. From the screen the user can open an ssh session to one of the test's EVE devices (a menu asks which when there are several), resume a paused test, stop a test, or stop everything. It stays open after the last test finishes, until the user quits. It is an alternative to the tmux UI, which remains the default on a terminal. The status query of an instance moves from the CLI into the instances package so that both use it. The EVE version under test is shown in bold at the right end of the first line of the dashboard. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
The entrypoint ran "go test -run ^Name$ ./tests/...", which builds every test package and prints "no tests to run" for all but one of them. It also reported success for a name which no test defines. Find the package which defines the test and run only that one. A name which no package defines is now an error. Signed-off-by: Milan Lenco <milan@zededa.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TestClusterVMApp and TestClusterPVCRedownload were not part of any test suite and therefore never ran in CI. Add them, and drop their notes about being left out. TestNodeClusterSuite had grown too long with them. Keep the functional cluster tests in it and move the fault-injecting and failure-handling ones (VMIRS stranded replicas, DNID and backup DNID, purge during failover, PVC redownload) into the new TestNodeClusterResilienceSuite, so the two can run in parallel. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
…elves Let Init take options and add WithControllerFaults, which the controller faults suite and each of its tests pass so that the fault proxy is on without the user setting EVETEST_CONTROLLER_FAULTS. An explicitly set EVETEST_CONTROLLER_FAULTS still wins, so the proxy can be forced on for every test or off to make the faults tests skip. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Pressing "r" in the dashboard queues the selected test or suite, once it is done, to run again in the same slot pool as the other tests. The test code under tests/, netmodels/ and matchers/ is read afresh when it starts, so edits made since the previous run are picked up. Before it is queued, the dashboard asks for the evetest and SDN image versions to run with, offering those of the previous run. This is how changes to the framework and the SDN, which are baked into the images, are picked up: images which are not local are pulled, and the rerun is refused if that fails. Pressing "x" on a test which is still queued cancels it, so that it does not run, until it is rerun. The orchestrator keeps a queue which accepts jobs after the first pass, reports idleness each time it is reached, and can be waited for. The output of an earlier run stays in the output view behind a divider and each run logs to a file of its own (<test>.2.log, ...). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
The recovery waits of TestIntermittentConnectivity and TestLACPBond now span more than one periodic connectivity test, which runs every 5 minutes by default, so a port error cleared only by the next test no longer depends on where that test falls. TestVMIRSStrandedReplicasRecovery waits longer for the app. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
The test isolates the second node by bringing down its cluster interconnect port, but looked it up as "dev2-eth1" while SeparateClusterPort names the port "<device name>-eth1", which is "edge-dev2-eth1". No port matched, so the node was never isolated and rejoined the cluster, failing the check that only the two surviving nodes report Ready. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
Check the errors returned by the cleanup calls, or ignore them explicitly where nothing can be done about a failed cleanup, use client.New instead of the deprecated client.NewClientWithOpts, drop a redundant embedded field selector, a blank assignment and a comparison of identical expressions, and make explicit that a fatal test failure ends the upgrade path. Exclude the cloud-init user data of the broker from yamllint, since it has to start with the "#cloud-config" header, and the Proxmox hook script from yetus, because perlcritic reports "source OK" for it, which yetus counts as an issue. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com>
13d9049 to
6be49e9
Compare
Done, let's see how it goes now. |
Description
The GitHub CI will run the evetest suites in a per-suite matrix (see #6680), so the suites run in parallel simply by
being spread across runners. Developers need a similar capability on their own machine: to run several tests in
parallel and make better use of their hardware, instead of running them one by one.
This PR adds that on top of the existing evetest.
make evetest NAME=TestA,TestB,TestC(ormake evetestwithoutNAMEfor all test suites) starts one evetest instance per test, each in its own container with its own API port, andEVETEST_PARALLELlimits how many run at the same time. What was added:evetest runcommand, run bymakein a container, which starts and watches the instances,writes the full output of each test to
~/.evetest/runs/<run-id>/<test>.log, prints a summary table at the endand exits non-zero if any test failed. A test stopped from outside is reported as cancelled, and
EVETEST_FAIL_FAST=truecancels the remaining tests after the first failure.EVETEST_UI):dashboard, the default on a terminal: one screen with a table of the tests and a scrollable output view of theselected one, with keys to open an ssh session to the EVE device, resume a paused test or stop a test.
tmux: one pane per test, a failed test moves to a window of its own.prefixed: interleaved output of all tests with a test name prefix.plain: status changes only, the default without a terminal. It is meant for runs with no one watching thescreen, e.g. started by a script or an AI agent, or with
nohupon a remote host; the full output of everytest is in its log file.
evetestCLI finds them without theport:
evetest instances(ps),evetest use <name>(remembered per terminal),-i <name>for a singlecommand, and the new
logsandstopcommands. With one running instance, nothing needs to be specified.The
evetestCLI is a separate host binary: install the new one (make -C evetest install-cli) to use all of this.EVETEST_SKIPto skip named tests and suites, amake list-test-suitestarget, and theentrypoint now runs only the Go package which defines the test (a mistyped test name is an error instead of a
successful run of nothing).
This work does not require a deep review. It is only an orchestrator and UI layer on top of the existing evetest,
the framework, the tests and the CI are not changed (and EVE is not touched at all of course).
How to test and validate this PR
The
evetestCLI is a separate binary on the host, so first install the new one. Without it, the CLI does not knowthe new commands (
instances,use,logs,stop,-i), and it cannot find the instances started by the newmake evetest:Run several tests in parallel and check the output of each UI:
makeexits with 0 whenall of them passed.
evetest/README.md.With the tests still running, in another terminal, try the CLI:
Stopped tests are reported as cancelled in the summary.
Try
EVETEST_FAIL_FAST=truetogether with a test which fails (for example one which is expected to fail onyour hardware) and check that the other tests are cancelled.
make evetest NAME=<single test>behaves as before, with a short "Starting container" line instead of the fulldocker runcommand. A name which no test defines now fails with an error.Changelog notes
None. This only changes how developers run the tests of the evetest framework. There are no user-facing changes.
PR Backports
Checklist
🤖 Generated with Claude Code