Skip to content

evetest: run several tests in parallel with a dashboard UI - #6783

Merged
milan-zededa merged 12 commits into
lf-edge:masterfrom
milan-zededa:evetest-orchestrator
Oct 18, 2026
Merged

milan-zededa merged 12 commits into
lf-edge:masterfrom
milan-zededa:evetest-orchestrator

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

The GitHub CI will run the evetest suites in a per-suite matrix (see #6680), so the suites run in parallel simply by
being spread across runners. Developers need a similar capability on their own machine: to run several tests in
parallel and make better use of their hardware, instead of running them one by one.

This PR adds that on top of the existing evetest. make evetest NAME=TestA,TestB,TestC (or make evetest without NAME for all test suites) starts one evetest instance per test, each in its own container with its own API port, and
EVETEST_PARALLEL limits how many run at the same time. What was added:

  • Orchestrator. A new evetest run command, run by make in a container, which starts and watches the instances,
    writes the full output of each test to ~/.evetest/runs/<run-id>/<test>.log, prints a summary table at the end
    and exits non-zero if any test failed. A test stopped from outside is reported as cancelled, and
    EVETEST_FAIL_FAST=true cancels the remaining tests after the first failure.
  • UIs (EVETEST_UI):
    • dashboard, the default on a terminal: one screen with a table of the tests and a scrollable output view of the
      selected one, with keys to open an ssh session to the EVE device, resume a paused test or stop a test.
    • tmux: one pane per test, a failed test moves to a window of its own.
    • prefixed: interleaved output of all tests with a test name prefix.
    • plain: status changes only, the default without a terminal. It is meant for runs with no one watching the
      screen, e.g. started by a script or an AI agent, or with nohup on a remote host; the full output of every
      test is in its log file.
  • CLI support. Running evetest containers register themselves, so the evetest CLI finds them without the
    port: evetest instances (ps), evetest use <name> (remembered per terminal), -i <name> for a single
    command, and the new logs and stop commands. With one running instance, nothing needs to be specified.
    The evetest CLI is a separate host binary: install the new one (make -C evetest install-cli) to use all of this.
  • Smaller additions. EVETEST_SKIP to skip named tests and suites, a make list-test-suites target, and the
    entrypoint now runs only the Go package which defines the test (a mistyped test name is an error instead of a
    successful run of nothing).

This work does not require a deep review. It is only an orchestrator and UI layer on top of the existing evetest,
the framework, the tests and the CI are not changed (and EVE is not touched at all of course).

How to test and validate this PR

The evetest CLI is a separate binary on the host, so first install the new one. Without it, the CLI does not know
the new commands (instances, use, logs, stop, -i), and it cannot find the instances started by the new
make evetest:

make -C evetest install-cli
  1. Run several tests in parallel and check the output of each UI:

    EVETEST_PARALLEL=2 make evetest NAME=TestDeviceInfo,TestDeviceMetrics,TestDeviceLogs
    EVETEST_UI=tmux EVETEST_PARALLEL=2 make evetest NAME=TestDeviceInfo,TestDeviceMetrics
    EVETEST_UI=prefixed make evetest NAME=TestDeviceInfo,TestDeviceMetrics
    EVETEST_UI=plain make evetest NAME=TestDeviceInfo,TestDeviceMetrics
    • Two tests run at a time, the third one waits in the queue until a slot is free.
    • At the end a summary with the result and duration of each test is printed, and make exits with 0 when
      all of them passed.
    • The keys of each UI are documented in the section "Running Several Tests in Parallel" of evetest/README.md.
  2. With the tests still running, in another terminal, try the CLI:

    evetest instances
    evetest use TestDeviceInfo
    evetest status
    evetest -i TestDeviceMetrics status
    evetest logs
    evetest stop --all

    Stopped tests are reported as cancelled in the summary.

  3. Try EVETEST_FAIL_FAST=true together with a test which fails (for example one which is expected to fail on
    your hardware) and check that the other tests are cancelled.

  4. make evetest NAME=<single test> behaves as before, with a short "Starting container" line instead of the full
    docker run command. A name which no test defines now fails with an error.

Changelog notes

None. This only changes how developers run the tests of the evetest framework. There are no user-facing changes.

PR Backports

  • 17.0-stable: To be backported later, together with the evetest framework as a whole.
  • 16.0-stable: To be backported later, together with the evetest framework as a whole.
  • 14.5-stable: To be backported later, together with the evetest framework as a whole.
  • 13.4-stable: To be backported later, together with the evetest framework as a whole.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't check them.

🤖 Generated with Claude Code

@milan-zededa milan-zededa added the testing New test or test framework changes label Oct 13, 2026
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@claude

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @milan-zededa's task in 3m 57s —— View job


Review complete

I reviewed the orchestrator, instance registry, CLI, and the three UI backends. This is a well-structured, self-contained layer on top of existing evetest; the test framework, tests and CI are untouched. Code quality is high (atomic registry writes, careful locking, thoughtful ownership/pid-reuse handling). I posted two minor inline findings:

  • orchestrator.go — ports reserved on failed ErrPortBusy retry attempts are leaked for the orchestrator's lifetime; only the last port is released. Bounded and rare, so low impact.
  • tmux.go — closing a test's pane returns exit 137 with ctx not cancelled, so the job is reported as FAILED rather than Cancelled.

Nothing blocking. A couple of smaller observations that I did not file as inline comments:

  • EVETEST_SKIP is honored when expanding "all suites" (allSuites), but not when an explicit NAME=TestA,TestB list is given — a skipped test in that list still starts an instance and then reports as passed via t.Skipf. Minor inconsistency.

  • The docker-args file is written with printf '%s\n' $(DOCKER_COMMON_ARGS), which word-splits on spaces, so any mount path or env value containing a space would break. This mirrors the pre-existing unquoted docker run expansion, so not a regression.

  • Read the diff and new files

  • Analyze orchestrator, instances, CLI, UI code

  • Post inline review comments
    • branch evetest-orchestrator

Comment thread evetest/orchestrator/orchestrator.go Resolved
Comment thread evetest/orchestrator/tmux.go Resolved
@milan-zededa
milan-zededa force-pushed the evetest-orchestrator branch 2 times, most recently from e6520c5 to 4b61366 Compare October 14, 2026 07:46
@milan-zededa

Copy link
Copy Markdown
Contributor Author

Addressed the remaining review observations:

  • docker-args word-splitting: the docker arguments built by evetest/Makefile are now shell-quoted (new shq helper), so paths and values with spaces or quotes stay single arguments, in both the single-run and the multi-test path.
  • EVETEST_SKIP is intentionally not applied to an explicit NAME=a,b list: naming a test explicitly overrides the skip list.

@codecov

codecov Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.39%. Comparing base (d06c165) to head (6be49e9).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6783      +/-   ##
==========================================
+ Coverage   28.09%   28.39%   +0.29%     
==========================================
  Files         534      545      +11     
  Lines       97719    99456    +1737     
==========================================
+ Hits        27453    28236     +783     
- Misses      68029    68795     +766     
- Partials     2237     2425     +188     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rucoder

rucoder commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

@milan-zededa I'm testing it. I found an issue with the same ip address of nodes but i used manual ssh. I need to recheck it

@eriknordmark

Copy link
Copy Markdown
Contributor

@milan-zededa could you rebase onto master? The eve (amd64, k, ai-generic) build fails with "Unsupported combination of ZARCH=amd64 and PLATFORM=ai-generic": the build workflow from master includes ai-generic, but this branch's kernel-version.mk predates it. Re-running won't fix it.

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

milan-zededa and others added 5 commits October 17, 2026 18:30
EVETEST_SKIP takes a comma-separated list of test/suite names to skip
from evetest.Init. A suite is skipped by its own function name; a
suite subtest variant (e.g. TestBootstrapWithAutoDiscoveredProxy) can
be skipped either by its own variant name or by the name of the
underlying test function shared by all of its variants (e.g.
TestBootstrapWithProxy), which skips every variant at once.

Signed-off-by: Milan Lenco <milan@zededa.com>
"make list-test-suites" (list-tests -suites-only) prints a plain list of
test suite names, one per line, for scripting -- e.g. generating a CI
job matrix without needing a host-side Go toolchain.

Signed-off-by: Milan Lenco <milan@zededa.com>
Every evetest container now registers itself in ~/.evetest/instances/
while running. The CLI uses the registry to pick the instance to talk to,
in this order: --instance/-i, EVETEST_INSTANCE, an explicit
EVETEST_API_PORT/EVETEST_API_ADDRESS, the instance chosen with
"evetest use" in the current terminal, the only running instance, and
finally an interactive menu when several are running.

The choice made by "evetest use" is remembered per terminal session (keyed
by the session leader and its start time), not globally.

A registered instance is considered gone only when connecting to it is
refused, so that a busy host running several instances does not make a
running one disappear from the CLI's view.

New commands: instances (ps), use, logs and stop. The CLI connects to an
instance only for commands which need one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
"make evetest NAME=a,b,c" runs the tests as parallel instances, at most
EVETEST_PARALLEL at a time (default 1). Without NAME all test suites are
run. A single name behaves as before.

The Makefile starts an orchestrator container ("evetest run", named
evetest-orchestrator-<run-id>) on the host network with the Docker socket
mounted. It allocates a free API port per test and starts one evetest
container per test, reusing the docker options the Makefile assembles for
a single run. Results are summarized at the end and make fails if any
test failed. EVETEST_FAIL_FAST cancels the remaining tests on the first
failure. A first Ctrl-C stops the instances gracefully, a second one
kills them.

A test stopped from outside ("evetest stop", docker stop, Ctrl-C) is
reported as cancelled, not passed: the container entrypoint now exits
with 128 plus the signal number instead of 0, and the orchestrator reports
such a job as cancelled, which does not trigger fail-fast.

EVETEST_UI selects the presentation: tmux (default on a terminal, one
pane per instance, a failed test's pane moves to its own window), prefixed
(interleaved output with a per-test prefix) and plain (status changes and
the tail of failed tests, default without a terminal). The orchestrator
gets a terminal for the tmux UI. The full output of every test is written
to ~/.evetest/runs/<run-id>/.

The EVE version under test is shown in the tmux status line and
orchestrator window, and printed at the start of the plain and prefixed
output. The Makefile passes it to the orchestrator container.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
EVETEST_UI=dashboard shows a multi-test run on a single screen: a table
with one row per test (state, time, last line of output) and a full-
screen scrollable output view of the selected test. A running test
reports whether it is paused at a checkpoint or after a failure. From
the screen the user can open an ssh session to one of the test's EVE
devices (a menu asks which when there are several), resume a paused
test, stop a test, or stop everything. It stays open after the last test
finishes, until the user quits.

It is an alternative to the tmux UI, which remains the default on a
terminal. The status query of an instance moves from the CLI into the
instances package so that both use it.

The EVE version under test is shown in bold at the right end of the
first line of the dashboard.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
milan-zededa and others added 7 commits October 17, 2026 18:30
The entrypoint ran "go test -run ^Name$ ./tests/...", which builds every
test package and prints "no tests to run" for all but one of them. It
also reported success for a name which no test defines.

Find the package which defines the test and run only that one. A name
which no package defines is now an error.

Signed-off-by: Milan Lenco <milan@zededa.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TestClusterVMApp and TestClusterPVCRedownload were not part of any
test suite and therefore never ran in CI. Add them, and drop their
notes about being left out.

TestNodeClusterSuite had grown too long with them. Keep the functional
cluster tests in it and move the fault-injecting and failure-handling
ones (VMIRS stranded replicas, DNID and backup DNID, purge during
failover, PVC redownload) into the new TestNodeClusterResilienceSuite,
so the two can run in parallel.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
…elves

Let Init take options and add WithControllerFaults, which the controller
faults suite and each of its tests pass so that the fault proxy is on
without the user setting EVETEST_CONTROLLER_FAULTS. An explicitly set
EVETEST_CONTROLLER_FAULTS still wins, so the proxy can be forced on for
every test or off to make the faults tests skip.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Pressing "r" in the dashboard queues the selected test or suite, once
it is done, to run again in the same slot pool as the other tests. The
test code under tests/, netmodels/ and matchers/ is read afresh when it
starts, so edits made since the previous run are picked up.

Before it is queued, the dashboard asks for the evetest and SDN image
versions to run with, offering those of the previous run. This is how
changes to the framework and the SDN, which are baked into the images,
are picked up: images which are not local are pulled, and the rerun is
refused if that fails.

Pressing "x" on a test which is still queued cancels it, so that it
does not run, until it is rerun.

The orchestrator keeps a queue which accepts jobs after the first pass,
reports idleness each time it is reached, and can be waited for. The
output of an earlier run stays in the output view behind a divider and
each run logs to a file of its own (<test>.2.log, ...).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The recovery waits of TestIntermittentConnectivity and TestLACPBond
now span more than one periodic connectivity test, which runs every 5
minutes by default, so a port error cleared only by the next test no
longer depends on where that test falls. TestVMIRSStrandedReplicasRecovery
waits longer for the app.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
The test isolates the second node by bringing down its cluster
interconnect port, but looked it up as "dev2-eth1" while
SeparateClusterPort names the port "<device name>-eth1", which is
"edge-dev2-eth1". No port matched, so the node was never isolated and
rejoined the cluster, failing the check that only the two surviving
nodes report Ready.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Check the errors returned by the cleanup calls, or ignore them
explicitly where nothing can be done about a failed cleanup, use
client.New instead of the deprecated client.NewClientWithOpts, drop a
redundant embedded field selector, a blank assignment and a comparison
of identical expressions, and make explicit that a fatal test failure
ends the upgrade path.

Exclude the cloud-init user data of the broker from yamllint, since it
has to start with the "#cloud-config" header, and the Proxmox hook
script from yetus, because perlcritic reports "source OK" for it, which
yetus counts as an issue.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@milan-zededa could you rebase onto master? The eve (amd64, k, ai-generic) build fails with "Unsupported combination of ZARCH=amd64 and PLATFORM=ai-generic": the build workflow from master includes ai-generic, but this branch's kernel-version.mk predates it. Re-running won't fix it.

Done, let's see how it goes now.

@github-actions
github-actions Bot requested a review from eriknordmark October 17, 2026 16:31

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run tests

@milan-zededa
milan-zededa merged commit e11827d into lf-edge:master Oct 18, 2026
59 of 60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants