Skip to content

[13.4-stable] evetest: sync with master and adapt the tests to EVE 13.4 - #6899

Merged
milan-zededa merged 6 commits into
lf-edge:13.4-stablefrom
milan-zededa:evetest-13.4-pr
Oct 26, 2026
Merged

milan-zededa merged 6 commits into
lf-edge:13.4-stablefrom
milan-zededa:evetest-13.4-pr

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Description

Bring evetest to 13.4-stable and make it work with EVE 13.4.

The PR has six commits:

  1. evetest: add the image publishing workflow to 13.4-stable: the workflow
    that builds and publishes the evetest images from this branch.
  2. Copy evetest from master: evetest is made to be in sync with master.
  3. evetest: adapt tests to EVE 17.0: the same adaptation as in the 17.0-stable
    and 16.0-stable PRs, kept as a separate commit so that the changes on top of
    it are easy to review. It removes the tests of features that are not in 17.0,
    makes UpdateApplication request a purge for adapter changes (as 17.0
    requires) and points evetest/go.mod to the 17.0-stable pillar.
  4. evetest: adapt tests to EVE 16.0: the same adaptation as in [16.0-stable] evetest: sync with master and adapt the tests to EVE 16.0 #6842
    (16.0-stable): eve-k and everything that exists only for it is removed from
    the framework and the test suites, and the tests of features missing from
    16.0 are dropped (TestPNAC, TestHardwareInventory, TestContainerdCPUs).
  5. evetest: adapt tests to EVE 14.5: the same adaptation as in [14.5-stable] evetest: sync with master and adapt the tests to EVE 14.5 #6873
    (14.5-stable). Vector-based log tests, bond and LACP tests, TestVCom,
    TestSwitchNIPortConfigRace and the tests that need fixes not backported
    there are removed, the periodic device info is no longer used as the "EVE
    still reaches the controller" signal, and the DNS tests expect the DHCP and
    static DNS servers combined.
  6. evetest: adapt tests to EVE 13.4: on top of that, the tests are made to
    match what 13.4 supports, and evetest/go.mod is updated to point to the
    pillar from 13.4-stable:
    • IPv6 configuration received from dhcpcd and multiple/domain-based NTP
      servers are not in 13.4, so TestDeviceIPv6Connectivity,
      TestApplicationIPv6Connectivity, TestDeviceNTPConfig and
      TestApplicationNTPConfig are removed, together with the network models
      that only these tests used.
    • debug.kernel.remote.loglevel and debug.syslog.remote.loglevel do not
      exist in 13.4, so TestRemoteLogLevelNone is removed and TestDeviceLogs
      sets only the local kernel log level.
    • The upgrade test starts from 13.4.3-lts. The OCI datastore variant starts
      from the 13.4-stable snapshot 0.0.0-13.4-stable-e7581dbd, because the
      downloader fix it needs is not part of any 13.4.x release yet.

How some tests had to be adapted specifically for 13.4

  • ControllerCert is persistent. 13.4 keeps the ControllerCert
    publication in /persist/status/zedagent instead of in memory (/run).
    WaitUntilHasControllerCert and the cert-change tests
    (TestControllerCertChange, TestEncryptedControllerCertChange) read it
    from the persistent path. The post-reboot step of these tests can no longer
    prove that zedagent refetched the certificates (the publication survives the
    reboot), so it only checks that both certificates are still published.
  • Local NI DHCP uses an all-ones netmask. 13.4 hands out /32 leases and
    rewrites application-to-application gateway routes to the NI bridge (dropped
    in later releases). TestApplicationGateway therefore accepts the propagated
    route via the NI bridge as well as via the app gateway
    (default via 172.28.2.[12]).
  • Static DNS servers are replaced, not appended. 13.4 replaces the static
    DNS servers of a port with the ones learned from dhcpcd. In
    TestDNSFunctionality the expected DNS servers of the first phase are the
    DHCP-provided ones only, and TestVGAPassthroughNoHostAccess no longer waits
    for the added DNS server to be reported.
  • SFTP datastore address. The eve-libs version used by the 13.4 pillar
    treats sftp://host:port as a bare host name and fails to resolve it
    (lookup sftp on ...: no such host). The framework (SFTPStorage.toProto)
    therefore sends the datastore FQDN as plain host:port. This is a framework
    change, not a test change, and it works with newer EVE versions as well.
  • Transient "Retry download" error. In 13.4 a volume can briefly carry a
    NOTICE-severity error while the datastore config is still being applied.
    The storage tests' volumeHasError helper ignores it.

Some tests need fixes that are not in 13.4-stable yet and are backported in
separate PRs:

PR dependencies

None.

How to test and validate this PR

Run the tests against a build of 13.4-stable (with the backports above for
the tests listed there), for example:

EVETEST_EVE_VERSION=<13.4-stable build> make evetest NAME=<TestFunctionName>

Changelog notes

No user-facing changes (evetest only).

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

Publish the evetest, broker, SDN and testapps images from 13.4-stable
the same way as from master.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 32.66%. Comparing base (8077a3d) to head (72812da).
⚠️ Report is 328 commits behind head on 13.4-stable.

Additional details and impacted files
@@               Coverage Diff               @@
##           13.4-stable    #6899      +/-   ##
===============================================
+ Coverage        24.78%   32.66%   +7.88%     
===============================================
  Files                8        9       +1     
  Lines             1138     1090      -48     
===============================================
+ Hits               282      356      +74     
+ Misses             788      654     -134     
- Partials            68       80      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

milan-zededa and others added 4 commits October 22, 2026 11:50
Start with a copy of evetest from master. Test changes required
to match EVE 17.0 will be done in the follow-up commits.

The copy also includes a few fixes that are not in master yet: more
generous timeouts in tests that depended on the periodic connectivity
test or on slow app start, the port label used to isolate a node in
TestClusterPVCRedownload, and the issues reported by yetus.

It also carries the evetest changes of lf-edge#6781 (TestAppTeardownWithRemovedAdapter)
and lf-edge#6794 (TestMgmtProxy CDI HTTP DataVolume cases and timeouts), which are
in master but were missing from the first copy.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Point evetest at the pillar from 17.0-stable (the eve-api version it
uses is unchanged) and make the tests match what 17.0 supports.

Framework: on 17.0 a change of the adapters of a deployed app is
rejected unless a purge is requested, so UpdateApplication now bumps the
purge counter instead of the restart counter for such changes.

Removed tests of features that are not in 17.0: the konvert suite,
vault key mode recovery, deferred queue metrics, native Kubernetes app
networking, the USB hub chain test, and the NIC count, ordered
interface, staged NIC and passthrough NIC change tests. The dynamic NIC
addition part of TestNetworkAdapterPassthrough and the usbaddr wildcard
phase of TestUSBFlashDriveHotplug are removed as well.

Adjusted tests to the 17.0 behavior: TestDNSFunctionality and the LPS
network test do not rely on the management dnsmasq (DNS servers are read
from /etc/resolv.conf), and the NTP servers reported for an app VIF are
only the NI-level ones. The upgrade test starts EVE-K from 17.0.1-rc2,
the first 17.0 tag with the longhorn disk path fix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Point evetest at the pillar from the tip of 16.0-stable. The eve-api
version used by that pillar is the one evetest already referenced.

The kubevirt hypervisor (eve-k) is not supported in 16.0-stable, so
remove it from the framework together with everything that exists only
for it: HypervisorKubevirt, the EdgeCluster and cluster config builders,
the cluster-info gRPC calls and "evetest cluster" command, the kubectl
helpers, RemoteConsole and DialViaSSH, and the K3s registry mirror setup.

Remove the tests that cover eve-k only (the cluster and kuberegistration
suites, the VMIRS purge test, the vault zvol trim test and the volume
size alignment test). Tests that run on both flavors are kept without
their eve-k specific parts.

TestPNAC is dropped as well, since SCEP enrollment is not implemented in
16.0-stable. The OCI datastore upgrade variant starts from the latest
16.0.x release (16.0.2-lts); if it fails because of the missing
downloader fix, backport the fix and bump the version.

TestHardwareInventory and TestContainerdCPUs are dropped too: 16.0-stable
does not report HwInventorySupport (lf-edge#5535) and has no ctrd_max_vcpus
confinement of containerd (lf-edge#6771).

The 16.0 sshd has a broken "Subsystem sftp" path, so scp over SFTP never
works there. scp now falls back to the legacy protocol (-O) when the SFTP
attempt fails, escaping the remote path with backslashes. The PCI access
tracer builds eve-tools/bpftrace-compiler from its vendor directory only
when the checkout has one, which 16.0-stable does not.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Point evetest at the pillar from the tip of 14.5-stable. The eve-api
version used by that pillar is the one evetest already referenced.

Vector-based log processing (vector.config) is not present in 14.5, so
the Vector tests of the telemetry suite are removed together with their
helpers and test data. timer.deviceinfo.interval is not configurable
either; the periodic device info is published every 10 minutes +/- 20%.
TestIntermittentConnectivity, TestDeviceIPv6Connectivity and the
app-routed management path test now use the periodic device metrics
(every 20s) as the "EVE still reaches the controller" signal, while the
deferred info test waits up to 13 minutes for the next device info.

Bond status reporting and 802.3ad bond creation, LPS network
configuration and the HTTP/protobuf vcomlink API are not available in
14.5, so remove TestActiveBackupBond, TestLACPBond,
TestVLANSubinterfacesOnTopOfLAGs, TestBootstrapWithLACPBond,
TestNetworkLocalChanges and TestVCom. TestSwitchNIPortConfigRace is
removed too, since it needs the IfInstanceID tracking of NIM interfaces
and bridges, whose backport would be very large. TestInfoRetriedAfterServerError,
TestDownloadBurstWithDeadMgmtPorts, TestHaltUnresponsiveGuest and
TestSwitchNIDeletedWithAppsAttached are removed as well, because they need
fixes that are not backported. The network models only these tests used are
removed too.

TestDeviceInfo waits up to 13 minutes for a device info that carries the
storage size, which the info sent right after the config can lack.
TestAppLocalInfo discards buffered app updates before issuing an LPS
command, so that a stale HALTING does not end the wait early.

DnsConfigExclusively is not honored by 14.5, so TestDNSFunctionality
expects the DHCP and static DNS servers of ethernet2 combined, its
/etc/resolv.conf check expects only the DHCP-provided servers (14.5 does
not write static ones there), and its broken-DNS fallback phase uses
ethernet3 alone.

The upgrade test starts from 14.5.0-lts. The OCI datastore variant starts
from the 14.5-stable snapshot 0.0.0-14.5-stable-42a70aa5, because the
downloader fix it needs is not part of any 14.5.x release yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Point evetest at the pillar from the tip of 13.4-stable. The eve-api
version used by that pillar is the one evetest already referenced.

debug.kernel.remote.loglevel and debug.syslog.remote.loglevel do not
exist in 13.4, so TestRemoteLogLevelNone is removed and TestDeviceLogs
only sets the local kernel log level.

The following tests are removed because the features they check are not
present in 13.4: TestDeviceIPv6Connectivity and
TestApplicationIPv6Connectivity (IPv6 configuration received from
dhcpcd), and TestDeviceNTPConfig and TestApplicationNTPConfig (multiple
and domain-based NTP servers, NtpServerExclusively). The network models
only these tests used are removed too.

EVE 13.4 keeps the ControllerCert publication in /persist/status instead
of in memory, so the certificate change tests read it from there and the
post-reboot check no longer proves a refetch. 13.4 hands out DHCP leases
with an all-ones netmask, so TestApplicationGateway accepts the propagated
route via the NI bridge as well as via the app gateway. 13.4 replaces the
static DNS servers of a port with the ones learned from dhcpcd, so
TestVGAPassthroughNoHostAccess does not wait for the added DNS server to
be reported.

The upgrade test starts from 13.4.3-lts. The OCI datastore variant starts
from the 13.4-stable snapshot 0.0.0-13.4-stable-e7581dbd, because the
downloader fix it needs is not part of any 13.4.x release yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa merged commit bfa78fd into lf-edge:13.4-stable Oct 26, 2026
43 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants