Skip to content

[17.0-stable] evetest: download burst over mostly dead mgmt ports - #6688

Closed
christoph-zededa wants to merge 10 commits into
lf-edge:17.0-stablefrom
christoph-zededa:backport-6651-evetest-17.0-stable
Closed

christoph-zededa wants to merge 10 commits into
lf-edge:17.0-stablefrom
christoph-zededa:backport-6651-evetest-17.0-stable

Conversation

@christoph-zededa

Copy link
Copy Markdown
Contributor

Description

Backport of #6651 to 17.0-stable, second part: the evetest that reproduces
the download burst, on top of the downloader backport #6684. This PR depends
on #6684 and shows its five commits until that one is merged; its own
commits are the last five.

Two framework commits first, since this branch's evetest predates what the
test needs:

  • evetest: add EVE upgrade testing support — cherry-picked with -x
    from master's 9b72db8, which is where the built-in HTTP image server
    came from: a dummy interface at 244.244.244.244, a file server, and the
    route to it through the SDN. It applies cleanly here except for the upgrade
    test suite itself, which needs RequireEdgeDevice.MinDiskSizeInMiB that
    this branch does not have, so evetest/tests/upgrade is left out; the
    commit message says so.
  • evetest: random image files and standalone content trees — a new
    commit lifting CreateRandomImageFile/AddImageServerFile (from master's
    2bea302) and EdgeDeviceConfig.AddContentTree (from 9ab3799), 64
    lines in total; the two master commits themselves are unrelated to this
    test and do not apply to this branch.

Then the three evetest commits of #6651, cherry-picked with -x:

  • evetest: network model with a single working uplink — clean.
  • evetest: bracket IPv6 addresses in scp targets — clean.
  • evetest: download burst over mostly dead mgmt ports — with three
    adaptations to this branch's framework: evetest.KiB is still evetest.KB
    here; the networking package's helpers_test.go does not exist here, so it
    holds only the helpers this test adds (the other three helpers master keeps
    in that file live in other files on this branch); and EdgeDevice.ReadFile
    returns no error here, so the failure diagnostics read the downloader's
    status files without the error branch.

PR dependencies

#6684, for the downloader fix, the downloader.transport.handlers property
the test sets, and the start-up log line it checks.

How to test and validate this PR

make pkgs eve on this branch, then
make evetest NAME=TestDownloadBurstWithDeadMgmtPorts. On master the run
takes about eight minutes including setup and ended with 100 of 100 content
trees loaded and 295 refused requests retried (see #6651). The framework and
the networking test package compile on this branch; the test has not been run
against a 17.0 image yet, which is what this draft is waiting for. Two things
that run would confirm: the image server route on this branch's SDN, and that
17.0's zedagent reports content-tree errors the way the parking phase waits
for.

Changelog notes

None, test infrastructure only.

PR Backports

  • 16.0-stable, 14.5-stable, 13.4-stable: No, there is no evetest on those
    branches.

Checklist

  • I've provided a proper description
  • I've added the proper documentation — none needed
  • I've tested my PR on amd64 device — not yet run against a 17.0 image (see above)
  • I've tested my PR on arm64 device — not applicable
  • I've written the test verification instructions
  • I've set the proper labels to this PR — stable to be set

For backport PRs:

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

christoph-zededa and others added 10 commits October 1, 2026 00:07
Update eve-libs from 24f79b12a2bd to 779d3f2350aa, the tip of its
17.0-stable branch, which adds one commit: the backport of
lf-edge/eve-libs#113. DronaRequest.WithContext lets the caller hand the
request the context it selects on itself, so that the transport observes
the same cancellation; WithCancel and Cancel, which always derived a
context of their own, are gone. The downloader fix backported next
builds on it.

Pillar and the evetest SDN VM module pin eve-libs; both move to the new
version, and only the four zedUpload files that carry the change are
re-vendored. No transitive dependency moves.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
download() and objectMetadata() hand each transfer to the zedUpload
transport with req.Post() and ignore what it returns. Post() does not
block: the request channel behind it holds eleven entries (the default
number of transport workers) and a request that finds it full is
refused with SyncerRetry. Both callers then sat in `range respChan`
waiting for a reply that no worker would ever produce, because the
stall watchdog only ran on incoming progress messages and a request
nobody picked up sends none. The DownloaderStatus stayed DOWNLOADING at
0 bytes with no error and RetryCount 0, the per-blob handler goroutine
was stuck inside handleCreate, so retries, datastore refreshes and
RefCount changes for that blob were never processed either, and only a
pillar restart recovered. A field node hit this on five container
layers when a 213-object deployment burst re-posted every layer
immediately for the next of four unrouted management ports, at up to 54
requests per second against the eleven-slot queue.

Fix:

- postWithRetry submits the request and retries a SyncerRetry refusal
  with a backoff of 100 ms to 5 s, giving up after maxStalledTime
  (timer.download.stalled) so the attempt then fails like any other
  download error and goes through the regular per-address and retry
  paths instead of hanging. The first refusal of a request is logged.
- awaitResponse replaces the bare `range respChan`: the stall check now
  also runs on a ticker, so a request that never produces a message is
  cancelled after maxStalledTime as well, and a closed channel is still
  reported as before.
- The request's cancel context is created by the caller, so a
  user-initiated cancel also wakes up a submission still waiting for
  room in the queue.

Both zedUpload callers in the downloader get the same treatment; the
transport itself is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit 5bf0c8e)
The downloader hands every download request to its zedUpload transport
through a queue that holds as many requests as the transport has
handlers, eleven, fixed in the library. A request that finds the queue
full is refused (SyncerRetry), which the downloader now retries; the
number itself has been out of reach of the controller. The report of the
field incident asks to consider a larger or configurable depth.

Add downloader.transport.handlers (default 11, the library's default;
1 to 1024). The downloader reads it when it creates its transport, so a
change takes effect at the next reboot, which the docs say; it also
logs the size it started with. A unit test pins the default to the
library's, so that a device that never sets the knob keeps the
transport it always had.

Besides letting a busy node run more handlers, the knob lets a test
shrink the queue to one request, which turns the refusal of a request
under a burst from an occasional event into a certain one.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit f86c85e)
The zedUpload transport posts every progress update and the final
response of a request with a plain send on the response channel the
downloader hands it, from goroutines that live only for that request,
and it never learns that the downloader is gone. The downloader stops
reading the moment it has the final response, or earlier when it gives
up on a request, so the sends that come after that block forever: the
progress reporter's tick that raced the final response and its own
final update after every completed transfer, and the whole tail of an
abandoned one. That is one or two parked goroutines per attempt; on a
node retrying hundreds of downloads over dead ports they add up to
tens of thousands, and pillar's memory grows until its cgroup limit
throttles every service and finally reboots the device, as it did on
the field node.

The library side of this is fixed in eve-libs PR lf-edge#103. This is the
pillar side, which does not depend on it: the downloader owns the
channel, so it gives the transport room for what it posts after the
final response (a buffer of four; two would do) and keeps a reader on
the channel of a request it has given up on until the transport's
final response arrives, which the cancelled request brings about soon.

The dial context watchers that nettrace leaked alongside could not be
stopped from pillar: net/http dials with a context detached from the
request's and drops its cancel function once the dial is done, so only
the library could end them, which eve-libs PR lf-edge#103 does. The eve-libs
bump later in this series brings that fix; the pillar-side handling
here stays as defence in depth.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 03d028e)
Two tests run the downloader's own download function against a local
HTTP server, over the real zedUpload transport, and then require that
no transport goroutine is left trying to post on the response channel,
counted from the goroutine profile. The server pauses halfway through
the blob, so that the transport's progress reporter posts at least one
update before the final response. One test lets the download complete;
the other has the server stall for good and the downloader give up for
lack of progress, with the stall timer shortened, so that the transport
posts its final response only after the downloader has returned.

The tests use nothing the fix introduces, so the fix can be reverted
under them. Without it, the giving-up test fails against either eve-libs
(the final response has no reader), and the completing test fails
against the previous, leaky eve-libs (its progress reporter posts after
the final response); with the fix both pass against both.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit ee6a552)
Adds end-to-end EVE upgrade testing to the evetest framework.

Framework additions:
- HTTP image server serving rootfs images to EVE devices from the evetest container.
- EdgeDeviceConfig.SetBaseOS to configure an EVE OS upgrade.
- EdgeDevice.UpgradeEVE to pull the target EVE image, push the upgrade
  config, and wait for the upgrade or revert to complete.
- New predefined test parameters: EVEVersionParameterKey, DiskSizeMiBParameterKey.

Upgrade test (tests/upgrade):
- TestEVEUpgrade: deploys a container app, upgrades EVE from an initial version
  (EVETEST_INITIAL_EVE_VERSION) to the tested target (EVETEST_EVE_VERSION), and
  verifies the app is still reachable after the upgrade or revert.
- TestUpgradeSuite: covers KVM→KVM, Kubevirt→Kubevirt, KVM→Kubevirt, and
  small-disk (expected revert) variants.

Also improves list-tests to correctly resolve cross-package parameter key
constants, numeric typed constants, and explicitly empty default values.

[17.0-stable] The upgrade test suite itself (evetest/tests/upgrade) is left
out of this backport: it needs RequireEdgeDevice.MinDiskSizeInMiB, which
this branch's framework does not have. The image server and the rest of
the framework support are carried over unchanged for the tests that need
them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
(cherry picked from commit 9b72db8)
Two small framework pieces that TestDownloadBurstWithDeadMgmtPorts needs
and that this branch's evetest does not have, lifted from master:

- CreateRandomImageFile and AddImageServerFile (from 2bea302, "evetest:
  port eden storage tests, add new datastore coverage") write a file into
  the directory of the built-in HTTP image server; the random content makes
  the SHA256 declared for the content tree meaningful.
- EdgeDeviceConfig.AddContentTree (from 9ab3799, "evetest: framework
  support for the conversion tests") adds a content tree that no volume
  and no application refers to, which volumemgr downloads on its own, so a
  download test exercises the download path in isolation.

Only these functions are carried over; the rest of the two master commits
is unrelated to the test and does not apply to this branch.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
FiveMgmtPortsOneIPv4Uplink has five ethernet ports, each on its own
bridge and network, with a shared dual-stack DNS server, of which only
the last one carries IPv4. eth0 to eth3 are IPv6-only SLAAC islands
(fd26:d0d0:1::/64 to 4::/64) with no outside reachability: the ports
come up with a global address and a working DNS server, so EVE treats
them as usable management ports, but an attempt to reach any IPv4
service from them fails immediately and locally ("no suitable address
found") without a packet leaving the device. eth4 (172.26.4.0/24, DHCP)
reaches the controller, the Internet and evetest's own image server.

The model stands for a device whose management ports are mostly dead
(mis-cabled, unrouted VLANs) with one working uplink listed last. The
address-family mismatch is deliberate: a network without a route or
with a firewall reject makes attempts sit in TCP timeouts as soon as
the SDN throttles its error replies under load, which smears the
synchronized waves of requests the download tests depend on, whereas
the local failure keeps them tight.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit 10f8c76)
scpFromEVE and scpToEVE built the remote target as root@<ip>:<path>. With
an IPv6 device address scp takes everything after the first colon as the
path and tries to resolve "fd26" as the host name, so ReadFile and
WriteFile failed on a device whose first management port is IPv6-only
while RunShellScript, which uses ssh, kept working.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit 1e3eadc)
The downloader hands each transfer request to its zedUpload transport
through a queue that holds as many requests as the transport has
workers, and a request that finds the queue full is refused. The
downloader used to ignore that refusal and wait for a reply that could
never come: the blob stayed DOWNLOAD_STARTED at 0% with no error,
nothing retried it, and the applications depending on it never came up.
A field node hit this on five container layers during a 213-object
deployment over four unrouted management ports, every failed attempt
re-posting at once for the next port.

TestDownloadBurstWithDeadMgmtPorts reproduces the shape of that burst
and requires every download to complete. Four of the five management
ports are IPv6-only while the datastore is IPv4
(netmodels.FiveMgmtPortsOneIPv4Uplink); the downloader tries them in
configuration order and does not skip failing ones, so every download
posts its request five times within milliseconds before the uplink,
listed last, serves it. The content trees are first deployed against a
datastore path that does not exist, so that every download parks in an
error state, and fixing all datastores in one configuration change then
re-drives all of them in the same instant, as a credential refresh did
on the field node. To make the refusal certain rather than occasional,
the test sets downloader.transport.handlers to one and reboots the
device, since the downloader sizes its transport when it starts, and
requires the downloader's start-up line reporting that size before
anything is deployed. The final phase requires at least one refused
request in the downloader's log, so that a run in which the queue never
filled fails instead of passing for the wrong reason.

The waits watch progress rather than the clock: the device relays every
state change through one report queue, so with a hundred content trees
the reports trail it by minutes, and a hang shows as a count that stops
moving while trees are still missing. On failure the test names the
trees left behind, the object states the device itself holds, and the
downloader's view of a few of them. The download retry timer is raised
to thirty minutes so that parked downloads stay quiet until the burst.
The content tree tracker and the device-side diagnostics live in the
networking package helpers for other tests to reuse. The property is
spelled out by its key, since the pillar version evetest builds against
predates the constant.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit fc76218)
@christoph-zededa

Copy link
Copy Markdown
Contributor Author

Folded into #6684, which now carries the evetest commits on top of the downloader backport, as one PR per stable branch.

@christoph-zededa
christoph-zededa deleted the backport-6651-evetest-17.0-stable branch September 30, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants