Repository navigation
[17.0-stable] downloader: don't hang on a refused zedUpload request - #6684
Merged
rene merged 13 commits intoOct 1, 2026
Merged
Conversation
Update eve-libs from 24f79b12a2bd to 779d3f2350aa, the tip of its 17.0-stable branch, which adds one commit: the backport of lf-edge/eve-libs#113. DronaRequest.WithContext lets the caller hand the request the context it selects on itself, so that the transport observes the same cancellation; WithCancel and Cancel, which always derived a context of their own, are gone. The downloader fix backported next builds on it. Pillar and the evetest SDN VM module pin eve-libs; both move to the new version, and only the four zedUpload files that carry the change are re-vendored. No transitive dependency moves. Signed-off-by: Christoph Ostarek <christoph@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
download() and objectMetadata() hand each transfer to the zedUpload transport with req.Post() and ignore what it returns. Post() does not block: the request channel behind it holds eleven entries (the default number of transport workers) and a request that finds it full is refused with SyncerRetry. Both callers then sat in `range respChan` waiting for a reply that no worker would ever produce, because the stall watchdog only ran on incoming progress messages and a request nobody picked up sends none. The DownloaderStatus stayed DOWNLOADING at 0 bytes with no error and RetryCount 0, the per-blob handler goroutine was stuck inside handleCreate, so retries, datastore refreshes and RefCount changes for that blob were never processed either, and only a pillar restart recovered. A field node hit this on five container layers when a 213-object deployment burst re-posted every layer immediately for the next of four unrouted management ports, at up to 54 requests per second against the eleven-slot queue. Fix: - postWithRetry submits the request and retries a SyncerRetry refusal with a backoff of 100 ms to 5 s, giving up after maxStalledTime (timer.download.stalled) so the attempt then fails like any other download error and goes through the regular per-address and retry paths instead of hanging. The first refusal of a request is logged. - awaitResponse replaces the bare `range respChan`: the stall check now also runs on a ticker, so a request that never produces a message is cancelled after maxStalledTime as well, and a closed channel is still reported as before. - The request's cancel context is created by the caller, so a user-initiated cancel also wakes up a submission still waiting for room in the queue. Both zedUpload callers in the downloader get the same treatment; the transport itself is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com> (cherry picked from commit 5bf0c8e)
The downloader hands every download request to its zedUpload transport through a queue that holds as many requests as the transport has handlers, eleven, fixed in the library. A request that finds the queue full is refused (SyncerRetry), which the downloader now retries; the number itself has been out of reach of the controller. The report of the field incident asks to consider a larger or configurable depth. Add downloader.transport.handlers (default 11, the library's default; 1 to 1024). The downloader reads it when it creates its transport, so a change takes effect at the next reboot, which the docs say; it also logs the size it started with. A unit test pins the default to the library's, so that a device that never sets the knob keeps the transport it always had. Besides letting a busy node run more handlers, the knob lets a test shrink the queue to one request, which turns the refusal of a request under a burst from an occasional event into a certain one. Signed-off-by: Christoph Ostarek <christoph@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit f86c85e)
The zedUpload transport posts every progress update and the final response of a request with a plain send on the response channel the downloader hands it, from goroutines that live only for that request, and it never learns that the downloader is gone. The downloader stops reading the moment it has the final response, or earlier when it gives up on a request, so the sends that come after that block forever: the progress reporter's tick that raced the final response and its own final update after every completed transfer, and the whole tail of an abandoned one. That is one or two parked goroutines per attempt; on a node retrying hundreds of downloads over dead ports they add up to tens of thousands, and pillar's memory grows until its cgroup limit throttles every service and finally reboots the device, as it did on the field node. The library side of this is fixed in eve-libs PR lf-edge#103. This is the pillar side, which does not depend on it: the downloader owns the channel, so it gives the transport room for what it posts after the final response (a buffer of four; two would do) and keeps a reader on the channel of a request it has given up on until the transport's final response arrives, which the cancelled request brings about soon. The dial context watchers that nettrace leaked alongside could not be stopped from pillar: net/http dials with a context detached from the request's and drops its cancel function once the dial is done, so only the library could end them, which eve-libs PR lf-edge#103 does. The eve-libs bump later in this series brings that fix; the pillar-side handling here stays as defence in depth. Signed-off-by: Christoph Ostarek <christoph@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 03d028e)
Two tests run the downloader's own download function against a local HTTP server, over the real zedUpload transport, and then require that no transport goroutine is left trying to post on the response channel, counted from the goroutine profile. The server pauses halfway through the blob, so that the transport's progress reporter posts at least one update before the final response. One test lets the download complete; the other has the server stall for good and the downloader give up for lack of progress, with the stall timer shortened, so that the transport posts its final response only after the downloader has returned. The tests use nothing the fix introduces, so the fix can be reverted under them. Without it, the giving-up test fails against either eve-libs (the final response has no reader), and the completing test fails against the previous, leaky eve-libs (its progress reporter posts after the final response); with the fix both pass against both. Signed-off-by: Christoph Ostarek <christoph@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit ee6a552)
This was referenced Sep 30, 2026
Adds end-to-end EVE upgrade testing to the evetest framework. Framework additions: - HTTP image server serving rootfs images to EVE devices from the evetest container. - EdgeDeviceConfig.SetBaseOS to configure an EVE OS upgrade. - EdgeDevice.UpgradeEVE to pull the target EVE image, push the upgrade config, and wait for the upgrade or revert to complete. - New predefined test parameters: EVEVersionParameterKey, DiskSizeMiBParameterKey. Upgrade test (tests/upgrade): - TestEVEUpgrade: deploys a container app, upgrades EVE from an initial version (EVETEST_INITIAL_EVE_VERSION) to the tested target (EVETEST_EVE_VERSION), and verifies the app is still reachable after the upgrade or revert. - TestUpgradeSuite: covers KVM→KVM, Kubevirt→Kubevirt, KVM→Kubevirt, and small-disk (expected revert) variants. Also improves list-tests to correctly resolve cross-package parameter key constants, numeric typed constants, and explicitly empty default values. [17.0-stable] The upgrade test suite itself (evetest/tests/upgrade) is left out of this backport: it needs RequireEdgeDevice.MinDiskSizeInMiB, which this branch's framework does not have. The image server and the rest of the framework support are carried over unchanged for the tests that need them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Milan Lenco <milan@zededa.com> (cherry picked from commit 9b72db8)
Two small framework pieces that TestDownloadBurstWithDeadMgmtPorts needs and that this branch's evetest does not have, lifted from master: - CreateRandomImageFile and AddImageServerFile (from 2bea302, "evetest: port eden storage tests, add new datastore coverage") write a file into the directory of the built-in HTTP image server; the random content makes the SHA256 declared for the content tree meaningful. - EdgeDeviceConfig.AddContentTree (from 9ab3799, "evetest: framework support for the conversion tests") adds a content tree that no volume and no application refers to, which volumemgr downloads on its own, so a download test exercises the download path in isolation. Only these functions are carried over; the rest of the two master commits is unrelated to the test and does not apply to this branch. Signed-off-by: Christoph Ostarek <christoph@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
FiveMgmtPortsOneIPv4Uplink has five ethernet ports, each on its own
bridge and network, with a shared dual-stack DNS server, of which only
the last one carries IPv4. eth0 to eth3 are IPv6-only SLAAC islands
(fd26:d0d0:1::/64 to 4::/64) with no outside reachability: the ports
come up with a global address and a working DNS server, so EVE treats
them as usable management ports, but an attempt to reach any IPv4
service from them fails immediately and locally ("no suitable address
found") without a packet leaving the device. eth4 (172.26.4.0/24, DHCP)
reaches the controller, the Internet and evetest's own image server.
The model stands for a device whose management ports are mostly dead
(mis-cabled, unrouted VLANs) with one working uplink listed last. The
address-family mismatch is deliberate: a network without a route or
with a firewall reject makes attempts sit in TCP timeouts as soon as
the SDN throttles its error replies under load, which smears the
synchronized waves of requests the download tests depend on, whereas
the local failure keeps them tight.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit 10f8c76)
scpFromEVE and scpToEVE built the remote target as root@<ip>:<path>. With an IPv6 device address scp takes everything after the first colon as the path and tries to resolve "fd26" as the host name, so ReadFile and WriteFile failed on a device whose first management port is IPv6-only while RunShellScript, which uses ssh, kept working. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com> (cherry picked from commit 1e3eadc)
The downloader hands each transfer request to its zedUpload transport through a queue that holds as many requests as the transport has workers, and a request that finds the queue full is refused. The downloader used to ignore that refusal and wait for a reply that could never come: the blob stayed DOWNLOAD_STARTED at 0% with no error, nothing retried it, and the applications depending on it never came up. A field node hit this on five container layers during a 213-object deployment over four unrouted management ports, every failed attempt re-posting at once for the next port. TestDownloadBurstWithDeadMgmtPorts reproduces the shape of that burst and requires every download to complete. Four of the five management ports are IPv6-only while the datastore is IPv4 (netmodels.FiveMgmtPortsOneIPv4Uplink); the downloader tries them in configuration order and does not skip failing ones, so every download posts its request five times within milliseconds before the uplink, listed last, serves it. The content trees are first deployed against a datastore path that does not exist, so that every download parks in an error state, and fixing all datastores in one configuration change then re-drives all of them in the same instant, as a credential refresh did on the field node. To make the refusal certain rather than occasional, the test sets downloader.transport.handlers to one and reboots the device, since the downloader sizes its transport when it starts, and requires the downloader's start-up line reporting that size before anything is deployed. The final phase requires at least one refused request in the downloader's log, so that a run in which the queue never filled fails instead of passing for the wrong reason. The waits watch progress rather than the clock: the device relays every state change through one report queue, so with a hundred content trees the reports trail it by minutes, and a hang shows as a count that stops moving while trees are still missing. On failure the test names the trees left behind, the object states the device itself holds, and the downloader's view of a few of them. The download retry timer is raised to thirty minutes so that parked downloads stay quiet until the burst. The content tree tracker and the device-side diagnostics live in the networking package helpers for other tests to reuse. The property is spelled out by its key, since the pillar version evetest builds against predates the constant. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com> (cherry picked from commit fc76218)
6 of 9 tasks
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 17.0-stable #6684 +/- ##
===============================================
+ Coverage 22.23% 22.72% +0.48%
===============================================
Files 508 518 +10
Lines 94275 96036 +1761
===============================================
+ Hits 20963 21824 +861
- Misses 71523 72227 +704
- Partials 1789 1985 +196 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
On this branch the evetest framework image is selected by VERSION and only built when no image of that version exists locally or on the registry, so a framework change without a version bump leaves the tests compiling against the old framework inside the old image: with the two preceding framework commits in place, TestDownloadBurstWithDeadMgmtPorts still failed to build for want of CreateRandomImageFile, AddContentTree and the image server accessors. Move to v0.0.2 so that `make evetest` builds the image from the current sources. Signed-off-by: Christoph Ostarek <christoph@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
original fix is in 3479e22 Signed-off-by: Christoph Ostarek <christoph@zededa.com>
continuation of 78099c1 Signed-off-by: Christoph Ostarek <christoph@zededa.com>
christoph-zededa
marked this pull request as ready for review
October 1, 2026 09:15
rene
approved these changes
Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6651 to
17.0-stable, in full, plus what its evetest needsfrom #6083, #6244 and #6539: the four downloader commits, cherry-picked
with
-xand preceded by an eve-libs bump to the tip of its17.0-stablebranch, followed by the evetest that reproduces the burst, together with
the two framework pieces it needs on this branch.
779d3f2350aa, the one commit eve-libs'
17.0-stablegained since the previous bump:the backport of zedUpload: let the caller inject the request's context eve-libs#113. The downloader fix below hands the
request the context it selects on, which needs that API; the goroutine
leak fixes of Fix goroutine leaks in zedUpload and nettrace eve-libs#103 were already picked up by the previous
bump on this branch.
itself: a request refused by the full transport queue is now retried with a
backoff and given up after
timer.download.stalled, and the stall checkalso runs on a ticker.
downloader.transport.handlers, default 11, takes effect at the nextreboot.
side of the zedUpload goroutine leak.
above against the real transport and a local HTTP server.
All four cherry-picks applied cleanly; the bump moves both modules that pin eve-libs here, pillar and the evetest SDN VM module.
The evetest part, commits six to ten:
-xfrommaster's 9b72db8 (evetest: add EVE upgrade testing support #6083), which is where the built-in HTTP image server came
from: a dummy interface at 244.244.244.244, a file server, and the route to
it through the SDN. It applies cleanly except for the upgrade test suite
itself, which needs
RequireEdgeDevice.MinDiskSizeInMiBthat this branchdoes not have, so
evetest/tests/upgradeis left out; the commit messagesays so.
commit lifting
CreateRandomImageFile/AddImageServerFile(from master's2bea302, Eden to evetest: rewrite networking, security, apps, lps and storage tests #6244) and
EdgeDeviceConfig.AddContentTree(from 9ab3799,evetest: EVE-kvm to EVE-K conversion test suite #6539), 64
lines in total; the two master commits themselves are unrelated to this
test and do not apply to this branch.
bracket IPv6 addresses in scp targets — clean cherry-picks.
adaptations to this branch's framework:
evetest.KiBis stillevetest.KBhere; the networking package's
helpers_test.godoes not exist here, so itholds only the helpers this test adds; and
EdgeDevice.ReadFilereturns noerror here, so the failure diagnostics read the downloader's status files
without the error branch.
How to test and validate this PR
make pkgs eveon this branch, thenmake evetest NAME=TestDownloadBurstWithDeadMgmtPorts. On master the runtakes about eight minutes including setup and ended with 100 of 100 content
trees loaded and 295 refused requests retried (see downloader: don't hang on a refused zedUpload request #6651). The framework
and the networking test package compile on this branch; the test has not
been run against a 17.0 image yet, which is what this draft is waiting for.
That run has to confirm the image server route on this branch's SDN and
that 17.0's zedagent reports content-tree errors the way the parking phase
waits for.
make -C pkg/pillar test, or nativelyCGO_ENABLED=0 go test ./cmd/downloader/ ./types/; they pass on this branch.the datastore ahead of the working one, and deploy many content trees at
once. Before this PR some blobs stay at
DOWNLOAD_STARTED0% until pillarrestarts; after it they all complete, the downloader logs
Transport request queue is full, retrying submission of ...for each refusal, logszedUpload transport handlers: Nat start, and zedbox's goroutine count(
NumGoRoutinesin the zedbox metric) no longer grows with failed attempts.See downloader: don't hang on a refused zedUpload request #6651 for the end-to-end results on master.
PR Backports
Checklist
docs/CONFIG-PROPERTIES.mdrowsstableto be setFor backport PRs:
And the last but not least:
check them.
🤖 Generated with Claude Code