Skip to content

[13.4-stable] downloader: don't hang on a refused zedUpload request - #6687

Merged
rene merged 5 commits into
lf-edge:13.4-stablefrom
christoph-zededa:backport-6651-13.4-stable
Oct 1, 2026
Merged

rene merged 5 commits into
lf-edge:13.4-stablefrom
christoph-zededa:backport-6651-13.4-stable

Conversation

@christoph-zededa

@christoph-zededa christoph-zededa commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

Backport of #6651 to 13.4-stable: the four downloader commits, cherry-picked with
-x, preceded by an eve-libs bump to the tip of its 13.4-stable branch.

  • Bump eve-libs to pick up DronaRequest.WithContext — 8e6dedf9138f to
    06fd0b63af43, the one commit eve-libs' 13.4-stable gained since the previous bump:
    the backport of zedUpload: let the caller inject the request's context eve-libs#113. The downloader fix below hands the
    request the context it selects on, which needs that API; the goroutine
    leak fixes of Fix goroutine leaks in zedUpload and nettrace eve-libs#103 were already picked up by the previous
    bump on this branch.
  • downloader: don't hang on a refused zedUpload request — the hang
    itself: a request refused by the full transport queue is now retried with a
    backoff and given up after timer.download.stalled, and the stall check
    also runs on a ticker.
  • downloader: make the transport's handler count configurable —
    downloader.transport.handlers, default 11, takes effect at the next
    reboot.
  • downloader: drain transport responses nobody waits for — the pillar
    side of the zedUpload goroutine leak.
  • downloader: test that the transport's senders can finish — tests for the
    above against the real transport and a local HTTP server.

Adaptations to this branch's older downloader, the same four as on 14.5-stable:

  • the fix's new constants are inserted on their own, since the progress-file helpers that surround them on master do not exist here;
  • docs/CONFIG-PROPERTIES.md still has the four-column table, so the reworded timer.download.stalled row and the new downloader.transport.handlers row are written in that format;
  • the key list in types/global_test.go gets only the new key, without the kube keys that arrived with it on master;
  • download() has no netdump path parameter here, so the drain test's call drops that argument.

The evetest network model and test of #6651 are not backported: there is no evetest on this branch.

How to test and validate this PR

  • Unit tests: make -C pkg/pillar test, or natively
    CGO_ENABLED=0 go test ./cmd/downloader/ ./types/; they pass on this branch.
  • On a device: give it a management port that has an address but no route to
    the datastore ahead of the working one, and deploy many content trees at
    once. Before this PR some blobs stay at DOWNLOAD_STARTED 0% until pillar
    restarts; after it they all complete, the downloader logs Transport request queue is full, retrying submission of ... for each refusal, logs
    zedUpload transport handlers: N at start, and zedbox's goroutine count
    (NumGoRoutines in the zedbox metric) no longer grows with failed attempts.
    See downloader: don't hang on a refused zedUpload request #6651 for the end-to-end results on master.

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation — docs/CONFIG-PROPERTIES.md rows in this branch's four-column format
  • I've tested my PR on amd64 device — unit tests on this branch; the end-to-end evidence is downloader: don't hang on a refused zedUpload request #6651's, on master
  • I've tested my PR on arm64 device — not tested
  • I've written the test verification instructions
  • I've set the proper labels to this PR — stable to be set

For backport PRs:

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

christoph-zededa and others added 5 commits October 1, 2026 00:16
Update eve-libs from 8e6dedf9138f to 06fd0b63af43, the tip of its
13.4-stable branch, which adds one commit: the backport of
lf-edge/eve-libs#113. DronaRequest.WithContext lets the caller hand the
request the context it selects on itself, so that the transport observes
the same cancellation; WithCancel and Cancel, which always derived a
context of their own, are gone. The downloader fix backported next
builds on it.

Pillar is the only module on this branch that pins eve-libs; it moves to
the new version, and only the four zedUpload files that carry the change
are re-vendored. No transitive dependency moves.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
download() and objectMetadata() hand each transfer to the zedUpload
transport with req.Post() and ignore what it returns. Post() does not
block: the request channel behind it holds eleven entries (the default
number of transport workers) and a request that finds it full is
refused with SyncerRetry. Both callers then sat in `range respChan`
waiting for a reply that no worker would ever produce, because the
stall watchdog only ran on incoming progress messages and a request
nobody picked up sends none. The DownloaderStatus stayed DOWNLOADING at
0 bytes with no error and RetryCount 0, the per-blob handler goroutine
was stuck inside handleCreate, so retries, datastore refreshes and
RefCount changes for that blob were never processed either, and only a
pillar restart recovered. A field node hit this on five container
layers when a 213-object deployment burst re-posted every layer
immediately for the next of four unrouted management ports, at up to 54
requests per second against the eleven-slot queue.

Fix:

- postWithRetry submits the request and retries a SyncerRetry refusal
  with a backoff of 100 ms to 5 s, giving up after maxStalledTime
  (timer.download.stalled) so the attempt then fails like any other
  download error and goes through the regular per-address and retry
  paths instead of hanging. The first refusal of a request is logged.
- awaitResponse replaces the bare `range respChan`: the stall check now
  also runs on a ticker, so a request that never produces a message is
  cancelled after maxStalledTime as well, and a closed channel is still
  reported as before.
- The request's cancel context is created by the caller, so a
  user-initiated cancel also wakes up a submission still waiting for
  room in the queue.

Both zedUpload callers in the downloader get the same treatment; the
transport itself is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit 5bf0c8e)
The downloader hands every download request to its zedUpload transport
through a queue that holds as many requests as the transport has
handlers, eleven, fixed in the library. A request that finds the queue
full is refused (SyncerRetry), which the downloader now retries; the
number itself has been out of reach of the controller. The report of the
field incident asks to consider a larger or configurable depth.

Add downloader.transport.handlers (default 11, the library's default;
1 to 1024). The downloader reads it when it creates its transport, so a
change takes effect at the next reboot, which the docs say; it also
logs the size it started with. A unit test pins the default to the
library's, so that a device that never sets the knob keeps the
transport it always had.

Besides letting a busy node run more handlers, the knob lets a test
shrink the queue to one request, which turns the refusal of a request
under a burst from an occasional event into a certain one.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit f86c85e)
The zedUpload transport posts every progress update and the final
response of a request with a plain send on the response channel the
downloader hands it, from goroutines that live only for that request,
and it never learns that the downloader is gone. The downloader stops
reading the moment it has the final response, or earlier when it gives
up on a request, so the sends that come after that block forever: the
progress reporter's tick that raced the final response and its own
final update after every completed transfer, and the whole tail of an
abandoned one. That is one or two parked goroutines per attempt; on a
node retrying hundreds of downloads over dead ports they add up to
tens of thousands, and pillar's memory grows until its cgroup limit
throttles every service and finally reboots the device, as it did on
the field node.

The library side of this is fixed in eve-libs PR lf-edge#103. This is the
pillar side, which does not depend on it: the downloader owns the
channel, so it gives the transport room for what it posts after the
final response (a buffer of four; two would do) and keeps a reader on
the channel of a request it has given up on until the transport's
final response arrives, which the cancelled request brings about soon.

The dial context watchers that nettrace leaked alongside could not be
stopped from pillar: net/http dials with a context detached from the
request's and drops its cancel function once the dial is done, so only
the library could end them, which eve-libs PR lf-edge#103 does. The eve-libs
bump later in this series brings that fix; the pillar-side handling
here stays as defence in depth.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 03d028e)
Two tests run the downloader's own download function against a local
HTTP server, over the real zedUpload transport, and then require that
no transport goroutine is left trying to post on the response channel,
counted from the goroutine profile. The server pauses halfway through
the blob, so that the transport's progress reporter posts at least one
update before the final response. One test lets the download complete;
the other has the server stall for good and the downloader give up for
lack of progress, with the stall timer shortened, so that the transport
posts its final response only after the downloader has returned.

The tests use nothing the fix introduces, so the fix can be reverted
under them. Without it, the giving-up test fails against either eve-libs
(the final response has no reader), and the completing test fails
against the previous, leaky eve-libs (its progress reporter posts after
the final response); with the fix both pass against both.

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit ee6a552)
@christoph-zededa
christoph-zededa marked this pull request as draft October 1, 2026 10:19
@christoph-zededa

Copy link
Copy Markdown
Contributor Author

marked as draft as lf-edge/eve-libs#119 is needed

@rene
rene marked this pull request as ready for review October 1, 2026 12:43
@rene
rene merged commit 60e462c into lf-edge:13.4-stable Oct 1, 2026
42 of 43 checks passed
rene pushed a commit that referenced this pull request Oct 5, 2026
Update eve-libs from 06fd0b63af43 to eddbd838d583, the tip of its
13.4-stable branch, which adds the three commits of
lf-edge/eve-libs#119: the backport of lf-edge/eve-libs#51, followed by
the two nettrace test-skip commits of lf-edge/eve-libs#84 and
lf-edge/eve-libs#82.

The race fix makes GetDoneParts take the request's read lock, which the
stats updater already holds for writing when it records the parts, and
has the AWS and Azure transports update the request's size and status
under the lock as well. Pillar runs its unit tests under the race
detector, and TestTransportSendersFinishAfterDownload, which #6687
brought to this branch, drives a request through a real transport whose
stats updater keeps writing those fields while the downloader reads them
from the progress messages; the CI run of that PR reported the race.

The two nettrace commits only change its tests, which are not vendored.
Pillar is the only module on this branch that pins eve-libs; it moves to
the new version, and only the four zedUpload files that the backport
touches are re-vendored. No transitive dependency moves.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants