Skip to content

[16.0-stable] pillar: retry deferred info messages the controller may still accept - #6849

Merged
rene merged 4 commits into
lf-edge:16.0-stablefrom
milan-zededa:16.0-backport-6304-deferred-info-retry
Oct 25, 2026
Merged

rene merged 4 commits into
lf-edge:16.0-stablefrom
milan-zededa:16.0-backport-6304-deferred-info-retry

Conversation

@milan-zededa

Copy link
Copy Markdown
Contributor

Description

Backport of #6304 to 16.0-stable.

The four pillar commits of the PR are cherry-picked with -x. The three
evetest commits (fault injection into controller answers) are part of the
evetest backport (#6842).

Two of the commits conflicted only in pkg/pillar/docs/zedagent.md: that file
in 16.0 does not have the "Deferred Queue Mechanism" section the commits
extend, so the doc changes are skipped (16.0's version is kept). The code
applied cleanly.

Without these commits TestInfoRetriedAfterServerError fails: an info message
that the controller answers with a server error is dropped instead of being
retried.

PR dependencies

None.

How to test and validate this PR

Covered by the evetest test TestInfoRetriedAfterServerError (added to 16.0-stable by #6842):

EVETEST_EVE_VERSION=<16.0-stable build with this PR> make evetest NAME=TestInfoRetriedAfterServerError

Changelog notes

Info messages (application, volume, network instance state and others) that the controller answers with a server error or rate limit are now retried with a growing delay instead of being dropped, so the controller view no longer stays stale.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

eriknordmark and others added 4 commits October 18, 2026 19:39
SendOnAllIntf hands back the HTTP response only when a send succeeded, or
when the caller asked to stop at the first port that answered with an error.
When it walks every port and none of them delivers, the status the server
did answer with is lost, leaving the caller unable to tell a rejection by
the controller from a link that never got there. Carry the last status code
seen in the return value so that a caller can base a retry decision on it.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
(cherry picked from commit 17fb3a3)
Signed-off-by: Milan Lenco <milan@zededa.com>
A device reports application, volume, content tree, blob, network instance,
hardware and cluster state to the controller only when that state changes;
nothing re-asserts it later. Yet any status between 400 and 599 made the
deferred queue log one line and then treat the message as if it had been
delivered, so a few seconds of controller-side trouble left the controller's
view of an object wrong until the object next changed - possibly never. The
discard was introduced for a controller that no longer knows about an object
and therefore rejects the request itself; server errors were swept into it
when the same flag started meaning "do not try the other ports".

Decide from the status code instead. A rejection of the request is still
given up on, now logged as an error since the state it carried is lost for
good. Anything the controller may yet accept - a server error, a rate limit,
or a refusal to authorize the device, which EVE answers by restarting
attestation - keeps the message queued and offers it again with a growing
delay. There is no attempt limit: a message this queue carries has no other
path to the controller, so the only alternative to retrying is losing it.

Since a status code means the controller was reached, the failure belongs to
that one message and the rest of the pass now continues, where before a
single refused item aborted the pass for every remaining item and priority
class and was then retried ahead of them all. A message waiting out its
delay moves behind its peers, so one object cannot hold up the reports for
all the others. Failures are reported to the sent-callback with a real
status, so a message the controller refused is no longer recorded as sent.

Retrying without end must not be silent, so a backlog of undelivered
messages and an individual message the controller keeps refusing are both
raised as warnings. Reporting an object again with unchanged content keeps
the retry state of the queued message, so that a device republishing the
same bytes neither restarts the delay nor hides how long the message has
been stuck.

That leaves the two flags with one meaning each: BailOnHTTPErr says how many
ports to try, exactly as in RequestOptions, and DiscardOnFailure - which used
to be called IgnoreErr, having never ignored an error - says that a payload
is superseded by the next periodic one and so is not worth keeping.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit dcf43d6)
Signed-off-by: Milan Lenco <milan@zededa.com>
The first edgeview status, the one that tells the controller edgeview is
active, was routed to the best-effort periodic queue in the hope of getting
it delivered - which had the opposite effect, since that queue discards a
message whose send fails instead of retrying it. It now takes the same path
as every other info message, where a controller-side failure is retried.

That leaves location info as the only user of the periodic queue, so also
correct its description: metrics and hardware health never went through a
deferred queue, and NTP sources deliberately use the reliable one.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
(cherry picked from commit 96019a0)
Signed-off-by: Milan Lenco <milan@zededa.com>
While walking the management ports, any non-zero sender status picked up on
the way was kept in the return value, including when a later port went on to
deliver the request. The caller then saw a successful send described by the
complaint of a port that failed - the deferred queue treated such a message
as not sent and kept it queued for another try, so a report that did reach
the controller could be sent a second time, and a config fetch that
succeeded on the second port still logged a controller upgrade.

Report the status of the attempt that produced the response. Within a port
this restores an invariant that already held one branch away: on any
non-2xx response the status collected from the source addresses tried
earlier is already discarded, and only the branch for a delivered response
kept it. Nothing is lost for diagnosis, since every certificate and
connection failure is logged where it happens. When no port delivers the
request the collected status still stands, which is what tells a caller
that the controller rejected the request rather than being unreachable.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
(cherry picked from commit 1ca0505)
Signed-off-by: Milan Lenco <milan@zededa.com>
@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 26.89%. Comparing base (c9e7c2a) to head (5850029).
⚠️ Report is 22 commits behind head on 16.0-stable.

Additional details and impacted files
@@               Coverage Diff               @@
##           16.0-stable    #6849      +/-   ##
===============================================
- Coverage        29.12%   26.89%   -2.24%     
===============================================
  Files               10       25      +15     
  Lines             1133     4332    +3199     
===============================================
+ Hits               330     1165     +835     
- Misses             768     2938    +2170     
- Partials            35      229     +194     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene
rene merged commit f9e5b3b into lf-edge:16.0-stable Oct 25, 2026
53 of 54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants