Repository navigation
[16.0-stable] pillar: retry deferred info messages the controller may still accept - #6849
Merged
rene merged 4 commits intoOct 25, 2026
Conversation
SendOnAllIntf hands back the HTTP response only when a send succeeded, or when the caller asked to stop at the first port that answered with an error. When it walks every port and none of them delivers, the status the server did answer with is lost, leaving the caller unable to tell a rejection by the controller from a link that never got there. Carry the last status code seen in the return value so that a caller can base a retry decision on it. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit 17fb3a3) Signed-off-by: Milan Lenco <milan@zededa.com>
A device reports application, volume, content tree, blob, network instance, hardware and cluster state to the controller only when that state changes; nothing re-asserts it later. Yet any status between 400 and 599 made the deferred queue log one line and then treat the message as if it had been delivered, so a few seconds of controller-side trouble left the controller's view of an object wrong until the object next changed - possibly never. The discard was introduced for a controller that no longer knows about an object and therefore rejects the request itself; server errors were swept into it when the same flag started meaning "do not try the other ports". Decide from the status code instead. A rejection of the request is still given up on, now logged as an error since the state it carried is lost for good. Anything the controller may yet accept - a server error, a rate limit, or a refusal to authorize the device, which EVE answers by restarting attestation - keeps the message queued and offers it again with a growing delay. There is no attempt limit: a message this queue carries has no other path to the controller, so the only alternative to retrying is losing it. Since a status code means the controller was reached, the failure belongs to that one message and the rest of the pass now continues, where before a single refused item aborted the pass for every remaining item and priority class and was then retried ahead of them all. A message waiting out its delay moves behind its peers, so one object cannot hold up the reports for all the others. Failures are reported to the sent-callback with a real status, so a message the controller refused is no longer recorded as sent. Retrying without end must not be silent, so a backlog of undelivered messages and an individual message the controller keeps refusing are both raised as warnings. Reporting an object again with unchanged content keeps the retry state of the queued message, so that a device republishing the same bytes neither restarts the delay nor hides how long the message has been stuck. That leaves the two flags with one meaning each: BailOnHTTPErr says how many ports to try, exactly as in RequestOptions, and DiscardOnFailure - which used to be called IgnoreErr, having never ignored an error - says that a payload is superseded by the next periodic one and so is not worth keeping. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit dcf43d6) Signed-off-by: Milan Lenco <milan@zededa.com>
The first edgeview status, the one that tells the controller edgeview is active, was routed to the best-effort periodic queue in the hope of getting it delivered - which had the opposite effect, since that queue discards a message whose send fails instead of retrying it. It now takes the same path as every other info message, where a controller-side failure is retried. That leaves location info as the only user of the periodic queue, so also correct its description: metrics and hardware health never went through a deferred queue, and NTP sources deliberately use the reliable one. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit 96019a0) Signed-off-by: Milan Lenco <milan@zededa.com>
While walking the management ports, any non-zero sender status picked up on the way was kept in the return value, including when a later port went on to deliver the request. The caller then saw a successful send described by the complaint of a port that failed - the deferred queue treated such a message as not sent and kept it queued for another try, so a report that did reach the controller could be sent a second time, and a config fetch that succeeded on the second port still logged a controller upgrade. Report the status of the attempt that produced the response. Within a port this restores an invariant that already held one branch away: on any non-2xx response the status collected from the source addresses tried earlier is already discarded, and only the branch for a delivered response kept it. Nothing is lost for diagnosis, since every certificate and connection failure is logged where it happens. When no port delivers the request the collected status still stands, which is what tells a caller that the controller rejected the request rather than being unreachable. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com> (cherry picked from commit 1ca0505) Signed-off-by: Milan Lenco <milan@zededa.com>
milan-zededa
requested review from
christoph-zededa and
rouming
as code owners
October 18, 2026 17:42
github-actions
Bot
requested review from
OhmSpectator,
eriknordmark and
uncleDecart
October 18, 2026 17:42
4 of 7 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 16.0-stable #6849 +/- ##
===============================================
- Coverage 29.12% 26.89% -2.24%
===============================================
Files 10 25 +15
Lines 1133 4332 +3199
===============================================
+ Hits 330 1165 +835
- Misses 768 2938 +2170
- Partials 35 229 +194 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
rene
approved these changes
Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6304 to
16.0-stable.The four pillar commits of the PR are cherry-picked with
-x. The threeevetest commits (fault injection into controller answers) are part of the
evetest backport (#6842).
Two of the commits conflicted only in
pkg/pillar/docs/zedagent.md: that filein 16.0 does not have the "Deferred Queue Mechanism" section the commits
extend, so the doc changes are skipped (16.0's version is kept). The code
applied cleanly.
Without these commits
TestInfoRetriedAfterServerErrorfails: an info messagethat the controller answers with a server error is dropped instead of being
retried.
PR dependencies
None.
How to test and validate this PR
Covered by the evetest test
TestInfoRetriedAfterServerError(added to16.0-stableby #6842):Changelog notes
Info messages (application, volume, network instance state and others) that the controller answers with a server error or rate limit are now retried with a growing delay instead of being dropped, so the controller view no longer stays stale.
Checklist
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.