Repository navigation
[13.4-stable] pillar: keep purges from leaving the old app generation behind - #6905
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 13.4-stable #6905 +/- ##
===============================================
+ Coverage 24.78% 32.66% +7.88%
===============================================
Files 8 9 +1
Lines 1138 1090 -48
===============================================
+ Hits 282 356 +74
+ Misses 788 654 -134
- Partials 68 80 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@milan-zededa you also need to include your Sign-off in all commits, DCO is warning about that... |
|
@claude is the partial backport OK? |
|
Claude finished @eriknordmark's task in 1m 51s —— View job Is the partial backport OK?Short answer: yes, the scope of this partial backport is sound. Item 3 is self-contained and independent of the two items that were left out. What was taken vs. left outThe original PR #6257 bundled three fixes:
Only item 3 (+ its unit tests) is backported. That is the right call:
Correctness of the backported hunkThe logic is consistent with the pre-existing state machine:
Both halves of the invariant are covered by the two new tests (drop-when-no-live-status, wait-when-live-status), and Codecov reports the modified lines as fully covered. Note (process, not code)The DCO check already flagged by @rene still applies: the commit carries the original author's
|
A purge must delete the previous generation of an app before it creates
the next one. In the kube path three defects stop this. The old VMIRS
then keeps running, and EVE reports the purge as complete.
1. Info() says a domain is gone when it only cannot see it.
Info() returns domainID 0 if it finds no replica on this node.
domainmgr copies that 0 into DomainStatus. zedmanager reads 0 as
"there is no domain", so it skips the teardown and releases the
purge. Info() now gets identity and existence from the cluster:
- domainID is workloadID(), a hash of the VMIRS metadata.uid. It was
rand.Uint32(), which has no relation to the cluster.
- Task() keeps the DomainStatus. The kube name now comes from the
status and not from the vmiList cache, which goes stale.
- Info() returns 0 only when the VMIRS is absent. If it cannot
confirm absence, it returns the last known id.
2. Nothing deletes an old generation after a reboot.
/run is tmpfs, so a reboot deletes the purge state. The VMIRS stays
in etcd. zedmanager finds no domain to halt and advances the counter,
so the new generation starts beside the old one. Start() now sweeps
first: it finds each older generation of the app in the cluster,
deletes it, and waits until the object and its pods are gone. The
sweep reads the cluster and does not need the old counter, so it
survives the reboot. If it cannot confirm that a generation is gone,
Start() fails. A failed start is safer than two generations that
share MACs, veth names and one RWO disk.
3. zedmanager waits for ever for a volume-ref removal.
On a purge, doInstall parks the old VolumeRefStatus and waits for
volumemgr to confirm the delete. If volumemgr has no live status for
that reference in this boot, the confirmation never comes. The app
stays in DownloadAndVerify and never asks for the new volume.
zedmanager now looks at volumemgr's live status first, and drops the
reference if it is already absent.
Each Kubernetes request that this change adds now has a deadline of
kubeapi.KubeAPITimeout, with the apiCtx helper. domainmgr calls Info on a
timer of 9 to 30 seconds and calls Start directly, and the watchdog of
zedbox restarts the device if that handler stops to report. A request
without a deadline on those paths can therefore stop the node. With a
deadline it fails, and Info reports UNKNOWN and keeps the id of the caller,
which is the contract on types.DomainStatus.DomainId. getVmirs already used
this same limit, so the new calls now match it.
Also in this change:
- StopReplicaVMI had an inverted IsNotFound test, so each successful
delete logged "Stop VMI Replicaset error <nil>" at error level. This
made a good teardown look like a bad one in the field. The return
values do not change.
- The kubevirt and k8s client constructors are now package-level vars.
The tests can then inject a fake client.
- Info() and sweepStaleGenerations() now call getConfig() themselves.
The receiver is a value, so a getConfig() call in a method they call
fills in a copy and leaves kubeConfig nil. A nil kubeConfig made
newKubevirtClient panic, which stopped zedbox and rebooted the node.
The client stubs in the tests now reject a nil config.
- docs/zedkube.md records what domainID means, what the sweep does, and
two gaps that stay open: the sweep does not delete the stale
generation's PVC, and it cannot run if a purge stops before Start().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrew Durbin <andrewd@zededa.com>
(cherry picked from commit 9c2eecf)
Partial backport: only item 3 (zedmanager waiting for ever for a
volume-ref removal) and its unit tests are taken. The kubevirt changes,
the docs and the DomainStatus change are specific to eve-k and are left
out.
Signed-off-by: Milan Lenco <milan@zededa.com>
3b8a461 to
80deaec
Compare
Done, added my sign-off |
Description
Backport of #6257 to
13.4-stable.Partial backport (the original PR is mostly about another hypervisor flavor), cherry-picked with
-xfrom9c2eecf8de; the commit message notes what was left out. Only the zedmanager part is backported, together with its unit tests: an oldVolumeRefStatuswaited forever for a volumemgr update that never comes after a reboot, which left the purge stuck. The backported part applies without conflicts.Without this fix
TestVMAppPurgeAfterPowerCyclefails: a purge issued while the device is rebooting leaves the application stuck.PR dependencies
None.
How to test and validate this PR
Covered by the evetest test
TestVMAppPurgeAfterPowerCycle(added to13.4-stableby #6899):Changelog notes
Fixed an application purge that got stuck when the device was rebooted in the middle of it.
Checklist
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.