Skip to content

[13.4-stable] pillar: keep purges from leaving the old app generation behind - #6905

Merged
rene merged 1 commit into
lf-edge:13.4-stablefrom
milan-zededa:13.4-backport-6257-purge-stale-volumerefs
Oct 27, 2026
Merged

rene merged 1 commit into
lf-edge:13.4-stablefrom
milan-zededa:13.4-backport-6257-purge-stale-volumerefs

Conversation

@milan-zededa

Copy link
Copy Markdown
Contributor

Description

Backport of #6257 to 13.4-stable.

Partial backport (the original PR is mostly about another hypervisor flavor), cherry-picked with -x from 9c2eecf8de; the commit message notes what was left out. Only the zedmanager part is backported, together with its unit tests: an old VolumeRefStatus waited forever for a volumemgr update that never comes after a reboot, which left the purge stuck. The backported part applies without conflicts.

Without this fix TestVMAppPurgeAfterPowerCycle fails: a purge issued while the device is rebooting leaves the application stuck.

PR dependencies

None.

How to test and validate this PR

Covered by the evetest test TestVMAppPurgeAfterPowerCycle (added to 13.4-stable by #6899):

EVETEST_EVE_VERSION=<13.4-stable build with this PR> make evetest NAME=TestVMAppPurgeAfterPowerCycle

Changelog notes

Fixed an application purge that got stuck when the device was rebooted in the middle of it.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 32.66%. Comparing base (8077a3d) to head (80deaec).
⚠️ Report is 335 commits behind head on 13.4-stable.

Additional details and impacted files
@@               Coverage Diff               @@
##           13.4-stable    #6905      +/-   ##
===============================================
+ Coverage        24.78%   32.66%   +7.88%     
===============================================
  Files                8        9       +1     
  Lines             1138     1090      -48     
===============================================
+ Hits               282      356      +74     
+ Misses             788      654     -134     
- Partials            68       80      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rene

rene commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

@milan-zededa you also need to include your Sign-off in all commits, DCO is warning about that...

@eriknordmark

Copy link
Copy Markdown
Contributor

@claude is the partial backport OK?

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @eriknordmark's task in 1m 51s —— View job


Is the partial backport OK?

Short answer: yes, the scope of this partial backport is sound. Item 3 is self-contained and independent of the two items that were left out.

What was taken vs. left out

The original PR #6257 bundled three fixes:

  1. Info() / DomainStatus.DomainId identity — kube/eve-k specific
  2. sweepStaleGenerations() on Start() — kube/eve-k specific
  3. zedmanager waiting forever for a volume-ref removal — generic pillar logic

Only item 3 (+ its unit tests) is backported. That is the right call:

  • No dependency on 1 & 2. The change lives entirely in zedmanager's doInstall (updatestatus.go:352-370) and operates on VolumeRefStatus from volumemgr via lookupVolumeRefStatus (handlevolumemgr.go:102). Nothing here touches domainmgr, kubevirt, or DomainStatus.DomainId, so dropping the kube-specific pieces leaves no dangling references.
  • It matches the reported failure. TestVMAppPurgeAfterPowerCycle is a VM app on the non-kube path; item 3 is exactly the fix that path needs. Items 1 & 2 only matter for the kube path, which the commit message correctly identifies as out of scope for 13.4-stable.

Correctness of the backported hunk

The logic is consistent with the pre-existing state machine:

  • Previously, a stale VolumeRefStatus with !PendingAdd was always parked (PendingAdd=true, re-appended, removed=true, early return) to wait for volumemgr's delete event.
  • Now it first checks lookupVolumeRefStatus(...) == nil. If volumemgr has no live status for the key, there is no delete event coming (ephemeral /run state wiped by reboot, or already deleted), so the entry is dropped immediately — it is not re-appended and removed is not set, which matches the existing "completion" path (where an already-PendingAdd entry is silently dropped). If a live status still exists, the old park-and-wait behavior is preserved unchanged.
  • MaybeRemoveVolumeRefConfig still runs in both branches, so the outbound VolumeRefConfig to volumemgr is unpublished regardless — the second unit test pins exactly this.

Both halves of the invariant are covered by the two new tests (drop-when-no-live-status, wait-when-live-status), and Codecov reports the modified lines as fully covered.

Note (process, not code)

The DCO check already flagged by @rene still applies: the commit carries the original author's Signed-off-by: Andrew Durbin from the -x cherry-pick, but @milan-zededa needs to add their own Signed-off-by line for DCO to pass.

Note: I could not run the unit tests locally (go test is blocked by the sandbox in this environment), so the correctness assessment above is from code reading; CI/Codecov coverage on this PR is green.


  • Read the backport commit and diff
  • Review the surrounding doInstall logic for correctness
  • Assess whether leaving out items 1 & 2 is safe for 13.4-stable
  • Post assessment
    • branch 13.4-backport-6257-purge-stale-volumerefs

A purge must delete the previous generation of an app before it creates
the next one. In the kube path three defects stop this. The old VMIRS
then keeps running, and EVE reports the purge as complete.

1. Info() says a domain is gone when it only cannot see it.

   Info() returns domainID 0 if it finds no replica on this node.
   domainmgr copies that 0 into DomainStatus. zedmanager reads 0 as
   "there is no domain", so it skips the teardown and releases the
   purge. Info() now gets identity and existence from the cluster:

   - domainID is workloadID(), a hash of the VMIRS metadata.uid. It was
     rand.Uint32(), which has no relation to the cluster.
   - Task() keeps the DomainStatus. The kube name now comes from the
     status and not from the vmiList cache, which goes stale.
   - Info() returns 0 only when the VMIRS is absent. If it cannot
     confirm absence, it returns the last known id.

2. Nothing deletes an old generation after a reboot.

   /run is tmpfs, so a reboot deletes the purge state. The VMIRS stays
   in etcd. zedmanager finds no domain to halt and advances the counter,
   so the new generation starts beside the old one. Start() now sweeps
   first: it finds each older generation of the app in the cluster,
   deletes it, and waits until the object and its pods are gone. The
   sweep reads the cluster and does not need the old counter, so it
   survives the reboot. If it cannot confirm that a generation is gone,
   Start() fails. A failed start is safer than two generations that
   share MACs, veth names and one RWO disk.

3. zedmanager waits for ever for a volume-ref removal.

   On a purge, doInstall parks the old VolumeRefStatus and waits for
   volumemgr to confirm the delete. If volumemgr has no live status for
   that reference in this boot, the confirmation never comes. The app
   stays in DownloadAndVerify and never asks for the new volume.
   zedmanager now looks at volumemgr's live status first, and drops the
   reference if it is already absent.

Each Kubernetes request that this change adds now has a deadline of
kubeapi.KubeAPITimeout, with the apiCtx helper. domainmgr calls Info on a
timer of 9 to 30 seconds and calls Start directly, and the watchdog of
zedbox restarts the device if that handler stops to report. A request
without a deadline on those paths can therefore stop the node. With a
deadline it fails, and Info reports UNKNOWN and keeps the id of the caller,
which is the contract on types.DomainStatus.DomainId. getVmirs already used
this same limit, so the new calls now match it.

Also in this change:

- StopReplicaVMI had an inverted IsNotFound test, so each successful
  delete logged "Stop VMI Replicaset error <nil>" at error level. This
  made a good teardown look like a bad one in the field. The return
  values do not change.
- The kubevirt and k8s client constructors are now package-level vars.
  The tests can then inject a fake client.
- Info() and sweepStaleGenerations() now call getConfig() themselves.
  The receiver is a value, so a getConfig() call in a method they call
  fills in a copy and leaves kubeConfig nil. A nil kubeConfig made
  newKubevirtClient panic, which stopped zedbox and rebooted the node.
  The client stubs in the tests now reject a nil config.
- docs/zedkube.md records what domainID means, what the sweep does, and
  two gaps that stay open: the sweep does not delete the stale
  generation's PVC, and it cannot run if a purge stops before Start().

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrew Durbin <andrewd@zededa.com>
(cherry picked from commit 9c2eecf)

Partial backport: only item 3 (zedmanager waiting for ever for a
volume-ref removal) and its unit tests are taken. The kubevirt changes,
the docs and the DomainStatus change are specific to eve-k and are left
out.

Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa force-pushed the 13.4-backport-6257-purge-stale-volumerefs branch from 3b8a461 to 80deaec Compare October 26, 2026 08:11
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@milan-zededa you also need to include your Sign-off in all commits, DCO is warning about that...

Done, added my sign-off

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rene
rene merged commit fb12578 into lf-edge:13.4-stable Oct 27, 2026
61 of 72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants