Skip to content

vault: clean up after a failed kvm→EVE-k vault migration - #6530

Closed
eriknordmark wants to merge 10 commits into
lf-edge:masterfrom
eriknordmark:vault-zvol-migration-cleanup
Closed

eriknordmark wants to merge 10 commits into
lf-edge:masterfrom
eriknordmark:vault-zvol-migration-cleanup

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #6036, closing the gaps in what the ZFS vault migration leaves
behind when it does not complete. It answers the two review comments that
arrived on #6036 just before it merged
(r3974771201,
r3974798061),
and then the failures that testing those fixes uncovered.

The migration copies the carried-over filesystem vault into a staging zvol,
parks the original as <vault>.old, and renames the copy into place. Four
things could go wrong with what that leaves on the pool.

A failed migration left the staging zvol behind. Every error after the zvol
was created returned without destroying it, so a partial copy stayed on the
pool, sized to the free space it was given. On EVE-k the next attempt cleared
it, but a device that reverts to EVE-kvm runs no migration code at all and never
reclaims it. The migration now discards it on every failure before the swap,
restores the parked vault if the second rename fails, and creates the etcd zvol
only after the free-space checks so a declined migration leaves nothing.

A partial copy could be adopted as the vault. Recovery decided on dataset
presence alone, so a staging zvol left by a failed migration was
indistinguishable from one whose swap lost power. That is reachable without a
power cut: a failed migration reports the vault as unusable, and with no
controller key and cleanup permitted, vaultmgr destroys the vault and sets up a
fresh one, arriving at recovery with no vault and a leftover copy. The migration
now records the staging dataset on /persist once its copy is complete, and
recovery promotes one only when that record names it.

Two paths destroyed the contents after a revert. Measured on a device: a
power cut between the two renames does not come back on EVE-k. It falls back to
EVE-kvm, which finds no vault and creates a fresh empty one, leaving the
contents only in <vault>.old and the staging zvol. Recovery then treated the
vault in place as authoritative and destroyed both, and the migration's own
entry cleanup did the same from the other side. Both now ask whether a
pre-migration vault is parked beside a vault that is not the migrated zvol, and
keep the datasets rather than destroy the only copies.

Nothing removed the parked vault once the update was committed. #6036
introduced <vault>.old to make the swap power-safe but tied its removal to
nothing, so on the revert path it stayed for the life of the device. Marking the
partition active is where that fallback stops being worth anything, so recovery
discards both datasets from then on and the migration stops declining over them.

One gap is deliberately left open: after a revert the running vault is the empty
one EVE-kvm created, and nothing on that boot can put the contents back, because
EVE-kvm carries no migration code. Closing it means deferring the migration
until the EVE-k partition is committed, which is the design question under
discussion on #6036 rather than something this change can decide.

A fault point compiled in only under FAULT_INJECTION=y parks the migration
inside the swap window so a test can cut power there; production builds get an
identity wrapper and consult no marker.

PR dependencies

None.

How to test and validate this PR

Unit coverage, make -C pkg/pillar test (pkg/pillar/vault): the recovery
decision over every combination of leftover datasets, with and without the swap
record and with the partition committed or not; the migration with a failure
injected at each step after the staging zvol is created; a failed swap rename;
a declined migration; and the on-disk swap record. Each case was checked against
the pre-change behaviour by neutering the fix and confirming the test fails on
its assertion rather than on a compile error.

Device coverage, on an image carrying this branch on top of #6063:

  • eden's kvm→k matrix, the three ZFS legs (twodisk-zfs, zfs-grow,
    zfs-notail), run twice — 3/3 both times, volumemgr readiness clearing in 46
    and 21 minutes of its 85-minute budget. [WIP] tests/update_eve_image: kvm→k cross-HV conversion + resize test suite eden#1209 adds a
    post-conversion assertion on the resulting layout, so those runs check the
    zvol vault, the etcd volume and the absence of leftovers directly.
  • TestKvmToKZFSVaultPowerCutMidSwap in evetest: kvm→k boot-disk conversion tests + volverify app #6267, on a FAULT_INJECTION=y image:
    cuts power inside the swap window, then asserts the contents survive, that an
    EVE-k boot passing over them keeps them, and that committing the partition
    drops both plus the swap record. The revert to EVE-kvm reproduced on every run;
    the final run returned ~830 MB to the pool at the commit.

Changelog notes

A kvm→EVE-k conversion interrupted while migrating a ZFS vault no longer leaves
volumes occupying the pool, and no longer discards the vault contents: they are
kept until the update is committed, and released afterwards. A migration whose
copy did not finish is never adopted as the device's vault.

PR Backports

This fixes EVE-k in-field conversion code that is not present in the current
LTS branches.

  • 17.0-stable: No.
  • 16.0-stable: No.
  • 14.5-stable: No.
  • 13.4-stable: No.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

Two notes for review. Yetus is likely to report a "new" revive finding at
pkg/pillar/cmd/vaultmgr/vaultmgr.go:117 (receiver named ctxPtr): it is
pre-existing and this branch only adds an import above it, but findings are
diffed by file and line. And the test job failed on
dpcmanager TestPNACDHCPReacquireMaxRetries, a package this branch does not
touch; a sibling PR fails the same job on a different unrelated test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3

@codecov

codecov Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 49.78355% with 116 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.56%. Comparing base (3b1c143) to head (263af5b).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/vault/handler_zfs.go 61.81% 37 Missing and 26 partials ⚠️
pkg/pillar/vault/handler_zfs_ops.go 21.31% 47 Missing and 1 partial ⚠️
pkg/pillar/cmd/vaultmgr/vaultmgr.go 0.00% 4 Missing ⚠️
.../pillar/vault/faultinjection_migration_disabled.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6530      +/-   ##
==========================================
+ Coverage   26.16%   26.56%   +0.39%     
==========================================
  Files         514      526      +12     
  Lines       94703    96572    +1869     
==========================================
+ Hits        24778    25650     +872     
- Misses      68014    68795     +781     
- Partials     1911     2127     +216     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark
eriknordmark marked this pull request as ready for review September 12, 2026 14:43
@eriknordmark
eriknordmark force-pushed the vault-zvol-migration-cleanup branch from 0939b37 to 5ff321a Compare September 14, 2026 16:00
@eriknordmark
eriknordmark requested a review from rene September 16, 2026 21:47
@eriknordmark
eriknordmark force-pushed the vault-zvol-migration-cleanup branch from 4aa650a to 263af5b Compare September 17, 2026 19:01
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 17, 2026
Integration branch on master with the conversion chain (lf-edge#6530, lf-edge#6063, fork#6)
and the purge/PVC robustness set (lf-edge#6442, lf-edge#6406, lf-edge#6478). Records the tips each
piece was replayed from, the branch-local adapters that no PR carries, and the
storage-resizer content pin.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 18, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rene

rene commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 3m 13s —— View job


Review complete

I reviewed the ZFS vault migration cleanup and recovery logic. Two findings, posted inline:

  • Read PR diff and changed files
  • Analyze vault ZFS migration cleanup logic
  • Analyze recovery decision & swap record
  • Analyze fault injection code
  • Post review feedback

Findings

  1. CurrentPartitionCommitted is never set on no-TPM devices (inline) — SetHandlerOptions is only called inside the if tpmEnabled { block in vaultmgr.go:358, so the no-TPM EVE-k migration/recovery paths always see committed=false. The post-commit cleanup of <vault>.old + staging never runs there, and a no-TPM foreign-vault device refuses to migrate forever — i.e. the exact leak/stranding this PR closes stays open on no-TPM. Unit tests set the field directly, so they don't catch the wiring.

  2. etcd zvol not cleaned on a failed migration (inline, lower severity) — created before the staging zvol but not removed by the failure defer; on a revert to EVE-kvm it lingers. Bounded in size, unlike the staging zvol, so much milder — but the same "revert leaves a dataset" shape.

The rest of the change looks solid: the swap-marker gating of promotion, the foreignVaultInPlace symmetry between recovery and migration, the restore-on-failed-second-rename, and the partitionCommitted handling all line up with the design described in the PR body, and the recovery decision table is well covered by the tests.

The pre-existing revive/ctxPtr note and the unrelated dpcmanager test failure called out in the PR body are not caused by this change.
· branch vault-zvol-migration-cleanup

Comment thread pkg/pillar/cmd/vaultmgr/vaultmgr.go Resolved
Comment thread pkg/pillar/vault/handler_zfs.go Resolved
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 18, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 18, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 18, 2026
Integration branch on master with the conversion chain (lf-edge#6530, lf-edge#6063, fork#6)
and the robustness set (lf-edge#6442, lf-edge#6406, lf-edge#6478, lf-edge#6505, lf-edge#6451, lf-edge#6590, lf-edge#6599).
Records the tips each piece was replayed from, where two PRs collide and which
side won, the branch-local adapters that no PR carries, and the
storage-resizer content pin.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 18, 2026
Integration branch on master with the conversion chain plus the fault-injection
stress harness (lf-edge#6530, lf-edge#6063, fork#6, fork#7) and the robustness set (lf-edge#6442,
lf-edge#6406, lf-edge#6478, lf-edge#6505, lf-edge#6451, lf-edge#6590, lf-edge#6599). Records the tips each piece was
replayed from, where two PRs collide and which side won, the branch-local
adapters that no PR carries, and the chaos storage-resizer content pin.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 18, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a4cd6bc)
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 18, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a4cd6bc)
eriknordmark and others added 7 commits September 23, 2026 16:17
A kvm->EVE-k vault migration copies the carried-over filesystem vault
into a staging zvol sized to the pool's free space. Every failure after
that zvol was created returned without destroying it, so the partial
copy stayed on the pool. On EVE-k the next boot's migration attempt
cleared it, but a device that falls back to EVE-kvm inherits a zvol
holding up to all the space the pool had free, and EVE-kvm has no
migration code that would ever reclaim it. A copy that runs out of room
part-way therefore left the pool full for the flavor the device fell
back to.

The migration now discards the staging zvol on every failure before the
swap, and puts the pre-migration vault back if the second rename of the
swap fails, so the vault is never left absent for a fallback boot to
find. The empty etcd zvol is created after the free-space checks rather
than before them, so a migration declined for lack of space leaves
nothing behind either.

The storage calls the migration and the recovery make now go through an
interface, which lets the tests drive both against an in-memory pool and
fail any single operation; they otherwise need a real ZFS pool and a
power cut.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Recovering an interrupted kvm->EVE-k vault migration decided purely on
which datasets were present: with the vault absent and a staging zvol
in place, that zvol was renamed into place as the vault. Nothing
recorded whether its copy had finished, so a staging zvol left by a
failed migration was indistinguishable from one whose swap lost power
between the two renames.

That is reachable without a power cut. A failed migration is reported as
a vault that cannot be unlocked, so the device waits for a key from the
controller; receiving none, and with cleanup still permitted, vaultmgr
destroys the vault and sets up a fresh one — which arrives at recovery
with no vault and a leftover staging zvol, and adopts its partial copy
as the device's vault.

The migration now records the staging dataset on /persist once its copy
is complete, and recovery promotes a staging zvol only when that record
names it; otherwise the leftover is discarded and, where the
pre-migration vault is still parked under its backup name, restored so
the migration can be retried. Removing the vault takes its migration
datasets with it, so nothing is left for a fresh vault to adopt.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Record what the kvm->EVE-k vault migration leaves behind and when, so
the reason a staging zvol is only sometimes promoted to the vault is
readable without the code: the swap record that distinguishes a
complete copy from a partial one, the restore of the pre-migration
vault when the swap cannot finish, and the fact that a staging zvol
occupies only the space written to it.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The one failure the unit tests cannot reach is power loss inside the
rename swap. Faking the storage operations can make any step return an
error, but only losing the machine at that instant says whether ZFS's
rename and the swap record's fsync really survive it -- which is the
property the recovery path is built on.

Add a fault point that parks the migration in that window instead of
failing it, so a test can cut power from outside at a known moment. It
is compiled in only under FAULT_INJECTION=y, wrapping the storage
operations; production builds get an identity wrapper and consult no
marker. The marker lives on /persist because the migration runs on the
first EVE-k boot, which /tmp does not survive, and because file presence
under /config is measured into PCR14, which seals the vault.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
A device that falls back to EVE-kvm inside the migration's rename swap
comes back with no vault at persist/vault -- it has been renamed aside --
so EVE-kvm creates a fresh, empty one. The contents then exist only in
the parked pre-migration vault and the completed staging copy. Recovery
treated any vault in place as authoritative and destroyed both, turning a
state that still holds the data into a permanent loss.

Recovery now discards leftovers only when the vault in place is the
migrated zvol. A filesystem vault beside migration datasets is not the
one the swap was producing, so the datasets are kept and the swap record
left alone, since it is what says the swap never completed. An unreadable
dataset type takes the keeping side.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
Keeping the migration datasets in recovery only closed one of two doors.
The migration's own entry cleanup drops them before copying, on the
premise that the vault it is about to migrate is the one they came from.
After a fallback to EVE-kvm inside the swap window that premise is false:
the vault in place is the fresh one kvm created, so the migration would
destroy the parked original and the completed copy and then migrate an
empty vault over the top.

Both callers now ask the same question -- is a pre-migration vault parked
while the vault in place is not the migrated zvol -- and the migration
declines rather than proceeding. Recovery's condition narrows to match:
a staging zvol with no parked backup is debris from an attempt that never
reached the swap, and stays droppable, so a device that merely failed a
copy still retries normally.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
The swap parks the pre-migration vault under <vault>.old so that losing
power between the two renames cannot leave the vault absent. Nothing
removed it on the path where it matters most: a device that reverts to
EVE-kvm keeps both that dataset and the completed copy, EVE-kvm runs no
vault migration code, and the space stays held for as long as the device
lives. Marking the partition active is the point where that fallback
stops being worth anything -- the device is not going back to the other
flavor -- so recovery now discards both datasets from then on, and the
migration stops declining to run over them.

vaultmgr reads the partition state once at startup and hands it to the
handler: a committed partition cannot become uncommitted, and one that
commits later is picked up on the next boot, which is when leftovers
matter again.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
The vault swap runs while a device upgrades to EVE-k, so the boot that
lands inside its window is as often the EVE-kvm fallback that follows a
failed upgrade as an EVE-k retry. EVE-kvm reached that boot with
persist/vault absent, read it as a fresh install and created an empty
vault, stranding every blob in the datasets the swap had parked and
leaving the device running on a vault with a new key. Recovery now runs
on both flavors, before either may decide the vault is missing.

What the two flavors do differs, because EVE-kvm cannot mount a zvol:
it puts the parked filesystem vault back and leaves the copy to the
next EVE-k attempt, where EVE-k promotes the completed copy instead. A
vault already in place beside a parked one is likewise never the
migrated vault on EVE-kvm, so it keeps both datasets whatever the
partition state says rather than reclaiming the pool space over the
only surviving copy of the contents.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the vault-zvol-migration-cleanup branch from ae691d5 to 4477e2d Compare September 23, 2026 23:27
eriknordmark and others added 2 commits September 23, 2026 16:52
A device converted from EVE-kvm runs the ZFS vault migration and its
recovery whether or not it has a TPM, and both ask whether the running
partition has been marked active before they let go of the pre-migration
vault and the staging copy. That answer was read only on the TPM path,
so on a no-TPM device it stayed false for the life of the boot: the
datasets were never released once the update committed, and a device
that fell back to EVE-kvm inside the swap window and acquired an empty
vault refused to migrate over it forever, leaving the contents in
datasets nothing would reclaim.

Read it for every device and leave the TPM key mode to the TPM path.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The migration creates the EVE-k etcd zvol on its way to the vault swap,
and a failure after that point left it on the pool. It is bounded by the
configured etcd size rather than free space, so it is milder than the
staging zvol, but it has the same ending: a device that falls back to
EVE-kvm runs no migration code and never reclaims it.

The failure cleanup now takes it along with the staging zvol, and only
when this attempt created it -- an etcd zvol already on the pool is the
device's etcd storage, not migration debris.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 24, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 24, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 24, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@eriknordmark

Copy link
Copy Markdown
Contributor Author

@claude

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @eriknordmark's task in 3m 9s —— View job


Re-review complete

  • Read PR diff and changed files
  • Verify the two earlier findings are addressed (no-TPM wiring, etcd zvol cleanup)
  • Re-review migration / recovery / fault-injection logic for new issues
  • Post review feedback

Both earlier findings are fixed correctly:

  1. No-TPM CurrentPartitionCommitted wiring — vaultmgr.go:355-373 now reads zboot.IsCurrentPartitionStateActive() into HandlerOptions unconditionally, and the TPM block only fills in TpmKeyOnlyMode. The no-TPM EVE-k recovery/migration paths now see the real committed state.
  2. etcd zvol leak on a failed migration — handler_zfs.go:396-426 creates the etcd zvol behind a !DatasetExist guard, records etcdCreatedHere, and the failure defer (now registered before CreateVaultZvol) destroys it only when this attempt created it. A pre-existing etcd zvol is left alone. Correct.

The core design holds together on re-read: the swap-marker gating of finish-swap, the foreignVaultInPlace symmetry between recovery and migration, the restore-on-failed-second-rename, and the partitionCommitted drop all match the PR description. The swapped guard correctly prevents the defer from tearing down a vault once it is live (e.g. a MountVaultZvol failure after the swap).

One new finding (posted inline, low/medium severity):

  • RemoveDefaultVault drops both migration datasets without the keep-leftovers protection (inline). It destroys staging + backup unconditionally; its comment assumes the datasets belong to the vault being destroyed, which is not true in the foreign-vault state. On the empty-controller-key + cleanup-allowed path this can discard the only copies of the contents before the partition is committed — contradicting the "kept until commit" guarantee. May be intended (controller explicitly authorizing a wipe); if so worth saying so, otherwise gate the drop on !foreignVaultInPlace(...) || CurrentPartitionCommitted.

The pre-existing revive/ctxPtr note and the unrelated dpcmanager test failure from the PR body are not caused by this change.
· branch vault-zvol-migration-cleanup

// creates next a staging zvol holding an unrelated, possibly partial copy.
// Non-fatal: nothing promotes a staging zvol without the swap marker, and
// failing here would leave the device with no vault at all.
if err := h.dropMigrationLeftovers(types.SealedDataset+vaultStagingSuffix,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RemoveDefaultVault drops both the staging and backup datasets unconditionally, without the foreignVaultInPlace / partitionCommitted check that recoverInterruptedVaultMigration and migrateVaultFsToZvol now apply. The comment justifies this with "the migration datasets belong to the vault that was just destroyed" — but that assumption does not hold in the keep-leftovers state, where the datasets belong to the original pre-migration vault, not to the (foreign, empty) vault sitting at SealedDataset.

Reachable path (TPM EVE-k, pre-commit): a revert leaves a foreign empty vault beside <vault>.old + staging holding the only copies of the contents. On the next EVE-k boot recoverInterruptedVaultMigration correctly returns keep-leftovers, but unlockVault → migrateVaultFsToZvol then refuses (foreign, not committed) and SetupDefaultVault fails. The controller sends an empty key with cleanup allowed (vaultmgr.go:621) → RemoveDefaultVault destroys the foreign vault and both leftover datasets, discarding the contents before the partition is committed — the exact "kept until the update is committed" guarantee this PR sets out to provide.

The counter-argument is that an empty controller key + cleanup-allowed is an explicit "wipe this vault" authorization, so data loss is expected. If that is the intent, worth stating it here (the current comment claims the opposite reasoning). Otherwise consider gating the drop on !foreignVaultInPlace(...) || CurrentPartitionCommitted, matching the recovery/migration decision.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but you need to fix the conflicts @eriknordmark

@eriknordmark

Copy link
Copy Markdown
Contributor Author

All commits in this PR merged to master as part of #6451.

eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 27, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

The flavor change moves the measurements the vault seal is bound to, so the
marker is read only once vaultmgr has published an unlock method: an
unmounted vault is an empty directory and would read as lost content. A
failed decline dumps the dataset's type and mount state, the vault
directory, vaultmgr's status and a newlog sweep, so a vault that did not
unlock can be told from one that lost its data.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 27, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

The flavor change moves the measurements the vault seal is bound to, so the
marker is read only once vaultmgr has published an unlock method: an
unmounted vault is an empty directory and would read as lost content. A
failed decline dumps the dataset's type and mount state, the vault
directory, vaultmgr's status and a newlog sweep, so a vault that did not
unlock can be told from one that lost its data.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Sep 28, 2026
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault
no longer fits twice in the pool, so the vault migration declines, and
assert the device is left with nothing the abandoned migration created: no
staging zvol, no parked backup, no etcd volume, no swap record, the vault
still a filesystem dataset with its content readable, and the pool's free
space back where it started. A leaked staging zvol is what makes that
worth asserting -- it holds whatever was copied into it, up to all the
space the pool had free, and the EVE-kvm the device falls back to has no
migration code that would reclaim it.

The condition is created by growing the vault rather than by filling the
pool, since the migration compares the two: the pool stays roughly half
free, well clear of the threshold at which EVE puts the device into
low-disk maintenance mode. A log assertion on the decline keeps a revert
for an unrelated reason from satisfying everything else. Against a build
without lf-edge#6530 the etcd-volume assertion fails, which is what
makes the run meaningful.

The pool sits on a disk of its own, as in the eden leg this is ported from: a
32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot
disk at all. The migration sizes its staging zvol from the pool's free space, so
where the pool lives and how big it is are the variables under test -- and a
pool on its own disk is also a different path through storage-init than one on
the boot disk's P3. The extra disk is partitioned offline and left unformatted,
so EVE formats it on the next boot as it would a factory-fresh disk.

The flavor change moves the measurements the vault seal is bound to, so the
marker is read only once vaultmgr has published an unlock method: an
unmounted vault is an empty directory and would read as lost content. A
failed decline dumps the dataset's type and mount state, the vault
directory, vaultmgr's status and a newlog sweep, so a vault that did not
unlock can be told from one that lost its data.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@eriknordmark
eriknordmark deleted the vault-zvol-migration-cleanup branch October 8, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants