Repository navigation
vault: clean up after a failed kvm→EVE-k vault migration - #6530
eriknordmark wants to merge 10 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6530 +/- ##
==========================================
+ Coverage 26.16% 26.56% +0.39%
==========================================
Files 514 526 +12
Lines 94703 96572 +1869
==========================================
+ Hits 24778 25650 +872
- Misses 68014 68795 +781
- Partials 1911 2127 +216 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
0939b37 to
5ff321a
Compare
4aa650a to
263af5b
Compare
Integration branch on master with the conversion chain (lf-edge#6530, lf-edge#6063, fork#6) and the purge/PVC robustness set (lf-edge#6442, lf-edge#6406, lf-edge#6478). Records the tips each piece was replayed from, the branch-local adapters that no PR carries, and the storage-resizer content pin. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @rene's task in 3m 13s —— View job Review completeI reviewed the ZFS vault migration cleanup and recovery logic. Two findings, posted inline:
Findings
The rest of the change looks solid: the swap-marker gating of promotion, the The pre-existing |
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Integration branch on master with the conversion chain (lf-edge#6530, lf-edge#6063, fork#6) and the robustness set (lf-edge#6442, lf-edge#6406, lf-edge#6478, lf-edge#6505, lf-edge#6451, lf-edge#6590, lf-edge#6599). Records the tips each piece was replayed from, where two PRs collide and which side won, the branch-local adapters that no PR carries, and the storage-resizer content pin. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Integration branch on master with the conversion chain plus the fault-injection stress harness (lf-edge#6530, lf-edge#6063, fork#6, fork#7) and the robustness set (lf-edge#6442, lf-edge#6406, lf-edge#6478, lf-edge#6505, lf-edge#6451, lf-edge#6590, lf-edge#6599). Records the tips each piece was replayed from, where two PRs collide and which side won, the branch-local adapters that no PR carries, and the chaos storage-resizer content pin. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit a4cd6bc)
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit a4cd6bc)
A kvm->EVE-k vault migration copies the carried-over filesystem vault into a staging zvol sized to the pool's free space. Every failure after that zvol was created returned without destroying it, so the partial copy stayed on the pool. On EVE-k the next boot's migration attempt cleared it, but a device that falls back to EVE-kvm inherits a zvol holding up to all the space the pool had free, and EVE-kvm has no migration code that would ever reclaim it. A copy that runs out of room part-way therefore left the pool full for the flavor the device fell back to. The migration now discards the staging zvol on every failure before the swap, and puts the pre-migration vault back if the second rename of the swap fails, so the vault is never left absent for a fallback boot to find. The empty etcd zvol is created after the free-space checks rather than before them, so a migration declined for lack of space leaves nothing behind either. The storage calls the migration and the recovery make now go through an interface, which lets the tests drive both against an in-memory pool and fail any single operation; they otherwise need a real ZFS pool and a power cut. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Recovering an interrupted kvm->EVE-k vault migration decided purely on which datasets were present: with the vault absent and a staging zvol in place, that zvol was renamed into place as the vault. Nothing recorded whether its copy had finished, so a staging zvol left by a failed migration was indistinguishable from one whose swap lost power between the two renames. That is reachable without a power cut. A failed migration is reported as a vault that cannot be unlocked, so the device waits for a key from the controller; receiving none, and with cleanup still permitted, vaultmgr destroys the vault and sets up a fresh one — which arrives at recovery with no vault and a leftover staging zvol, and adopts its partial copy as the device's vault. The migration now records the staging dataset on /persist once its copy is complete, and recovery promotes a staging zvol only when that record names it; otherwise the leftover is discarded and, where the pre-migration vault is still parked under its backup name, restored so the migration can be retried. Removing the vault takes its migration datasets with it, so nothing is left for a fresh vault to adopt. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Record what the kvm->EVE-k vault migration leaves behind and when, so the reason a staging zvol is only sometimes promoted to the vault is readable without the code: the swap record that distinguishes a complete copy from a partial one, the restore of the pre-migration vault when the swap cannot finish, and the fact that a staging zvol occupies only the space written to it. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The one failure the unit tests cannot reach is power loss inside the rename swap. Faking the storage operations can make any step return an error, but only losing the machine at that instant says whether ZFS's rename and the swap record's fsync really survive it -- which is the property the recovery path is built on. Add a fault point that parks the migration in that window instead of failing it, so a test can cut power from outside at a known moment. It is compiled in only under FAULT_INJECTION=y, wrapping the storage operations; production builds get an identity wrapper and consult no marker. The marker lives on /persist because the migration runs on the first EVE-k boot, which /tmp does not survive, and because file presence under /config is measured into PCR14, which seals the vault. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
A device that falls back to EVE-kvm inside the migration's rename swap comes back with no vault at persist/vault -- it has been renamed aside -- so EVE-kvm creates a fresh, empty one. The contents then exist only in the parked pre-migration vault and the completed staging copy. Recovery treated any vault in place as authoritative and destroyed both, turning a state that still holds the data into a permanent loss. Recovery now discards leftovers only when the vault in place is the migrated zvol. A filesystem vault beside migration datasets is not the one the swap was producing, so the datasets are kept and the swap record left alone, since it is what says the swap never completed. An unreadable dataset type takes the keeping side. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
Keeping the migration datasets in recovery only closed one of two doors. The migration's own entry cleanup drops them before copying, on the premise that the vault it is about to migrate is the one they came from. After a fallback to EVE-kvm inside the swap window that premise is false: the vault in place is the fresh one kvm created, so the migration would destroy the parked original and the completed copy and then migrate an empty vault over the top. Both callers now ask the same question -- is a pre-migration vault parked while the vault in place is not the migrated zvol -- and the migration declines rather than proceeding. Recovery's condition narrows to match: a staging zvol with no parked backup is debris from an attempt that never reached the swap, and stays droppable, so a device that merely failed a copy still retries normally. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
The swap parks the pre-migration vault under <vault>.old so that losing power between the two renames cannot leave the vault absent. Nothing removed it on the path where it matters most: a device that reverts to EVE-kvm keeps both that dataset and the completed copy, EVE-kvm runs no vault migration code, and the space stays held for as long as the device lives. Marking the partition active is the point where that fallback stops being worth anything -- the device is not going back to the other flavor -- so recovery now discards both datasets from then on, and the migration stops declining to run over them. vaultmgr reads the partition state once at startup and hands it to the handler: a committed partition cannot become uncommitted, and one that commits later is picked up on the next boot, which is when leftovers matter again. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3
The vault swap runs while a device upgrades to EVE-k, so the boot that lands inside its window is as often the EVE-kvm fallback that follows a failed upgrade as an EVE-k retry. EVE-kvm reached that boot with persist/vault absent, read it as a fresh install and created an empty vault, stranding every blob in the datasets the swap had parked and leaving the device running on a vault with a new key. Recovery now runs on both flavors, before either may decide the vault is missing. What the two flavors do differs, because EVE-kvm cannot mount a zvol: it puts the parked filesystem vault back and leaves the copy to the next EVE-k attempt, where EVE-k promotes the completed copy instead. A vault already in place beside a parked one is likewise never the migrated vault on EVE-kvm, so it keeps both datasets whatever the partition state says rather than reclaiming the pool space over the only surviving copy of the contents. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ae691d5 to
4477e2d
Compare
A device converted from EVE-kvm runs the ZFS vault migration and its recovery whether or not it has a TPM, and both ask whether the running partition has been marked active before they let go of the pre-migration vault and the staging copy. That answer was read only on the TPM path, so on a no-TPM device it stayed false for the life of the boot: the datasets were never released once the update committed, and a device that fell back to EVE-kvm inside the swap window and acquired an empty vault refused to migrate over it forever, leaving the contents in datasets nothing would reclaim. Read it for every device and leave the TPM key mode to the TPM path. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The migration creates the EVE-k etcd zvol on its way to the vault swap, and a failure after that point left it on the pool. It is bounded by the configured etcd size rather than free space, so it is milder than the staging zvol, but it has the same ending: a device that falls back to EVE-kvm runs no migration code and never reclaims it. The failure cleanup now takes it along with the staging zvol, and only when this attempt created it -- an etcd zvol already on the pool is the device's etcd storage, not migration debris. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @eriknordmark's task in 3m 9s —— View job Re-review complete
Both earlier findings are fixed correctly:
The core design holds together on re-read: the swap-marker gating of One new finding (posted inline, low/medium severity):
The pre-existing |
| // creates next a staging zvol holding an unrelated, possibly partial copy. | ||
| // Non-fatal: nothing promotes a staging zvol without the swap marker, and | ||
| // failing here would leave the device with no vault at all. | ||
| if err := h.dropMigrationLeftovers(types.SealedDataset+vaultStagingSuffix, |
There was a problem hiding this comment.
RemoveDefaultVault drops both the staging and backup datasets unconditionally, without the foreignVaultInPlace / partitionCommitted check that recoverInterruptedVaultMigration and migrateVaultFsToZvol now apply. The comment justifies this with "the migration datasets belong to the vault that was just destroyed" — but that assumption does not hold in the keep-leftovers state, where the datasets belong to the original pre-migration vault, not to the (foreign, empty) vault sitting at SealedDataset.
Reachable path (TPM EVE-k, pre-commit): a revert leaves a foreign empty vault beside <vault>.old + staging holding the only copies of the contents. On the next EVE-k boot recoverInterruptedVaultMigration correctly returns keep-leftovers, but unlockVault → migrateVaultFsToZvol then refuses (foreign, not committed) and SetupDefaultVault fails. The controller sends an empty key with cleanup allowed (vaultmgr.go:621) → RemoveDefaultVault destroys the foreign vault and both leftover datasets, discarding the contents before the partition is committed — the exact "kept until the update is committed" guarantee this PR sets out to provide.
The counter-argument is that an empty controller key + cleanup-allowed is an explicit "wipe this vault" authorization, so data loss is expected. If that is the intent, worth stating it here (the current comment claims the opposite reasoning). Otherwise consider gating the drop on !foreignVaultInPlace(...) || CurrentPartitionCommitted, matching the recovery/migration decision.
rene
left a comment
There was a problem hiding this comment.
LGTM, but you need to fix the conflicts @eriknordmark
|
All commits in this PR merged to master as part of #6451. |
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. The flavor change moves the measurements the vault seal is bound to, so the marker is read only once vaultmgr has published an unlock method: an unmounted vault is an empty directory and would read as lost content. A failed decline dumps the dataset's type and mount state, the vault directory, vaultmgr's status and a newlog sweep, so a vault that did not unlock can be told from one that lost its data. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. The flavor change moves the measurements the vault seal is bound to, so the marker is read only once vaultmgr has published an unlock method: an unmounted vault is an empty directory and would read as lost content. A failed decline dumps the dataset's type and mount state, the vault directory, vaultmgr's status and a newlog sweep, so a vault that did not unlock can be told from one that lost its data. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drive a kvm→EVE-K update on a device whose /persist is ZFS and whose vault no longer fits twice in the pool, so the vault migration declines, and assert the device is left with nothing the abandoned migration created: no staging zvol, no parked backup, no etcd volume, no swap record, the vault still a filesystem dataset with its content readable, and the pool's free space back where it started. A leaked staging zvol is what makes that worth asserting -- it holds whatever was copied into it, up to all the space the pool had free, and the EVE-kvm the device falls back to has no migration code that would reclaim it. The condition is created by growing the vault rather than by filling the pool, since the migration compares the two: the pool stays roughly half free, well clear of the threshold at which EVE puts the device into low-disk maintenance mode. A log assertion on the decline keeps a revert for an unrelated reason from satisfying everything else. Against a build without lf-edge#6530 the etcd-volume assertion fails, which is what makes the run meaningful. The pool sits on a disk of its own, as in the eden leg this is ported from: a 32 GiB boot disk beside a 32 GiB disk carrying /persist, and no P3 on the boot disk at all. The migration sizes its staging zvol from the pool's free space, so where the pool lives and how big it is are the variables under test -- and a pool on its own disk is also a different path through storage-init than one on the boot disk's P3. The extra disk is partitioned offline and left unformatted, so EVE formats it on the next boot as it would a factory-fresh disk. The flavor change moves the measurements the vault seal is bound to, so the marker is read only once vaultmgr has published an unlock method: an unmounted vault is an empty directory and would read as lost content. A failed decline dumps the dataset's type and mount state, the vault directory, vaultmgr's status and a newlog sweep, so a vault that did not unlock can be told from one that lost its data. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Description
Follow-up to #6036, closing the gaps in what the ZFS vault migration leaves
behind when it does not complete. It answers the two review comments that
arrived on #6036 just before it merged
(r3974771201,
r3974798061),
and then the failures that testing those fixes uncovered.
The migration copies the carried-over filesystem vault into a staging zvol,
parks the original as
<vault>.old, and renames the copy into place. Fourthings could go wrong with what that leaves on the pool.
A failed migration left the staging zvol behind. Every error after the zvol
was created returned without destroying it, so a partial copy stayed on the
pool, sized to the free space it was given. On EVE-k the next attempt cleared
it, but a device that reverts to EVE-kvm runs no migration code at all and never
reclaims it. The migration now discards it on every failure before the swap,
restores the parked vault if the second rename fails, and creates the etcd zvol
only after the free-space checks so a declined migration leaves nothing.
A partial copy could be adopted as the vault. Recovery decided on dataset
presence alone, so a staging zvol left by a failed migration was
indistinguishable from one whose swap lost power. That is reachable without a
power cut: a failed migration reports the vault as unusable, and with no
controller key and cleanup permitted, vaultmgr destroys the vault and sets up a
fresh one, arriving at recovery with no vault and a leftover copy. The migration
now records the staging dataset on
/persistonce its copy is complete, andrecovery promotes one only when that record names it.
Two paths destroyed the contents after a revert. Measured on a device: a
power cut between the two renames does not come back on EVE-k. It falls back to
EVE-kvm, which finds no vault and creates a fresh empty one, leaving the
contents only in
<vault>.oldand the staging zvol. Recovery then treated thevault in place as authoritative and destroyed both, and the migration's own
entry cleanup did the same from the other side. Both now ask whether a
pre-migration vault is parked beside a vault that is not the migrated zvol, and
keep the datasets rather than destroy the only copies.
Nothing removed the parked vault once the update was committed. #6036
introduced
<vault>.oldto make the swap power-safe but tied its removal tonothing, so on the revert path it stayed for the life of the device. Marking the
partition active is where that fallback stops being worth anything, so recovery
discards both datasets from then on and the migration stops declining over them.
One gap is deliberately left open: after a revert the running vault is the empty
one EVE-kvm created, and nothing on that boot can put the contents back, because
EVE-kvm carries no migration code. Closing it means deferring the migration
until the EVE-k partition is committed, which is the design question under
discussion on #6036 rather than something this change can decide.
A fault point compiled in only under
FAULT_INJECTION=yparks the migrationinside the swap window so a test can cut power there; production builds get an
identity wrapper and consult no marker.
PR dependencies
None.
How to test and validate this PR
Unit coverage,
make -C pkg/pillar test(pkg/pillar/vault): the recoverydecision over every combination of leftover datasets, with and without the swap
record and with the partition committed or not; the migration with a failure
injected at each step after the staging zvol is created; a failed swap rename;
a declined migration; and the on-disk swap record. Each case was checked against
the pre-change behaviour by neutering the fix and confirming the test fails on
its assertion rather than on a compile error.
Device coverage, on an image carrying this branch on top of #6063:
twodisk-zfs,zfs-grow,zfs-notail), run twice — 3/3 both times, volumemgr readiness clearing in 46and 21 minutes of its 85-minute budget. [WIP] tests/update_eve_image: kvm→k cross-HV conversion + resize test suite eden#1209 adds a
post-conversion assertion on the resulting layout, so those runs check the
zvol vault, the etcd volume and the absence of leftovers directly.
TestKvmToKZFSVaultPowerCutMidSwapin evetest: kvm→k boot-disk conversion tests + volverify app #6267, on aFAULT_INJECTION=yimage:cuts power inside the swap window, then asserts the contents survive, that an
EVE-k boot passing over them keeps them, and that committing the partition
drops both plus the swap record. The revert to EVE-kvm reproduced on every run;
the final run returned ~830 MB to the pool at the commit.
Changelog notes
A kvm→EVE-k conversion interrupted while migrating a ZFS vault no longer leaves
volumes occupying the pool, and no longer discards the vault contents: they are
kept until the update is committed, and released afterwards. A migration whose
copy did not finish is never adopted as the device's vault.
PR Backports
This fixes EVE-k in-field conversion code that is not present in the current
LTS branches.
Checklist
And the last but not least:
check them.
Two notes for review. Yetus is likely to report a "new" revive finding at
pkg/pillar/cmd/vaultmgr/vaultmgr.go:117(receiver namedctxPtr): it ispre-existing and this branch only adds an import above it, but findings are
diffed by file and line. And the
testjob failed ondpcmanager TestPNACDHCPReacquireMaxRetries, a package this branch does nottouch; a sibling PR fails the same job on a different unrelated test.
🤖 Generated with Claude Code
https://claude.ai/code/session_01H2fNrog4UH7mbhrHVGkLN3