Skip to content

kvm to k conversion: allow and preserve app volumes (gate-on-shrink + migrate to PVC) - #6658

Merged
rene merged 10 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-volmig
Oct 16, 2026
Merged

rene merged 10 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-volmig

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Description

This is the third and last piece of the kvm-to-k migration. If there is no need to shrink a filesystem, this PR allows the conversion to proceed even when there are volumes. That is the case e.g., if /persist is on a 2nd disk or if the device was installed with EVE-kvm and the large partitions in 17.0.


Enables an EVE-kvm→EVE-k base-OS conversion to proceed when app volumes exist, and preserves the volume data across it, instead of refusing the upgrade outright.

The cross-flavor gate now blocks only when the boot-disk repartition would shrink /persist; a grow-only or already-large-geometry conversion is allowed. When allowed, EVE-k fills each app volume's Longhorn PVC from the volume the device had on EVE-kvm, so app-written data survives:

  • On ext4 /persist, the first EVE-k boot moves the encrypted volume files out of the Longhorn-owned /persist/vault/volumes; clear volumes stay in /persist/clear/volumes.
  • On ZFS /persist, the volumes are zvols. The vault migration parks the EVE-kvm vault, with its zvols, at persist/vault.old instead of destroying it, and the PVCs are filled from the zvols' device nodes.

The EVE-kvm copies stay until the EVE-k partition is committed, so a fallback to EVE-kvm before then (a power failure in the test window, say) finds its volumes and vault where it left them; on ZFS that fallback undoes the vault swap. After the commit, volumemgr removes each carried volume once its PVC holds the data, and destroys the parked vault when it is empty. docs/EVE-K.md describes the carry, the cleanup and the fallback. The EVE-k descheduler readiness also gates on a Longhorn schedulable disk.

Commits (bottom→top):

  • baseosmgr: gate cross-flavor update on shrink
  • vault: keep the kvm vault until volumes carry
  • upgradeconverter: relocate kvm volumes on EVE-k
  • volumehandlers: migrate kvm volume to PVC on EVE-k
  • volumemgr: drain carried kvm volumes after commit
  • kubeapi: gate descheduler on Longhorn schedulable disk

How to test and validate this PR

  • Unit: diskconvert's TestWillShrinkPersist and TestCheckOnly* cover the gate's decision; the upgradeconverter tests cover the relocate and restore; vault's handler_zfs_test.go covers keeping the parked vault on EVE-k and reverting a completed swap on EVE-kvm; csihandler_test.go covers finding a carried zvol; volumemgr's kvmcarry_test.go covers what the drain removes and when.
  • E2E: TestKvmToKVolumeMigration (evetest konvert) deploys an app with blank, downloaded and clear-text volumes on EVE-kvm on the EVE-k geometry, converts with the app kept in place, asserts the data survives without a re-download, and asserts the EVE-kvm copies are removed after the commit. PERSIST_FILESYSTEM=zfs runs it on ZFS; that variant and the clear-text volume are in evetest: kvm→k boot-disk conversion tests + volverify app #6267. It passed on ZFS against this branch.
  • Not covered end to end: a grow with volumes present (e.g. /persist on a second disk), and the refusal of a shrink with volumes present. TestKvmToKRepartitionVolmig and TestKvmToKRepartitionAppVolume (evetest: kvm→k boot-disk conversion tests + volverify app #6267) take a volume through the shrink, but only on builds that relax this gate for testing.

To confirm the gate by hand: with app volumes on the device, a grow or proceed conversion goes ahead and the volumes migrate; a conversion that would shrink /persist is refused with an explanatory BaseOsStatus.Error.

Changelog notes

An EVE-kvm→EVE-k upgrade is now permitted while application volumes exist (when the boot-disk conversion does not have to shrink storage), on ext4 and ZFS /persist, and application volume data is preserved across the conversion.

PR Backports

  • 16.0-stable: No, feature is not present there.
  • 14.5-stable: No, feature is not present there.
  • 13.4-stable: No, feature is not present there.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't check them.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 56.81818% with 171 lines in your changes missing coverage. Please review.
✅ Project coverage is 27.82%. Comparing base (b73cbcb) to head (01a4ff0).
⚠️ Report is 61 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/vault/handler_zfs.go 35.48% 57 Missing and 3 partials ⚠️
pkg/pillar/volumehandlers/csihandler.go 37.50% 28 Missing and 2 partials ⚠️
...illar/cmd/upgradeconverter/relocate_kvm_volumes.go 51.66% 22 Missing and 7 partials ⚠️
...pillar/cmd/upgradeconverter/restore_kvm_volumes.go 58.82% 16 Missing and 5 partials ⚠️
pkg/pillar/cmd/volumemgr/kvmcarry.go 81.72% 11 Missing and 6 partials ⚠️
pkg/pillar/cmd/baseosmgr/convert.go 0.00% 8 Missing ⚠️
pkg/pillar/kubeapi/descheduler.go 0.00% 3 Missing ⚠️
pkg/pillar/cmd/baseosmgr/paths.go 0.00% 1 Missing ⚠️
pkg/pillar/cmd/volumemgr/volumemgr.go 0.00% 1 Missing ⚠️
pkg/pillar/vault/handler_zfs_ops.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6658      +/-   ##
==========================================
+ Coverage   27.34%   27.82%   +0.47%     
==========================================
  Files         524      538      +14     
  Lines       96107    98201    +2094     
==========================================
+ Hits        26280    27321    +1041     
- Misses      67731    68549     +818     
- Partials     2096     2331     +235     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread docs/EVE-K.md Resolved
eriknordmark and others added 4 commits October 10, 2026 17:57
A kvm<->k base-OS update was blocked whenever any volume existed,
because the cross-flavor /persist layout change could lose volumes.
But only a conversion that shrinks /persist actually disturbs them: a
grow-only conversion (consume the free tail) or a no-op on already-large
geometry leaves /persist untouched. Run the read-only storage-resizer
pre-flight check (CheckOnly) before committing to the download and allow
the update through when the decision is grow/proceed, blocking only on a
shrink (or when the disk cannot be read to decide). The grow-vs-shrink
choice is geometry-stable between this probe and the later authoritative
arming in maybeConvert, so a non-shrink decision here cannot become a
shrink before arming. The check is reached through the baseosmgr seams
so unit tests can stub it rather than probe the real boot disk.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On a ZFS /persist the EVE-kvm app volumes are zvols under the vault
dataset, which the fs-to-zvol vault migration destroyed together with
the old vault right after the swap. That lost every blank and downloaded
volume on a kvm-to-k conversion, and a device falling back to EVE-kvm
before the EVE-k partition is committed was left with a zvol vault it
cannot mount. The old vault now stays parked at persist/vault.old with
its zvols so EVE-k can roll them into PVCs, and an EVE-kvm boot that
finds the migrated zvol beside the parked vault drops the zvol and the
etcd zvol and puts its vault back.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On EVE-k, Longhorn owns /persist/vault/volumes. Before converting a
device from EVE-kvm, baseosmgr records a pending conversion in
/persist/status; the first EVE-k boot then relocates the carried-over
EVE-kvm app volumes into /persist/vault/volumes-kvm, hands Longhorn a
clean empty directory, and removes the marker. Without the marker the
directory is left alone, so an EVE-k to EVE-k upgrade keeps Longhorn's
data and any local-path PVCs, and a conversion retried after a fallback
relocates again. The counterpart restores the volumes on an automatic
rollback to EVE-kvm so the kvm volumemgr finds its app volumes where it
expects them, which is why the restore ships in the EVE-kvm release
that predates enabling the upgrade. On a ZFS /persist the app volumes
are zvols read in place, so the first EVE-k boot only removes the
marker and the restore is a no-op. Includes unit tests for the marker
and for the relocate and restore cores.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a device upgraded from EVE-kvm creates an app volume on EVE-k, the
CSI handler fills the Longhorn PVC from the carried-over EVE-kvm volume
so the app's data survives the conversion, instead of regenerating the
volume from its image (which loses app-written data) or creating a
blank PVC. On ext4 /persist the source is the volume file, which
upgradeconverter relocates to /persist/vault/volumes-kvm when encrypted
and leaves in /persist/clear/volumes when clear. On ZFS it is the zvol's
device node, under the kvm vault the vault migration parks at
persist/vault.old when encrypted and under persist/clear/volumes when
clear; a carried zvol whose device node is missing fails the create
instead of producing an empty PVC. Without a carried volume the normal
image or blank path is taken. Container volumes are excluded, since
they are rebuilt from the preserved containerd content.

The source is left in place after the rollout so an automatic rollback
to EVE-kvm can still find it.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rene

rene commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Markdown errors are still there...

eriknordmark and others added 2 commits October 14, 2026 19:24
A kvm-to-k conversion leaves the EVE-kvm app volumes on /persist after
EVE-k has rolled them into PVCs: the relocated files on ext4, and on ZFS
the zvols together with the parked kvm vault. Once the EVE-k partition
is committed, so no fallback to EVE-kvm can need them, the gc tick now
removes each carried volume whose PVC holds its data, and any that no
volume has claimed for the vdisk GC time, and destroys the parked vault
when its last zvol is gone. A volume whose rollout failed is kept.
EVE-K.md describes the carry, its cleanup and the EVE-kvm fallback end
to end.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The node descheduler readiness check gated only on the Longhorn
daemonsets being ready; also require at least one schedulable disk on
the node so the descheduler does not act before the node can actually
host replicas.

The schedulable-disk and CDI/volume-create readiness this originally
also added for volumemgr are now provided by lf-edge#6121
(checkLonghornSchedulable / ClusterStorageReadyForVolumes), so this no
longer introduces its own waitForLonghornReady / waitForCDIReady helpers
or a WaitForCDI option -- it reuses lf-edge#6121's.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eriknordmark and others added 4 commits October 14, 2026 19:24
Mounting the migrated vault zvol on EVE-k can fail with "block device required": right after the vault datasets are renamed, /dev/zvol/persist/vault can still be the directory of links udev made for the filesystem dataset of that name, and the wait only checked that the path existed. The wait now requires a block device and logs each state the path passes through. If the link is still wrong when it times out, MountVaultZvol mounts the /dev/zd* node that zvol_id names as the vault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
On ext4 /persist, the drain after a kvm-to-k conversion never recognized a carried-over kvm volume file as belonging to its volume, because it matched on the file extension. That extension is the EVE-kvm content format, and an EVE-k volume reports PVC once its PVC exists. So each file was treated as an orphan: it lingered for the vdisk gc grace period, and was then removed even when its rollout had failed and the file was still needed for the retry. Files are now matched by volume UUID and generation alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
FAULT_INJECTION=y images can now fail the disk-key presence check on
demand, so a device test can show that the vault key survives a TPM
error at that check on a real boot. A count or "always" written to
/persist/status/tpm-key-presence-fault fails that many checks, or every
check, with the rest of the TPM untouched. The count is decremented on
disk before each failure, so a reboot neither repeats nor loses one.
Images built without the tag ask the TPM directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
A TPM that fails the check for the sealed disk key, briefly or on every attempt, must not make EVE seal a fresh key over the one the vault is encrypted with. On a FAULT_INJECTION=y image the test fails the presence check twice and then persistently across reboots, and requires that no fresh key is generated and that the vault opened afterwards is the original one, by the local TPM seal. It skips on an image without the fault point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
@eriknordmark eriknordmark changed the title kvm↔k conversion: allow and preserve app volumes (gate-on-shrink + migrate to PVC) kvm to k conversion: allow and preserve app volumes (gate-on-shrink + migrate to PVC) Oct 15, 2026
@rene
rene merged commit 1b0088d into lf-edge:master Oct 16, 2026
60 of 65 checks passed
@eriknordmark
eriknordmark deleted the kvm-to-k-volmig branch October 16, 2026 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants