Repository navigation
volumehandlers: don't adopt a partly uploaded PVC - #6760
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6760 +/- ##
==========================================
+ Coverage 28.48% 28.82% +0.34%
==========================================
Files 540 551 +11
Lines 98258 100063 +1805
==========================================
+ Hits 27990 28846 +856
- Misses 67998 68747 +749
- Partials 2270 2470 +200 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Claude finished @milan-zededa's task in 2m 4s —— View job Answers to the two review questionsQ1 — You're right that if the state is neither
So Q2 — I posted a detailed inline reply on that line. Short version: these are CDI/virtctl annotations, not EVE-written, and a completed upload still carries The rest of the change looks sound — the |
05a5633 to
e6c81da
Compare
| if handler.status.ReferenceName == "" { | ||
| handler.log.Noticef("CreateVolume: PVC %s already exists, skipping creation", pvcName) | ||
| return pvcName, nil | ||
| state, uerr := pvcUploadState(pvcName, handler.log) |
There was a problem hiding this comment.
On upgrade, an app may already be running on this partly uploaded PVC. Its VMIRS restarts the VMI on boot, and KubeVirt 1.7.3 doesn't block a VMI on a plain PVC (that's only for DataVolumes). CDI 1.57.1 only refuses an in-use PVC before its upload pod exists, and the pod from the interrupted upload is likely still around. So this re-drive could write under a live guest, or retry virtctl forever. Could we skip the re-drive while a non-CDI pod uses the claim?
There was a problem hiding this comment.
Done in 908aee7: before re-driving, CreateVolume lists the pods using the PVC (ignoring CDI's own, app=containerized-data-importer, and finished ones) and refuses with a non-transient error naming them, so it neither writes under a live guest nor retries virtctl.
| return pvcName, nil | ||
| } | ||
| handler.log.Noticef("CreateVolume: PVC %s exists but upload not complete, re-driving upload", pvcName) | ||
| handler.log.Noticef("CreateVolume: PVC %s exists but upload not complete (upload state %s), re-driving upload", |
There was a problem hiding this comment.
With this change, an incomplete carried-over EVE-kvm PVC now falls through to RolloutDiskToPVC at line 324 while the PVC exists. That call still passes exists=false, so virtctl runs without --no-create and with --size, which goes against the function's own contract. Lines 262 and 300 pass pvcExists. Should 324 do the same?
There was a problem hiding this comment.
Yes, fixed in 908aee7: that call now passes pvcExists.
| return false, fmt.Errorf("failed to get clientset: %v", err) | ||
| return PVCNoUpload, transientf("failed to get clientset: %v", err) | ||
| } | ||
| pvc, err := clientset.CoreV1().PersistentVolumeClaims(EVEKubeNameSpace). |
There was a problem hiding this comment.
GetPVCUploadState is now also called from Populate and CreateVolume, but its PersistentVolumeClaims().Get still uses context.Background(). Could you give it a context.WithTimeout deadline so a slow apiserver can't block volumemgr on these paths?
There was a problem hiding this comment.
Done in 908aee7: the Get now uses a kubeAPITimeout deadline.
e6c81da to
908aee7
Compare
908aee7 to
15d3304
Compare
On EVE-k, a reboot while CDI is uploading a volume's image leaves the PVC holding part of the data. After the reboot volumemgr adopts PVCs it finds and marked this one created, so the app started on a volume with missing blocks and no error was reported. Decide from the PVC itself whether an upload was started and whether it finished: virtctl marks every PVC it uploads into with CDI's upload target annotation, and CDI records completion in the pod phase. A PVC with an unfinished upload is no longer adopted after a reboot; it goes through the create path, which re-drives the upload into it. The same check now applies to volumes without an image reference, such as one filled from a disk file, when deciding whether an existing PVC can be reused and whether the volume can be reported created. The re-drive is refused while a pod other than CDI's own uses the PVC, since an app activated by an older EVE may already be running on it and CDI does not stop an upload into a claim that is in use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kgg2Z8uxUyj6gY12oHbqxT Signed-off-by: eriknordmark <erik@zededa.com>
15d3304 to
dbe66c8
Compare
Description
On EVE-k, a reboot while CDI is uploading a volume's image leaves a PVC holding part of the data. After the reboot, volumemgr adopts the PVC it finds and reports the volume created, so the app starts on a volume with missing blocks and no error.
The PVC now records whether an upload was started and whether it finished: virtctl puts CDI's
storage.upload.targetannotation on every PVC it uploads into, and CDI records completion in the pod phase. A PVC whose upload is unfinished is not adopted after a reboot; it goes through the create path, which re-drives the upload into the existing PVC. The same check decides whether an existing PVC may be reused and whether the volume can be reported created, including for volumes without an image reference that are filled by an upload.Fixes #6758
How to test and validate this PR
Unit tests:
go test -tags k ./kubeapi/ ./volumehandlers/inpkg/pillar.TestCSIHandler_PopulateRejectsPartialUploadand the no-reference cases ofTestCSIHandler_HandleCreatedRequiresUploadfail without the change.On a device: interrupt a volume's upload with a reboot (an EVE-k app with a multi-GiB volume, reboot while
cdi-upload-serveris running). After the reboot, volumemgr logsPopulate: PVC <name> upload state incompleteandexists but upload not complete (upload state incomplete), re-driving upload, and the volume reaches CREATED only afterimage upload completed.Tested on an amd64 QEMU device (evetest) running this change on top of the kvm→k volume-migration branch (#6658) with kernel 6.18, where the CDI upload of a 2 GiB carried data volume is OOM-killed (#6756) and the oversized error from #6757 made zedmanager fatal, rebooting the device mid-upload. After that reboot volumemgr logged
Populate: PVC <name> upload state incompletefor that PVC andupload state completefor the app's finished image volume, thenexists but upload not complete (upload state incomplete), re-driving upload. The re-driven upload hit the same OOM, so the app stayed out of RUNNING instead of starting on the partial volume. In another run the re-driven upload completed, the app started, and its in-guest verify of the 2 GiB volume was clean. Without this change the same sequence started the app, and its in-guest verify found zeroed blocks.Changelog notes
EVE-k: an app volume whose image upload was interrupted by a reboot is uploaded again instead of being used with missing data.
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Kgg2Z8uxUyj6gY12oHbqxT