Skip to content

volumehandlers: don't adopt a partly uploaded PVC - #6760

Merged
eriknordmark merged 1 commit into
lf-edge:masterfrom
eriknordmark:pvc-partial-upload
Oct 30, 2026
Merged

eriknordmark merged 1 commit into
lf-edge:masterfrom
eriknordmark:pvc-partial-upload

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

On EVE-k, a reboot while CDI is uploading a volume's image leaves a PVC holding part of the data. After the reboot, volumemgr adopts the PVC it finds and reports the volume created, so the app starts on a volume with missing blocks and no error.

The PVC now records whether an upload was started and whether it finished: virtctl puts CDI's storage.upload.target annotation on every PVC it uploads into, and CDI records completion in the pod phase. A PVC whose upload is unfinished is not adopted after a reboot; it goes through the create path, which re-drives the upload into the existing PVC. The same check decides whether an existing PVC may be reused and whether the volume can be reported created, including for volumes without an image reference that are filled by an upload.

Fixes #6758

How to test and validate this PR

Unit tests: go test -tags k ./kubeapi/ ./volumehandlers/ in pkg/pillar. TestCSIHandler_PopulateRejectsPartialUpload and the no-reference cases of TestCSIHandler_HandleCreatedRequiresUpload fail without the change.

On a device: interrupt a volume's upload with a reboot (an EVE-k app with a multi-GiB volume, reboot while cdi-upload-server is running). After the reboot, volumemgr logs Populate: PVC <name> upload state incomplete and exists but upload not complete (upload state incomplete), re-driving upload, and the volume reaches CREATED only after image upload completed.

Tested on an amd64 QEMU device (evetest) running this change on top of the kvm→k volume-migration branch (#6658) with kernel 6.18, where the CDI upload of a 2 GiB carried data volume is OOM-killed (#6756) and the oversized error from #6757 made zedmanager fatal, rebooting the device mid-upload. After that reboot volumemgr logged Populate: PVC <name> upload state incomplete for that PVC and upload state complete for the app's finished image volume, then exists but upload not complete (upload state incomplete), re-driving upload. The re-driven upload hit the same OOM, so the app stayed out of RUNNING instead of starting on the partial volume. In another run the re-driven upload completed, the app started, and its in-guest verify of the 2 GiB volume was clean. Without this change the same sequence started the app, and its in-guest verify found zeroed blocks.

Changelog notes

EVE-k: an app volume whose image upload was interrupted by a reboot is uploaded again instead of being used with missing data.

PR Backports

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kgg2Z8uxUyj6gY12oHbqxT

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 56.25000% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.82%. Comparing base (02ad7ce) to head (dbe66c8).
⚠️ Report is 13 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/kubeapi/cdiupload.go 43.47% 26 Missing ⚠️
pkg/pillar/volumehandlers/csihandler.go 73.52% 7 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6760      +/-   ##
==========================================
+ Coverage   28.48%   28.82%   +0.34%     
==========================================
  Files         540      551      +11     
  Lines       98258   100063    +1805     
==========================================
+ Hits        27990    28846     +856     
- Misses      67998    68747     +749     
- Partials     2270     2470     +200     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread pkg/pillar/volumehandlers/csihandler.go Resolved
Comment thread pkg/pillar/kubeapi/cdiupload.go Resolved
@milan-zededa

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @milan-zededa's task in 2m 4s —— View job


Answers to the two review questions

Q1 — csihandler.go:126: why check handler.status.ReferenceName when the remaining state is PVCNoUpload?

You're right that if the state is neither PVCUploadIncomplete nor PVCUploadComplete it is PVCNoUpload. The ReferenceName check is exactly what gives PVCNoUpload its two different meanings on this path:

  • ReferenceName != "" (image volume): PVCNoUpload means the image was never uploaded, so the PVC has no content → must not be reported created → the condition fires (error).
  • ReferenceName == "" (blank CreatePVC volume): PVCNoUpload is the normal, expected end state → should be reported created → the condition must not fire.

So (handler.status.ReferenceName != "" && state != kubeapi.PVCUploadComplete) is what rejects a PVCNoUpload image volume while leaving a blank volume alone. Without the ReferenceName term you'd have to either wrongly reject every blank volume or wrongly accept an image volume that was never uploaded. The check is needed and correct. (It also matters for the transient-error case: on error GetPVCUploadState returns PVCNoUpload, and the ReferenceName == "" branch intentionally lets a blank volume report created rather than be parked on a transient API error.)

Q2 — cdiupload.go:137: pre-upgrade PVCs without the new annotation classify as PVCNoUpload.

I posted a detailed inline reply on that line. Short version: these are CDI/virtctl annotations, not EVE-written, and a completed upload still carries storage.pod.phase: Succeeded, so it classifies as PVCUploadComplete even without storage.upload.target — completed uploads are safe. The genuine risk is a once-completed image PVC that has since lost both annotations (CDI garbage-collection), which would then be rejected in HandleCreated / needlessly re-driven in CreateVolume. Worth confirming the eve-k CDI config keeps storage.pod.phase on completed PVCs. See the inline comment for the specifics.


The rest of the change looks sound — the PVCUploadIncomplete / PVCNoUpload / PVCUploadComplete split is applied consistently across Populate, CreateVolume, and HandleCreated, and the tests cover the reference / no-reference × state matrix.
· branch pvc-partial-upload

Comment thread pkg/pillar/kubeapi/cdiupload.go Resolved
if handler.status.ReferenceName == "" {
handler.log.Noticef("CreateVolume: PVC %s already exists, skipping creation", pvcName)
return pvcName, nil
state, uerr := pvcUploadState(pvcName, handler.log)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On upgrade, an app may already be running on this partly uploaded PVC. Its VMIRS restarts the VMI on boot, and KubeVirt 1.7.3 doesn't block a VMI on a plain PVC (that's only for DataVolumes). CDI 1.57.1 only refuses an in-use PVC before its upload pod exists, and the pod from the interrupted upload is likely still around. So this re-drive could write under a live guest, or retry virtctl forever. Could we skip the re-drive while a non-CDI pod uses the claim?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 908aee7: before re-driving, CreateVolume lists the pods using the PVC (ignoring CDI's own, app=containerized-data-importer, and finished ones) and refuses with a non-transient error naming them, so it neither writes under a live guest nor retries virtctl.

return pvcName, nil
}
handler.log.Noticef("CreateVolume: PVC %s exists but upload not complete, re-driving upload", pvcName)
handler.log.Noticef("CreateVolume: PVC %s exists but upload not complete (upload state %s), re-driving upload",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change, an incomplete carried-over EVE-kvm PVC now falls through to RolloutDiskToPVC at line 324 while the PVC exists. That call still passes exists=false, so virtctl runs without --no-create and with --size, which goes against the function's own contract. Lines 262 and 300 pass pvcExists. Should 324 do the same?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, fixed in 908aee7: that call now passes pvcExists.

return false, fmt.Errorf("failed to get clientset: %v", err)
return PVCNoUpload, transientf("failed to get clientset: %v", err)
}
pvc, err := clientset.CoreV1().PersistentVolumeClaims(EVEKubeNameSpace).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetPVCUploadState is now also called from Populate and CreateVolume, but its PersistentVolumeClaims().Get still uses context.Background(). Could you give it a context.WithTimeout deadline so a slow apiserver can't block volumemgr on these paths?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 908aee7: the Get now uses a kubeAPITimeout deadline.

@andrewd-zededa andrewd-zededa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

On EVE-k, a reboot while CDI is uploading a volume's image leaves the
PVC holding part of the data. After the reboot volumemgr adopts PVCs it
finds and marked this one created, so the app started on a volume with
missing blocks and no error was reported.

Decide from the PVC itself whether an upload was started and whether it
finished: virtctl marks every PVC it uploads into with CDI's upload
target annotation, and CDI records completion in the pod phase. A PVC
with an unfinished upload is no longer adopted after a reboot; it goes
through the create path, which re-drives the upload into it. The same
check now applies to volumes without an image reference, such as one
filled from a disk file, when deciding whether an existing PVC can be
reused and whether the volume can be reported created.

The re-drive is refused while a pod other than CDI's own uses the PVC,
since an app activated by an older EVE may already be running on it and
CDI does not stop an upload into a claim that is in use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kgg2Z8uxUyj6gY12oHbqxT
Signed-off-by: eriknordmark <erik@zededa.com>

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lf-edge lf-edge deleted a comment from github-actions Bot Oct 29, 2026
@lf-edge lf-edge deleted a comment from github-actions Bot Oct 29, 2026
@eriknordmark
eriknordmark merged commit c8c3ab4 into lf-edge:master Oct 30, 2026
87 of 95 checks passed
@eriknordmark
eriknordmark deleted the pvc-partial-upload branch October 30, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EVE-k: partly uploaded PVC used as complete after a reboot

4 participants