Repository navigation
kubeapi: give up on a PVC upload whose CDI upload server crash-loops - #6857
Closed
vignesh-zededa wants to merge 1 commit into
Closed
vignesh-zededa wants to merge 1 commit into
vignesh-zededa wants to merge 1 commit into
Conversation
Once virtctl image-upload has sent the data, it waits up to 24 h for CDI's server-side processing, polling the PVC's pod-phase annotation and printing nothing. When the upload server dies in that phase, for example OOM-killed in the qemu-img conversion (lf-edge#6756), its pod restarts and never reaches Succeeded, and virtctl keeps waiting. RolloutDiskToPVC only noticed when its exec timeout fired, about 34 minutes per attempt for a 2.7 GiB volume, and then retried up to 10 times. The volume sat in INIT with no error for hours, while the pod's restarts and OOMKilled reason were visible in the API all along. Watch the upload server while each virtctl attempt runs. Every 15 s, look up its pod (CDI's uploadPodName annotation on the PVC, or cdi-upload-<pvc>) and count container restarts since the watch began: restarts the pod already had belong to earlier attempts, and a pod that CDI replaces does not reset the count. After 3, cancel the attempt and return a transient error that names the pod, the count and the last termination reason, preferring OOMKilled, without spending the remaining tries on a failure that repeats each time. volumemgr's bounded re-drive of transient errors is unchanged. Fixes lf-edge#6856. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: vignesh-zededa <vignesh@zededa.com>
github-actions
Bot
requested review from
andrewd-zededa,
eriknordmark and
naiming-zededa
October 19, 2026 10:03
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6857 +/- ##
==========================================
+ Coverage 28.11% 28.43% +0.32%
==========================================
Files 534 546 +12
Lines 97719 99530 +1811
==========================================
+ Hits 27469 28303 +834
- Misses 68016 68801 +785
- Partials 2234 2426 +192 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
Closing: Rene is taking the CDI upload OOM / fail-fast work forward. #6856 stays open to track it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #6856.
When the CDI upload server dies during CDI's server-side processing, EVE-k reports nothing for hours. #6756 is the current way to hit this: on the 6.18
hwekernel the upload server is OOM-killed in theqemu-imgconversion of a multi-GiB image.Why it was silent
virtctl image-uploadhas sent the data, it waits for processing by polling the PVC'scdi.kubevirt.io/storage.pod.phaseannotation for up to 24 h (processingWaitTotalin virtctl v1.7.3), and prints nothing while it waits.Succeeded, so virtctl keeps waiting.RolloutDiskToPVConly noticed when its exec timeout fired, which is1200 + 600 + 120 × GBseconds (about 34 minutes for a 2.7 GiB volume), and then retried up to 10 times.On a physical node this showed as the volume in INIT with an empty error and the app instance Unknown for 44 minutes before anyone looked at the node. The worst case is several hours. The upload pod showed
Restarts: 2, Last State: Terminated, Reason: OOMKilledthe whole time.Change
While each virtctl attempt runs,
RolloutDiskToPVCnow watches the CDI upload server (newuploadserverwatch.go):Every 15 s it looks up the upload pod, from CDI's
cdi.kubevirt.io/storage.uploadPodNameannotation on the PVC, falling back tocdi-upload-<pvc>.It counts container restarts since the watch began. Restarts the pod already had belong to earlier attempts and don't count. If CDI replaces the pod, the count carries over.
After 3 restarts it cancels the attempt and
RolloutDiskToPVCreturns a transient error such as:It returns immediately rather than spending the remaining tries on a failure that repeats each time. volumemgr's bounded re-drive of transient errors is unchanged, so each re-drive now also fails within minutes, with the reason visible on the volume and app instance.
Lookup errors don't trip the watch; the attempt's exec timeout remains the backstop.
This doesn't fix the OOM itself (#6756). It makes any upload-server crash loop visible within minutes instead of hours.
The watcher lives in its own file, so this doesn't conflict with #6760, which edits
cdiupload.go.How to test and validate this PR
Automated, run in CI via
make test:New tests in
uploadserverwatch_test.go(fake clientset) cover:OOMKilledreason is reported;OOMKilledtermination reason is preferred, with a fallback to the exit code;Manual, on an EVE-k node:
hwekernel (EVE-k on kernel 6.18: CDI upload pod OOM-killed by dirty page cache #6756), deploy an app whose image volume is about 2–3 GiB. Within a few minutes of the upload server's third restart, the volume and app instance must show... CDI upload server cdi-upload-<pvc> restarted 3 times (last exit: OOMKilled). Before this change they showed no error for hours.echo 0 > /sys/kernel/mm/lru_gen/enabled), or on a kernel without EVE-k on kernel 6.18: CDI upload pod OOM-killed by dirty page cache #6756, the same app must deploy normally, and the volumemgr log must have noabandoning the upload attemptline.Changelog notes
App volume uploads on EVE-k that fail inside the CDI upload server now report an error within minutes, instead of leaving the volume stuck without an error for hours.
PR Backports
base.Command.ErrorOutputfrom kubeapi: carry only virtctl's stderr in a PVC upload error #6776, which was not backported. A backport would need adapting if wanted.transientfis also missing there).Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
🤖 Generated with Claude Code