Skip to content

kubeapi: give up on a PVC upload whose CDI upload server crash-loops - #6857

Closed
vignesh-zededa wants to merge 1 commit into
lf-edge:masterfrom
vignesh-zededa:kubeapi-upload-server-failfast
Closed

vignesh-zededa wants to merge 1 commit into
lf-edge:masterfrom
vignesh-zededa:kubeapi-upload-server-failfast

Conversation

@vignesh-zededa

Copy link
Copy Markdown
Contributor

Description

Fixes #6856.

When the CDI upload server dies during CDI's server-side processing, EVE-k reports nothing for hours. #6756 is the current way to hit this: on the 6.18 hwe kernel the upload server is OOM-killed in the qemu-img conversion of a multi-GiB image.

Why it was silent

  • Once virtctl image-upload has sent the data, it waits for processing by polling the PVC's cdi.kubevirt.io/storage.pod.phase annotation for up to 24 h (processingWaitTotal in virtctl v1.7.3), and prints nothing while it waits.
  • The killed upload server's pod restarts and never reaches Succeeded, so virtctl keeps waiting.
  • RolloutDiskToPVC only noticed when its exec timeout fired, which is 1200 + 600 + 120 × GB seconds (about 34 minutes for a 2.7 GiB volume), and then retried up to 10 times.

On a physical node this showed as the volume in INIT with an empty error and the app instance Unknown for 44 minutes before anyone looked at the node. The worst case is several hours. The upload pod showed Restarts: 2, Last State: Terminated, Reason: OOMKilled the whole time.

Change

While each virtctl attempt runs, RolloutDiskToPVC now watches the CDI upload server (new uploadserverwatch.go):

  • Every 15 s it looks up the upload pod, from CDI's cdi.kubevirt.io/storage.uploadPodName annotation on the PVC, falling back to cdi-upload-<pvc>.

  • It counts container restarts since the watch began. Restarts the pod already had belong to earlier attempts and don't count. If CDI replaces the pod, the count carries over.

  • After 3 restarts it cancels the attempt and RolloutDiskToPVC returns a transient error such as:

    PVC Upload for pvc:<pvc> failed: CDI upload server cdi-upload-<pvc> restarted 3 times (last exit: OOMKilled)
    

    It returns immediately rather than spending the remaining tries on a failure that repeats each time. volumemgr's bounded re-drive of transient errors is unchanged, so each re-drive now also fails within minutes, with the reason visible on the volume and app instance.

  • Lookup errors don't trip the watch; the attempt's exec timeout remains the backstop.

This doesn't fix the OOM itself (#6756). It makes any upload-server crash loop visible within minutes instead of hours.

The watcher lives in its own file, so this doesn't conflict with #6760, which edits cdiupload.go.

How to test and validate this PR

Automated, run in CI via make test:

make -C pkg/pillar test TESTPKGS="./kubeapi/... ./volumehandlers/... ./cmd/volumemgr/..."

New tests in uploadserverwatch_test.go (fake clientset) cover:

  • no upload pod;
  • only restarts after the watch began are counted, and the OOMKilled reason is reported;
  • the count carries across a pod replacement;
  • the pod name is taken from the PVC annotation;
  • an OOMKilled termination reason is preferred, with a fallback to the exit code;
  • the watch loop reports once and stops, and stops quietly on cancel.

Manual, on an EVE-k node:

  1. Crash loop. On a master build with the 6.18 hwe kernel (EVE-k on kernel 6.18: CDI upload pod OOM-killed by dirty page cache #6756), deploy an app whose image volume is about 2–3 GiB. Within a few minutes of the upload server's third restart, the volume and app instance must show ... CDI upload server cdi-upload-<pvc> restarted 3 times (last exit: OOMKilled). Before this change they showed no error for hours.
  2. Regression. With MGLRU off (echo 0 > /sys/kernel/mm/lru_gen/enabled), or on a kernel without EVE-k on kernel 6.18: CDI upload pod OOM-killed by dirty page cache #6756, the same app must deploy normally, and the volumemgr log must have no abandoning the upload attempt line.

Changelog notes

App volume uploads on EVE-k that fail inside the CDI upload server now report an error within minutes, instead of leaving the volume stuck without an error for hours.

PR Backports

  • 17.0-stable: No for now; it builds on base.Command.ErrorOutput from kubeapi: carry only virtctl's stderr in a PVC upload error #6776, which was not backported. A backport would need adapting if wanted.
  • 16.0-stable: No, same reason (transientf is also missing there).
  • 14.5-stable: No, as the feature is not available there.
  • 13.4-stable: No, as the feature is not available there.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

Once virtctl image-upload has sent the data, it waits up to 24 h for
CDI's server-side processing, polling the PVC's pod-phase annotation and
printing nothing. When the upload server dies in that phase, for example
OOM-killed in the qemu-img conversion (lf-edge#6756), its pod
restarts and never reaches Succeeded, and virtctl keeps waiting.
RolloutDiskToPVC only noticed when its exec timeout fired, about 34
minutes per attempt for a 2.7 GiB volume, and then retried up to 10
times. The volume sat in INIT with no error for hours, while the pod's
restarts and OOMKilled reason were visible in the API all along.

Watch the upload server while each virtctl attempt runs. Every 15 s,
look up its pod (CDI's uploadPodName annotation on the PVC, or
cdi-upload-<pvc>) and count container restarts since the watch began:
restarts the pod already had belong to earlier attempts, and a pod that
CDI replaces does not reset the count. After 3, cancel the attempt and
return a transient error that names the pod, the count and the last
termination reason, preferring OOMKilled, without spending the remaining
tries on a failure that repeats each time. volumemgr's bounded re-drive
of transient errors is unchanged.

Fixes lf-edge#6856.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: vignesh-zededa <vignesh@zededa.com>
@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.42105% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.43%. Comparing base (d20adfb) to head (09046aa).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/kubeapi/vitoapiserver.go 0.00% 24 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6857      +/-   ##
==========================================
+ Coverage   28.11%   28.43%   +0.32%     
==========================================
  Files         534      546      +12     
  Lines       97719    99530    +1811     
==========================================
+ Hits        27469    28303     +834     
- Misses      68016    68801     +785     
- Partials     2234     2426     +192     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vignesh-zededa

Copy link
Copy Markdown
Contributor Author

Closing: Rene is taking the CDI upload OOM / fail-fast work forward. #6856 stays open to track it.

@vignesh-zededa
vignesh-zededa deleted the kubeapi-upload-server-failfast branch October 19, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EVE-k: a crash-looping CDI upload server leaves the volume silent for hours

1 participant