[Bug]: Vgpu paused label deadlock issue

[arpittolani]

Describe the bug

If the nvidia-vgpu-device-manager pod is terminated while applying a vGPU config, the nvidia.com/gpu.deploy.sandbox-validator and nvidia.com/gpu.deploy.sandbox-device-plugin node labels are left permanently at paused-for-vgpu-change. Nothing ever restores them, and the cluster ends up in a deadlock that does not self-heal.

The sequence:

1. nvidia-k8s-vgpu-dm calls shutdownGPUOperands(), which rewrites the operand labels to paused-for-vgpu-change (cmd/nvidia-k8s-vgpu-dm/main.go, maybeSetPaused).
2. Restoration is registered as a Go defer calling rescheduleGPUOperands(). If the process receives SIGKILL — pod deleted, node reboot, OOM — the deferred call never runs.
3. The nvidia-sandbox-validator DaemonSet is gated on nodeSelector: nvidia.com/gpu.deploy.sandbox-validator: "true", so no validator pod is scheduled on the node.
4. The validator is the only writer of /run/nvidia/validations/vgpu-manager-ready (gpu-operator/cmd/nvidia-validator/main.go, vGPUManagerStatusFile).
5. nvidia-vgpu-device-manager's init container polls for that file forever, so the device manager never runs — and therefore can never reach the code that would un-pause the labels.

The GPU Operator does not recover from this either. addGPUStateLabels() in controllers/state_manager.go only sets a state label when it is absent or empty:

for key, value := range effective {
    if v, ok := labels[key]; !ok || v == "" {
        labels[key] = value
        modified = true
    }
}

A value of paused-for-vgpu-change is deliberately honored (per the comment above the function, so that k8s-driver-manager pauses are not stomped). The consequence is that a pause orphaned by a killed process is sticky forever.

Net effect: ClusterPolicy reports states not ready: [state-vgpu-device-manager] indefinitely, with no error in any pod log, and no nvidia-sandbox-validator pod present on the affected nodes.

To Reproduce

1. Deploy GPU Operator with sandboxWorkloads.enabled=true, sandboxWorkloads.defaultWorkload=vm-vgpu, vgpuManager.enabled=true, vgpuDeviceManager.enabled=true.
2. Trigger a vGPU config change, e.g. oc label node <node> --overwrite nvidia.com/vgpu.config=L40-8Q.
3. While nvidia-vgpu-device-manager is applying the config, delete the pod: oc delete pod nvidia-vgpu-device-manager-xxxxx -n nvidia-gpu-operator.
4. Observe the labels remain paused and never recover:

$ oc get nodes -L nvidia.com/gpu.deploy.sandbox-validator,nvidia.com/gpu.deploy.sandbox-device-plugin

5. Observe no nvidia-sandbox-validator pod on the node, /run/nvidia/validations/vgpu-manager-ready absent, and the replacement device-manager pod stuck in Init:0/1 indefinitely.

Expected behavior

The paused state should be recoverable without manual intervention. Some options:

• Reconcile the pause on startup: if nvidia-k8s-vgpu-dm starts and finds the labels already at paused-for-vgpu-change with no config change in progress, restore them to true.
• Persist the pre-pause label values (annotation or ConfigMap) so a restarted process can restore the correct prior state rather than relying on in-memory package variables.
• Handle SIGTERM/SIGINT explicitly to run the restore before exiting, rather than relying solely on defer.
• Alternatively, allow the GPU Operator to time out a paused-for-* value and reset it to true when the pausing workload is no longer running.

Environment (please provide the following information):

• NVIDIA Driver Version: 580.105.06 (vGPU Manager)
• Host OS: Red Hat Enterprise Linux CoreOS 9.6.20251013-1
• Kernel Version: 5.14.0-570.54.1.el9_6.x86_64
• Container Runtime Version: cri-o 1.32.9-2.rhaos4.19.git2de10fd.el9
• CPU Architecture: x86_64
• GPU Model(s): NVIDIA L40
• Kubernetes Distro and Version: OpenShift 4.19 (hosted control plane), Kubernetes v1.32.9
• NVIDIA GPU Operator version: v25.10.1

Information to attach

k8s-driver-manager log showing the labels already stuck in the paused state at container start:

time=... level=info msg=Getting current value of the "nvidia.com/gpu.deploy.sandbox-validator" node label
time=... level=info msg=Current value of "nvidia.com/gpu.deploy.sandbox-validator"=paused-for-vgpu-change
time=... level=info msg=Getting current value of the "nvidia.com/gpu.deploy.sandbox-device-plugin" node label
time=... level=info msg=Current value of "nvidia.com/gpu.deploy.sandbox-device-plugin"=paused-for-vgpu-change

Pod list — no nvidia-sandbox-validator present, device manager stuck:

$ oc get pods -n nvidia-gpu-operator
NAME                                                 READY   STATUS     RESTARTS   AGE
gpu-operator-7db9c8f69b-vtwrp                        1/1     Running    0          68d
nvidia-vgpu-device-manager-b2k5t                     0/1     Init:0/1   0          33d
nvidia-vgpu-manager-daemonset-9.6.20251013-1-w46vb   2/2     Running    0          33d

Init container waiting on the marker the (absent) validator would write:

$ oc logs nvidia-vgpu-device-manager-b2k5t -c vgpu-manager-validation
waiting for NVIDIA vGPU Manager to be setup
waiting for NVIDIA vGPU Manager to be setup
...

Host state — driver healthy, VFs enabled, but the validations directory holds only the validator's workload-type file (written earlier in validate(), before the readiness marker step), confirming the validator ran at some point and never completed:

$ ls -la /run/nvidia/validations/
total 4
drwxr-xr-x. 2 root root 80 ...  .
drwxr-xr-x. 4 root root 80 ...  ..
-rw-------. 1 root root  8 ...  workload-type

$ lsmod | grep -iE 'nvidia'
nvidia_vgpu_vfio      159744  12
nvidia              14422016  4 nvidia_vgpu_vfio
mdev                   20480  1 nvidia_vgpu_vfio

ClusterPolicy status:

message: 'ClusterPolicy is not ready, states not ready: [state-vgpu-device-manager]'
reason: OperandNotReady
state: notReady

Workaround

Manually restore the labels:

oc label node <node> --overwrite nvidia.com/gpu.deploy.sandbox-validator=true
oc label node <node> --overwrite nvidia.com/gpu.deploy.sandbox-device-plugin=true

The validator pod then schedules, writes vgpu-manager-ready, and the device manager proceeds normally.