[Bug]: Failed to allocate device vector A (error code CUDA-capable device(s) is/are busy or unavailable)!

[ctrought]

Describe the bug

Related: #1546

After upgrading OpenShift from 4.18 to 4.20 + nvidia operator the drivers no longer installed succesfully (auto upgrade, unpinned). After pinning the driver version back to 580.X the driver install completes successfully for the V100 GPUs, but the validator pods fail to complete with the error blow. Attempted changing kernelModuleType from auto to proprietary per case #1546 but did not seem to have any effect.

$ oc logs nvidia-cuda-validator-rl8qc -c cuda-validation
Failed to allocate device vector A (error code CUDA-capable device(s) is/are busy or unavailable)!
[Vector addition of 50000 elements]

$ oc exec nvidia-driver-daemonset-9.6.20260804-0-6d4fn -c nvidia-driver-ctr -- nvidia-smi
Tue Sep  1 16:29:59 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.178.04             Driver Version: 580.178.04     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla V100-PCIE-32GB           On  |   00000000:5E:00.0 Off |                    0 |
| N/A   32C    P0             24W /  250W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

To Reproduce
V100 GPUs, OpenShift 4.20, NVIDIA operator 26.7.0, 580.X driver, cluster policy provided below.

Expected behavior
Post driver install, nvidia validator pods should complete successfully.

Environment (please provide the following information):

• GPU Operator Version: 26.7.0
• OS: CoreOS 9.6
• Kernel Version: 5.14.0-570.132.1.el9_6.x86_64
• Kubernetes Distro and Version: OpenShift 4.20.33

$ oc get ds
NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                                                                  AGE
gpu-feature-discovery                     3         3         3       3            3           nvidia.com/gpu.deploy.gpu-feature-discovery=true                                                               113d
nvidia-container-toolkit-daemonset        3         3         3       3            3           nvidia.com/gpu.deploy.container-toolkit=true                                                                   113d
nvidia-dcgm                               3         3         3       3            3           nvidia.com/gpu.deploy.dcgm=true                                                                                113d
nvidia-dcgm-exporter                      3         3         3       3            3           nvidia.com/gpu.deploy.dcgm-exporter=true                                                                       113d
nvidia-device-plugin-daemonset            3         3         3       3            3           nvidia.com/gpu.deploy.device-plugin=true                                                                       113d
nvidia-device-plugin-mps-control-daemon   0         0         0       0            0           nvidia.com/gpu.deploy.device-plugin=true,nvidia.com/mps.capable=true                                           113d
nvidia-driver-daemonset-9.6.20260804-0    3         3         3       2            3           feature.node.kubernetes.io/system-os_release.OSTREE_VERSION=9.6.20260804-0,nvidia.com/gpu.deploy.driver=true   12d
nvidia-mig-manager                        0         0         0       0            0           nvidia.com/gpu.deploy.mig-manager=true                                                                         113d
nvidia-node-status-exporter               3         3         3       3            3           nvidia.com/gpu.deploy.node-status-exporter=true                                                                113d
nvidia-operator-validator                 3         3         0       3            0           nvidia.com/gpu.deploy.operator-validator=true                                                                  113d

$ oc get pods
NAME                                           READY   STATUS                  RESTARTS          AGE
console-plugin-nvidia-gpu-bb6b54f46-kn4vs      1/1     Running                 0                 3d15h
gpu-feature-discovery-24bj5                    1/1     Running                 5 (17h ago)       17h
gpu-feature-discovery-wr5hv                    1/1     Running                 5 (111m ago)      112m
gpu-feature-discovery-xvcz7                    1/1     Running                 5 (128m ago)      129m
gpu-operator-cbfc69446-lhgr9                   1/1     Running                 0                 10d
nvidia-container-toolkit-daemonset-2crzs       1/1     Running                 0                 17h
nvidia-container-toolkit-daemonset-ftsh7       1/1     Running                 0                 129m
nvidia-container-toolkit-daemonset-mtt88       1/1     Running                 0                 112m
nvidia-cuda-validator-chkg7                    0/1     Init:CrashLoopBackOff   6 (118s ago)      7m49s
nvidia-cuda-validator-f78jc                    0/1     Init:CrashLoopBackOff   6 (3m32s ago)     9m16s
nvidia-cuda-validator-qqqpq                    0/1     Init:Error              2 (22s ago)       26s
nvidia-dcgm-7qnhx                              1/1     Running                 0                 112m
nvidia-dcgm-87ht5                              1/1     Running                 0                 17h
nvidia-dcgm-exporter-ckpth                     1/1     Running                 5 (127m ago)      129m
nvidia-dcgm-exporter-gc82g                     1/1     Running                 5 (17h ago)       17h
nvidia-dcgm-exporter-nqx48                     1/1     Running                 5 (111m ago)      112m
nvidia-dcgm-s9smw                              1/1     Running                 0                 129m
nvidia-device-plugin-daemonset-fn7qx           1/1     Running                 0                 17h
nvidia-device-plugin-daemonset-pm4c8           1/1     Running                 0                 129m
nvidia-device-plugin-daemonset-zchqx           1/1     Running                 0                 112m
nvidia-driver-daemonset-9.6.20260804-0-6d4fn   2/2     Running                 0                 129m
nvidia-driver-daemonset-9.6.20260804-0-qlpmm   2/2     Running                 0                 113m
nvidia-driver-daemonset-9.6.20260804-0-v4gh2   2/2     Running                 0                 17h
nvidia-node-status-exporter-4m4ll              1/1     Running                 0                 10d
nvidia-node-status-exporter-8dnnp              1/1     Running                 0                 10d
nvidia-node-status-exporter-plgk4              1/1     Running                 0                 10d
nvidia-operator-validator-866bd                0/1     Init:CrashLoopBackOff   18 (4m15s ago)    129m
nvidia-operator-validator-qnjzn                0/1     Init:CrashLoopBackOff   156 (2m47s ago)   17h
nvidia-operator-validator-wjmbc                0/1     Init:2/4                17 (118s ago)     112m


apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  creationTimestamp: "2026-05-11T13:59:15Z"
  generation: 5
  name: gpu-cluster-policy
  resourceVersion: "21268861508"
  uid: f617b135-dbdf-49c6-be28-73d0e9a34fb2
spec:
  ccManager:
    enabled: true
  cdi:
    default: false
    enabled: true
    nriPluginEnabled: false
  daemonsets:
    rollingUpdate:
      maxUnavailable: "1"
    updateStrategy: RollingUpdate
  dcgm:
    enabled: true
  dcgmExporter:
    config:
      name: ""
    enabled: true
    serviceMonitor:
      enabled: true
  devicePlugin:
    config:
      default: ""
      name: ""
    enabled: true
    mps:
      root: /run/nvidia/mps
  driver:
    certConfig:
      name: ""
    enabled: true
    image: driver
    kernelModuleConfig:
      name: ""
    kernelModuleType: proprietary
    licensingConfig:
      nlsEnabled: true
      secretName: ""
    repoConfig:
      configMapName: ""
    repository: nvcr.io/nvidia
    upgradePolicy:
      autoUpgrade: true
      drain:
        deleteEmptyDir: false
        enable: false
        force: false
        timeoutSeconds: 300
      maxParallelUpgrades: 1
      maxUnavailable: 25%
      podDeletion:
        deleteEmptyDir: false
        force: false
        timeoutSeconds: 300
      waitForCompletion:
        timeoutSeconds: 0
    useNvidiaDriverCRD: false
    version: 580.178.04
    virtualTopology:
      config: ""
  gdrcopy:
    enabled: false
  gds:
    enabled: false
  gfd:
    enabled: true
  kataSandboxDevicePlugin:
    enabled: true
  mig:
    strategy: single
  migManager:
    enabled: true
  nodeStatusExporter:
    enabled: true
  operator:
    runtimeClass: nvidia
    use_ocp_driver_toolkit: true
  sandboxDevicePlugin:
    enabled: true
  sandboxWorkloads:
    defaultWorkload: container
    enabled: false
    mode: kubevirt
  toolkit:
    enabled: true
    installDir: /usr/local/nvidia
  validator:
    plugin:
      env: []
  vfioManager:
    enabled: true
  vgpuDeviceManager:
    enabled: true
  vgpuManager:
    enabled: false
status:
  conditions:
  - lastTransitionTime: "2026-08-21T22:53:43Z"
    message: ""
    reason: Error
    status: "False"
    type: Ready
  - lastTransitionTime: "2026-08-21T22:53:43Z"
    message: 'ClusterPolicy is not ready; states not ready: [state-driver state-operator-validation]'
    reason: OperandNotReady
    status: "True"
    type: Error
  namespace: nvidia-gpu-operator
  state: notReady


$ oc describe node gpu-worker | grep -i nvidia.com/gpu
                    nvidia.com/gpu-driver-upgrade-state=upgrade-done
                    nvidia.com/gpu.compute.major=7
                    nvidia.com/gpu.compute.minor=0
                    nvidia.com/gpu.count=1
                    nvidia.com/gpu.deploy.client=true
                    nvidia.com/gpu.deploy.container-toolkit=true
                    nvidia.com/gpu.deploy.dcgm=true
                    nvidia.com/gpu.deploy.dcgm-exporter=true
                    nvidia.com/gpu.deploy.device-plugin=true
                    nvidia.com/gpu.deploy.driver=true
                    nvidia.com/gpu.deploy.gpu-feature-discovery=true
                    nvidia.com/gpu.deploy.node-status-exporter=true
                    nvidia.com/gpu.deploy.nvsm=
                    nvidia.com/gpu.deploy.operator-validator=true
                    nvidia.com/gpu.family=volta
                    nvidia.com/gpu.machine=UCSC-C240-M5SX
                    nvidia.com/gpu.memory=32768
                    nvidia.com/gpu.mode=compute
                    nvidia.com/gpu.present=true
                    nvidia.com/gpu.product=Tesla-V100-PCIE-32GB
                    nvidia.com/gpu.replicas=1
                    nvidia.com/gpu.sharing-strategy=none
                    nvidia.com/gpu.workload.config=container
                    nvidia.com/gpu-driver-upgrade-enabled: true
  nvidia.com/gpu:     1
  nvidia.com/gpu:     1
  nvidia.com/gpu     0                   0 <---- (allocated request/limit on this particular node)

I threw it to AI and this was its assessment. Setting with_workload under validator.cuda.env to false did allow bypassing this error, but I am wondering if that expected or what the implications are given this was not required before.

Recommended Resolution

This appears to be a v26.7.0 validator packaging regression because NVIDIA still lists V100 on OpenShift as supported.

Raise an NVIDIA support case with the missing sm_70 evidence. The practical workaround is to disable only the built-in CUDA workload validation using validator.cuda.env.WITH_WORKLOAD=false, then validate the GPUs separately with a CUDA 12.x image compiled for sm_70. Changing the R580 driver alone will not fix this validator binary.

After change:

$ oc get pods
NAME                                           READY   STATUS    RESTARTS       AGE
console-plugin-nvidia-gpu-bb6b54f46-kn4vs      1/1     Running   0              3d15h
gpu-feature-discovery-24bj5                    1/1     Running   5 (18h ago)    18h
gpu-feature-discovery-wr5hv                    1/1     Running   5 (135m ago)   136m
gpu-feature-discovery-xvcz7                    1/1     Running   5 (151m ago)   153m
gpu-operator-cbfc69446-lhgr9                   1/1     Running   0              10d
nvidia-container-toolkit-daemonset-2crzs       1/1     Running   0              18h
nvidia-container-toolkit-daemonset-ftsh7       1/1     Running   0              153m
nvidia-container-toolkit-daemonset-mtt88       1/1     Running   0              136m
nvidia-dcgm-7qnhx                              1/1     Running   0              136m
nvidia-dcgm-87ht5                              1/1     Running   0              18h
nvidia-dcgm-exporter-ckpth                     1/1     Running   5 (151m ago)   153m
nvidia-dcgm-exporter-gc82g                     1/1     Running   5 (18h ago)    18h
nvidia-dcgm-exporter-nqx48                     1/1     Running   5 (135m ago)   136m
nvidia-dcgm-s9smw                              1/1     Running   0              153m
nvidia-device-plugin-daemonset-fn7qx           1/1     Running   0              18h
nvidia-device-plugin-daemonset-pm4c8           1/1     Running   0              153m
nvidia-device-plugin-daemonset-zchqx           1/1     Running   0              136m
nvidia-driver-daemonset-9.6.20260804-0-6d4fn   2/2     Running   0              153m
nvidia-driver-daemonset-9.6.20260804-0-qlpmm   2/2     Running   0              136m
nvidia-driver-daemonset-9.6.20260804-0-v4gh2   2/2     Running   0              18h
nvidia-node-status-exporter-4m4ll              1/1     Running   0              10d
nvidia-node-status-exporter-8dnnp              1/1     Running   0              10d
nvidia-node-status-exporter-plgk4              1/1     Running   0              10d
nvidia-operator-validator-62hbr                1/1     Running   0              4m39s
nvidia-operator-validator-67d6k                1/1     Running   0              4m39s
nvidia-operator-validator-rhdk7                1/1     Running   0              4m40s

[CPPavithra]

Looks like this comes from a commit which bumped CUDA_SAMPLES_VERSION from 12.9 to 13.2 in docker/Dockerfile. The sample's arch list dropped sm_70 in the process:

• v12.9: 50 52 60 61 70 72 75 80
• v13.2: 75 80 86 87 89 90 100 110 120
  Two options-
• also build the sample with 12.9 and pick at runtime by compute capability, so Volta still gets validated (a plain revert to 12.9 would lose sm_110)
• or skip the workload validation when the GPU is below the sample's minimum CC
  I would like to take this up, i am leaning towards the first. Will open a PR soon.

[rajathagasthya]

Thanks for the analysis @CPPavithra! GPU Operator 26.7.0 bumped the validator's CUDA sample from 12.9 to 13.2 (#2620), and CUDA 13 removed support for GPUs older than Turing, so the vectorAdd check can no longer run on Volta.

We will figure out what the right next steps are and provide an update here.