Skip to content

GPU Scheduling is failing after deploying GPU Operator #396

Description

@akyriako

Steps to reproduce:

  1. Choose HCE2.0 for the nodes
  2. Deploy CCE AI Suite Add-in
  3. Deploy GPU Operator
  4. Deploy cuda-vectoradd workload

You get: UnexpectedAdmissionError

Name: cuda-vectoradd Namespace: default Priority: 0 Service Account: default Node: 10.10.10.84/ Start Time: Sat, 18 Apr 2026 09:31:59 +0200 Labels: Annotations: cce.io/gpu-decision: {"cuda-vectoradd":{"nvidia0":1000}} Status: Failed Reason: UnexpectedAdmissionError Message: Pod was rejected: Allocate failed due to rpc error: code = Unknown desc = invalid allocation request for "nvidia.com/gpu": invalid request: unknown device: nvidia0, which is unexpected IP: IPs: Containers: cuda-vectoradd: Image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04 Port: Host Port: Limits: nvidia.com/gpu: 1 Requests: nvidia.com/gpu: 1 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8k6b8 (ro) Volumes: kube-api-access-8k6b8: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt Optional: false DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s nvidia.com/gpu=true:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m13s default-scheduler Successfully assigned default/cuda-vectoradd to 10.10.10.84 Warning UnexpectedAdmissionError 3m13s kubelet Allocate failed due to rpc error: code = Unknown desc = invalid allocation request for "nvidia.com/gpu": invalid request: unknown device: nvidia0, which is unexpected

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions