From 592dba7b3f89e936ed05e00e13716acb16788a4c Mon Sep 17 00:00:00 2001 From: Wojciech Rymer Date: Wed, 7 Oct 2026 13:13:53 +0200 Subject: [PATCH 1/3] helm chart update --- cookbooks/cosmos3/nim/README.md | 2 +- cookbooks/cosmos3/nim/deployment.md | 4 +- cookbooks/cosmos3/nim/helm.md | 190 +++++++++++++++++++++++++++- 3 files changed, 189 insertions(+), 7 deletions(-) diff --git a/cookbooks/cosmos3/nim/README.md b/cookbooks/cosmos3/nim/README.md index 28a2c814..feb08a97 100644 --- a/cookbooks/cosmos3/nim/README.md +++ b/cookbooks/cosmos3/nim/README.md @@ -159,7 +159,7 @@ pages and examples as their source of truth and default to read-only assistance. - [Deployment](deployment.md) - [Configuration](configuration.md) - [Support matrix](support-matrix.md) -- [Helm deployment status](helm.md) +- [Helm deployment](helm.md) - [Bring your own checkpoint](bring-your-own-checkpoint.md) ### Use the APIs diff --git a/cookbooks/cosmos3/nim/deployment.md b/cookbooks/cosmos3/nim/deployment.md index c3a4bb71..0021a9da 100644 --- a/cookbooks/cosmos3/nim/deployment.md +++ b/cookbooks/cosmos3/nim/deployment.md @@ -79,8 +79,8 @@ requires an explicit precision. Nano-DROID currently has BF16 profiles only. ## Before you deploy Verify the host against [Prerequisites](prerequisites.md) and choose a -compatible configuration from the [Support matrix](support-matrix.md). For the -current availability of Helm guidance, see [Helm deployment](helm.md). +compatible configuration from the [Support matrix](support-matrix.md). For a +Kubernetes deployment, follow [Deploy with Helm](helm.md). ## Authenticate to NGC diff --git a/cookbooks/cosmos3/nim/helm.md b/cookbooks/cosmos3/nim/helm.md index a524ad74..d023f8cc 100644 --- a/cookbooks/cosmos3/nim/helm.md +++ b/cookbooks/cosmos3/nim/helm.md @@ -1,8 +1,190 @@ -# Cosmos3 Certified NIM Helm deployment +# Deploy the Cosmos3 Certified NIM with Helm -Helm deployment guidance is not included in this cookbook yet. Do not adapt a -chart or values file from another NIM. Use the documented -[Docker deployment](deployment.md) for the current Cosmos3 Certified NIM image. +Use this page to deploy the Cosmos3 Certified NIM on Kubernetes with the shared +`nim-wfm` Helm chart. It follows the NVIDIA +[Deploy with Helm](https://docs.nvidia.com/nim/cosmos/latest/helm.html) guide +with Cosmos3 image and runtime settings. Model selection, hardware floors, and +verification are the same as in [Deployment](deployment.md). + +## Prerequisites + +- A Kubernetes cluster with GPU nodes that match a row in the + [Support matrix](support-matrix.md), with the + [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/) + installed. +- Helm and `kubectl` access to the target namespace. The commands below use the + default namespace. +- A storage class for the persistent model cache. +- `NGC_API_KEY` exported as described in + [Network and NGC access](prerequisites.md#network-and-ngc-access). + +## Create secrets + +The chart needs an image pull secret and a generic secret with a key named +`NGC_API_KEY`: + +```bash +kubectl create secret docker-registry ngc-secret \ + --docker-server=nvcr.io \ + --docker-username='$oauthtoken' \ + --docker-password="$NGC_API_KEY" +kubectl create secret generic ngc-api \ + --from-literal=NGC_API_KEY="$NGC_API_KEY" +``` + +## Fetch the chart + +Choose a `nim-wfm` chart version in the +[NGC Catalog](https://catalog.ngc.nvidia.com/) and download it: + +```bash +export NIM_WFM_VERSION='' +helm fetch "https://helm.ngc.nvidia.com/nim/charts/nim-wfm-${NIM_WFM_VERSION}.tgz" \ + --username='$oauthtoken' \ + --password="$NGC_API_KEY" +``` + +The chart README lists every value supported by that version: + +```bash +helm show readme "nim-wfm-${NIM_WFM_VERSION}.tgz" +helm show values "nim-wfm-${NIM_WFM_VERSION}.tgz" +``` + +## Configure values + +Save the following as `cosmos3-values.yaml`. It deploys the Nano Generator, +optimized for latency, on one GPU: + +```yaml +image: + repository: nvcr.io/nim/nvidia/cosmos3 + tag: "2.0.0" +imagePullSecrets: + - name: ngc-secret +model: + ngcAPISecret: ngc-api + nimCache: /opt/nim/.cache +persistence: + enabled: true + size: +resources: + limits: + nvidia.com/gpu: 1 +env: + - name: NIM_MODEL_TYPE + value: generator + - name: NIM_MODEL_VARIANT + value: nano + - name: NIM_PERF_PROFILE + value: latency +``` + +Adjust it for the chosen configuration: + +- **Model and GPUs:** Change `NIM_MODEL_VARIANT`, `NIM_PERF_PROFILE`, and + `nvidia.com/gpu` together using the + [Generator configurations](support-matrix.md#generator-configurations). The + GPU count does not select a GPU type; use `nodeSelector`, `affinity`, or + `tolerations` to target eligible nodes. +- **Cache size:** Replace `` with a volume size for the selected + model artifacts and materialization. No single floor is published. +- **Memory:** A pod memory limit counts as system memory during profile + selection. Keep any limit at or above the host RAM value of the selected row. +- **Shared memory:** The Docker launch allocates 16 GiB of `/dev/shm`. If + `helm template` output shows no `/dev/shm` mount, add a memory-backed + `emptyDir` through `extraVolumes` and `extraVolumeMounts`. +- **Chart-owned settings:** Use `model.nimCache`, `model.ngcAPISecret`, + `model.apiPort`, `model.jsonLogging`, and `model.logLevel` instead of + `NIM_CACHE_PATH`, `NGC_API_KEY`, `NIM_HTTP_API_PORT`, `NIM_LOGGING_JSONL`, + and `NIM_LOG_LEVEL`. Add other [Configuration](configuration.md) variables to + `env`. + +To check profile compatibility before a cold download, run the Docker +[pre-download profile preflight](deployment.md#run-the-pre-download-profile-preflight) +on a host with the same GPUs and selectors. + +### Reasoner + +For Reasoner, replace `env` with the Reasoner selectors and choose the GPU +count from the [Reasoner configurations](support-matrix.md#reasoner-configurations). +Reasoner does not use `NIM_PERF_PROFILE`: + +```yaml +env: + - name: NIM_MODEL_TYPE + value: reasoner + - name: NIM_MODEL_VARIANT + value: nano +``` + +On DGX Spark/GB10 or Jetson AGX Thor nodes, also set +`NIM_GPU_MEMORY_UTILIZATION` to `"0.80"` for image-only workloads or `"0.70"` +for video or mixed-media workloads; see +[Set the Reasoner memory share](deployment.md#set-the-reasoner-memory-share-on-unified-memory-systems). + +## Install + +Review the rendered manifests, then install the release: + +```bash +helm template cosmos3 "nim-wfm-${NIM_WFM_VERSION}.tgz" -f cosmos3-values.yaml +helm install cosmos3 "nim-wfm-${NIM_WFM_VERSION}.tgz" -f cosmos3-values.yaml +kubectl get pods -w +``` + +Cold download, materialization, and model load can take much longer than pod +startup. Send requests only after the pod reports `Ready`. + +## Verify + +The chart creates a `ClusterIP` Service with no ingress, and the NIM does not +handle authentication. Port-forward the Service for testing: + +```bash +kubectl port-forward service/cosmos3-nim-wfm 8000:http-api +``` + +In another terminal: + +```bash +export NIM_URL=http://localhost:8000 +curl -fsS "$NIM_URL/v1/health/ready" +curl -fsS "$NIM_URL/v1/metadata" | python3 -m json.tool +``` + +Confirm that metadata reports `generator` and `/v1/infer`, or `reasoner` and +`/v1/chat/completions`. Then run a first request from +[Use an existing NIM endpoint](README.md#use-an-existing-nim-endpoint). + +## Storage and scaling + +With `persistence.enabled` and the default `statefulSet.enabled: true`, each +replica gets its own persistent volume claim and downloads the model before it +becomes ready. To share one cache across replicas, use a `ReadWriteMany` +storage class and set `persistence.accessMode: ReadWriteMany`. With +`statefulSet.enabled: false` and a `ReadWriteOnce` volume, scaling beyond one +pod is likely to fail. The chart also supports `nfs` and `hostPath`; +`hostPath` ties pods to one node and has security implications. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Pod stays `Pending` | Run `kubectl describe pod ` and check `Events` for insufficient GPUs, untolerated taints, or an unbound volume claim | +| Pod restarts while preparing the model workspace | The startup probe ran out before the download finished; increase `startupProbe.failureThreshold` | +| Profile selection fails | Compare GPU type, count, free memory, and pod memory limit with the [Support matrix](support-matrix.md); see [Troubleshooting](operations.md#troubleshooting) | + +## Remove the deployment + +```bash +helm uninstall cosmos3 +kubectl get pvc +``` + +StatefulSet volume claims are retained by default and keep the model cache. +Delete them and the `ngc-secret` and `ngc-api` secrets only when they are no +longer needed. From bcf60dd62a9ac6d1cc6116c21ccfc99b62e86367 Mon Sep 17 00:00:00 2001 From: Wojciech Rymer Date: Wed, 7 Oct 2026 12:54:43 +0000 Subject: [PATCH 2/3] docs: update Cosmos3 NIM Helm guide for nim-wfm 1.3.0 Pin the nim-wfm chart to 1.3.0 and fill in values validated by a deployment of the Nano Generator: cache size, shared-memory limit, pod user, and the new gRPC/inference-protocol chart-owned settings. Carry over content from the docs.nvidia.com Helm page being retired: OpenTelemetry env, ServiceMonitor/ingress/autoscaling pointers, storage options table, and the StatefulSet-disabled scaling troubleshooting row. Co-Authored-By: Claude Opus 5.5 --- cookbooks/cosmos3/nim/helm.md | 104 ++++++++++++++++++++++++++-------- 1 file changed, 80 insertions(+), 24 deletions(-) diff --git a/cookbooks/cosmos3/nim/helm.md b/cookbooks/cosmos3/nim/helm.md index d023f8cc..5dc80d32 100644 --- a/cookbooks/cosmos3/nim/helm.md +++ b/cookbooks/cosmos3/nim/helm.md @@ -4,10 +4,8 @@ SPDX-License-Identifier: OpenMDW-1.1 --> # Deploy the Cosmos3 Certified NIM with Helm Use this page to deploy the Cosmos3 Certified NIM on Kubernetes with the shared -`nim-wfm` Helm chart. It follows the NVIDIA -[Deploy with Helm](https://docs.nvidia.com/nim/cosmos/latest/helm.html) guide -with Cosmos3 image and runtime settings. Model selection, hardware floors, and -verification are the same as in [Deployment](deployment.md). +`nim-wfm` Helm chart. Model selection, hardware floors, and verification are +the same as in [Deployment](deployment.md). ## Prerequisites @@ -37,11 +35,10 @@ kubectl create secret generic ngc-api \ ## Fetch the chart -Choose a `nim-wfm` chart version in the -[NGC Catalog](https://catalog.ngc.nvidia.com/) and download it: +This page uses `nim-wfm` chart version 1.3.0: ```bash -export NIM_WFM_VERSION='' +export NIM_WFM_VERSION='1.3.0' helm fetch "https://helm.ngc.nvidia.com/nim/charts/nim-wfm-${NIM_WFM_VERSION}.tgz" \ --username='$oauthtoken' \ --password="$NGC_API_KEY" @@ -70,7 +67,9 @@ model: nimCache: /opt/nim/.cache persistence: enabled: true - size: + size: 200Gi +sharedMemory: + sizeLimit: 16Gi resources: limits: nvidia.com/gpu: 1 @@ -90,18 +89,25 @@ Adjust it for the chosen configuration: [Generator configurations](support-matrix.md#generator-configurations). The GPU count does not select a GPU type; use `nodeSelector`, `affinity`, or `tolerations` to target eligible nodes. -- **Cache size:** Replace `` with a volume size for the selected - model artifacts and materialization. No single floor is published. +- **Cache size:** The Nano Generator cache uses about 22 GiB after download + and materialization; `200Gi` leaves room for other variants and profile + changes. No single floor is published, so size the volume for the selected + model with headroom. - **Memory:** A pod memory limit counts as system memory during profile selection. Keep any limit at or above the host RAM value of the selected row. -- **Shared memory:** The Docker launch allocates 16 GiB of `/dev/shm`. If - `helm template` output shows no `/dev/shm` mount, add a memory-backed - `emptyDir` through `extraVolumes` and `extraVolumeMounts`. +- **Shared memory:** The chart mounts a memory-backed `emptyDir` at `/dev/shm` + by default. `sharedMemory.sizeLimit: 16Gi` matches the 16 GiB that the Docker + launch allocates. +- **Pod user:** The chart runs the pod as UID and GID 1000 with + `fsGroup: 1000` (`podSecurityContext`). The cache volume must be writable by + that user; storage that ignores `fsGroup`, such as some NFS exports and + `hostPath` directories, needs matching ownership. - **Chart-owned settings:** Use `model.nimCache`, `model.ngcAPISecret`, - `model.apiPort`, `model.jsonLogging`, and `model.logLevel` instead of - `NIM_CACHE_PATH`, `NGC_API_KEY`, `NIM_HTTP_API_PORT`, `NIM_LOGGING_JSONL`, - and `NIM_LOG_LEVEL`. Add other [Configuration](configuration.md) variables to - `env`. + `model.apiPort`, `model.grpcPort`, `model.inferenceProtocol`, + `model.jsonLogging`, and `model.logLevel` instead of `NIM_CACHE_PATH`, + `NGC_API_KEY`, `NIM_HTTP_API_PORT`, `NIM_GRPC_API_PORT`, + `NIM_INFERENCE_PROTOCOL`, `NIM_LOGGING_JSONL`, and `NIM_LOG_LEVEL`. Add other + [Configuration](configuration.md) variables to `env`. To check profile compatibility before a cold download, run the Docker [pre-download profile preflight](deployment.md#run-the-pre-download-profile-preflight) @@ -126,6 +132,48 @@ On DGX Spark/GB10 or Jetson AGX Thor nodes, also set for video or mixed-media workloads; see [Set the Reasoner memory share](deployment.md#set-the-reasoner-memory-share-on-unified-memory-systems). +### OpenTelemetry + +To export traces and metrics over OTLP, add these entries to `env`. The +endpoint assumes an OpenTelemetry Collector DaemonSet that listens on host +port 4318; change `OTEL_EXPORTER_OTLP_ENDPOINT` for other collector layouts: + +```yaml + - name: NIM_ENABLE_OTEL + value: "1" + - name: OTEL_SERVICE_NAME + value: cosmos3 + - name: OTEL_TRACES_EXPORTER + value: otlp + - name: OTEL_METRICS_EXPORTER + value: otlp + - name: HOST_IP + valueFrom: + fieldRef: + fieldPath: status.hostIP + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: "http://$(HOST_IP):4318" +``` + +`HOST_IP` must come before `OTEL_EXPORTER_OTLP_ENDPOINT` so that Kubernetes +can expand it. + +### Other chart features + +The chart README documents these optional features; all are off by default: + +- **Prometheus metrics:** Set `metrics.serviceMonitor.enabled: true` to create + a Prometheus Operator `ServiceMonitor` that scrapes `/v1/metrics` on the + `http-api` port. See [Metrics](operations.md#metrics). +- **Ingress:** `ingress.enabled`, `ingress.className`, `ingress.hosts`, and + `ingress.tls`. The NIM does not authenticate requests, so protect any + ingress outside the cluster. +- **Autoscaling:** `autoscaling.enabled` with `minReplicas`, `maxReplicas`, and + `metrics`. CPU and memory metrics are of limited use for scaling a NIM; use + custom metrics, for example through `prometheus-adapter`. Each new replica + must download or mount the model before it becomes ready; see + [Storage and scaling](#storage-and-scaling). + ## Install Review the rendered manifests, then install the release: @@ -162,13 +210,20 @@ Confirm that metadata reports `generator` and `/v1/infer`, or `reasoner` and ## Storage and scaling -With `persistence.enabled` and the default `statefulSet.enabled: true`, each -replica gets its own persistent volume claim and downloads the model before it -becomes ready. To share one cache across replicas, use a `ReadWriteMany` -storage class and set `persistence.accessMode: ReadWriteMany`. With -`statefulSet.enabled: false` and a `ReadWriteOnce` volume, scaling beyond one -pod is likely to fail. The chart also supports `nfs` and `hostPath`; -`hostPath` ties pods to one node and has security implications. +The chart supports these cache options; use only one: + +| Option | Values | Notes | +| --- | --- | --- | +| StatefulSet volume claims | `persistence.enabled: true` with the default `statefulSet.enabled: true` | Default. Each replica gets its own claim and downloads the model before it becomes ready | +| Shared volume claim | `persistence.enabled: true`, `statefulSet.enabled: false`, `persistence.accessMode: ReadWriteMany` | One cache shared by all replicas; needs a `ReadWriteMany` storage class | +| Existing claim | `persistence.existingClaim: ` | Reuse a pre-filled cache. Run one replica unless the claim is `ReadWriteMany` | +| Direct NFS | `nfs.enabled: true`, `nfs.server`, `nfs.path` | Mount options cannot be set per pod; an NFS-backed claim is usually a better choice | +| `hostPath` | `hostPath.enabled: true`, `hostPath.path` | Ties pods to one node and has security implications | + +With `statefulSet.enabled: false` and a `ReadWriteOnce` volume, scaling or +rolling upgrades beyond one pod are likely to fail. Use a `ReadWriteMany` +storage class, manually cloned `ReadOnlyMany` volumes of a pre-filled cache, +or direct NFS. Without persistence, every pod start downloads the model again. ## Troubleshooting @@ -176,6 +231,7 @@ pod is likely to fail. The chart also supports `nfs` and `hostPath`; | --- | --- | | Pod stays `Pending` | Run `kubectl describe pod ` and check `Events` for insufficient GPUs, untolerated taints, or an unbound volume claim | | Pod restarts while preparing the model workspace | The startup probe ran out before the download finished; increase `startupProbe.failureThreshold` | +| Scaling or upgrade fails with `statefulSet.enabled: false` | The cache volume is `ReadWriteOnce`; see [Storage and scaling](#storage-and-scaling) | | Profile selection fails | Compare GPU type, count, free memory, and pod memory limit with the [Support matrix](support-matrix.md); see [Troubleshooting](operations.md#troubleshooting) | ## Remove the deployment From 41bae2ccb3be78d44730f16a706a2630667b4c7a Mon Sep 17 00:00:00 2001 From: Wojciech Rymer Date: Tue, 13 Oct 2026 12:47:37 +0200 Subject: [PATCH 3/3] docs: tighten Cosmos3 NIM Helm guide and route Kubernetes users to it Carry over the remaining content from the retiring docs.nvidia.com Helm page: the 220Gi cache example, storage class, MIG resources, startup probe budget, replicaCount, and StatefulSet cache pre-population. Drop the gRPC and inference-protocol chart values that the Cosmos3 NIM does not document, replace the unmeasured cache-size claim, correct the tolerations wording, and list chart-owned settings in a table. Point the bundled assistant skill at helm.md for Kubernetes deployments. --- .../.agents/skills/cosmos3-nim-user/SKILL.md | 6 ++- cookbooks/cosmos3/nim/helm.md | 45 ++++++++++++------- 2 files changed, 34 insertions(+), 17 deletions(-) diff --git a/cookbooks/cosmos3/nim/.agents/skills/cosmos3-nim-user/SKILL.md b/cookbooks/cosmos3/nim/.agents/skills/cosmos3-nim-user/SKILL.md index 84df4088..ed187ea6 100644 --- a/cookbooks/cosmos3/nim/.agents/skills/cosmos3-nim-user/SKILL.md +++ b/cookbooks/cosmos3/nim/.agents/skills/cosmos3-nim-user/SKILL.md @@ -58,11 +58,13 @@ Guide the customer through the canonical pages in this order: Transfer eligibility. 3. `deployment.md` for the exact image, authentication, pre-download profile preflight, cache, runtime selector, launch, readiness, logs, and cleanup. + For Kubernetes, use `helm.md` for the chart, secrets, values file, and + verification. 4. `configuration.md` only when a documented non-default setting is needed. 5. The existing-endpoint preflight above before the first inference request. -Use only the exact image and commands in `deployment.md`; never substitute -`latest`. After the image pull, run the documented pre-download profile +Use only the exact image and commands in `deployment.md` or, for Kubernetes, +`helm.md`; never substitute `latest`. After the image pull, run the documented pre-download profile preflight with the intended selectors and GPUs. For a Reasoner on DGX Spark/GB10 or Jetson AGX Thor, every preflight and launch command shown to the customer, starting with the first one, must include diff --git a/cookbooks/cosmos3/nim/helm.md b/cookbooks/cosmos3/nim/helm.md index 5dc80d32..809bc85e 100644 --- a/cookbooks/cosmos3/nim/helm.md +++ b/cookbooks/cosmos3/nim/helm.md @@ -67,7 +67,7 @@ model: nimCache: /opt/nim/.cache persistence: enabled: true - size: 200Gi + size: 220Gi sharedMemory: sizeLimit: 16Gi resources: @@ -87,12 +87,12 @@ Adjust it for the chosen configuration: - **Model and GPUs:** Change `NIM_MODEL_VARIANT`, `NIM_PERF_PROFILE`, and `nvidia.com/gpu` together using the [Generator configurations](support-matrix.md#generator-configurations). The - GPU count does not select a GPU type; use `nodeSelector`, `affinity`, or - `tolerations` to target eligible nodes. -- **Cache size:** The Nano Generator cache uses about 22 GiB after download - and materialization; `200Gi` leaves room for other variants and profile - changes. No single floor is published, so size the volume for the selected - model with headroom. + GPU count does not select a GPU type; use `nodeSelector` or `affinity` to + target eligible GPU nodes, and `tolerations` if those nodes are tainted. On + nodes with MIG enabled, request the MIG resource instead of + `nvidia.com/gpu`, for example `nvidia.com/mig-4g.24gb: 2`. +- **Cache size:** `220Gi` is a starting size for large model files. Set `persistence.storageClass` if the + cluster has no default storage class. - **Memory:** A pod memory limit counts as system memory during profile selection. Keep any limit at or above the host RAM value of the selected row. - **Shared memory:** The chart mounts a memory-backed `emptyDir` at `/dev/shm` @@ -102,12 +102,18 @@ Adjust it for the chosen configuration: `fsGroup: 1000` (`podSecurityContext`). The cache volume must be writable by that user; storage that ignores `fsGroup`, such as some NFS exports and `hostPath` directories, needs matching ownership. -- **Chart-owned settings:** Use `model.nimCache`, `model.ngcAPISecret`, - `model.apiPort`, `model.grpcPort`, `model.inferenceProtocol`, - `model.jsonLogging`, and `model.logLevel` instead of `NIM_CACHE_PATH`, - `NGC_API_KEY`, `NIM_HTTP_API_PORT`, `NIM_GRPC_API_PORT`, - `NIM_INFERENCE_PROTOCOL`, `NIM_LOGGING_JSONL`, and `NIM_LOG_LEVEL`. Add other - [Configuration](configuration.md) variables to `env`. + +Set these through chart values rather than `env`: + +| Chart value | Replaces | +| --- | --- | +| `model.nimCache` | `NIM_CACHE_PATH` | +| `model.ngcAPISecret` | `NGC_API_KEY` | +| `model.apiPort` | `NIM_HTTP_API_PORT` | +| `model.jsonLogging` | `NIM_LOGGING_JSONL` | +| `model.logLevel` | `NIM_LOG_LEVEL` | + +Add other [Configuration](configuration.md) variables to `env`. To check profile compatibility before a cold download, run the Docker [pre-download profile preflight](deployment.md#run-the-pre-download-profile-preflight) @@ -185,7 +191,10 @@ kubectl get pods -w ``` Cold download, materialization, and model load can take much longer than pod -startup. Send requests only after the pod reports `Ready`. +startup. Send requests only after the pod reports `Ready`. Kubernetes restarts +the container if it is not ready within `startupProbe.initialDelaySeconds` plus +`startupProbe.failureThreshold` × `startupProbe.periodSeconds`; raise +`failureThreshold` for slow first downloads. ## Verify @@ -220,10 +229,16 @@ The chart supports these cache options; use only one: | Direct NFS | `nfs.enabled: true`, `nfs.server`, `nfs.path` | Mount options cannot be set per pod; an NFS-backed claim is usually a better choice | | `hostPath` | `hostPath.enabled: true`, `hostPath.path` | Ties pods to one node and has security implications | +Set `replicaCount` for a fixed number of replicas. With the default +StatefulSet, you can pre-populate caches by scaling up to the maximum replica +count once and then scaling down; the retained claims let later scale-ups +start without a new download. + With `statefulSet.enabled: false` and a `ReadWriteOnce` volume, scaling or rolling upgrades beyond one pod are likely to fail. Use a `ReadWriteMany` storage class, manually cloned `ReadOnlyMany` volumes of a pre-filled cache, -or direct NFS. Without persistence, every pod start downloads the model again. +or direct NFS. Without persistence, the cache is an `emptyDir` and every pod +start downloads the model again. ## Troubleshooting