[dadamsncsa]

Retargeted to main so CI runs on this head. It still contains #177's commits: review and merge #177 first.

Closes #174. Part of #60. Stacked on #177 (the multinode adapter): review and merge that first.

What (infra/delta, plus tests)

1. The GAP-098 pair, on both clusters. FI_CXI_RDZV_THRESHOLD=8192 FI_CXI_RDZV_GET_MIN=0 are added to LLMHUB_VEC_INF_ENV (delta.env) until the images bake them.

   • DeltaAI: with the image's own libfabric settings, cross-node all_reduce returned wrong data on 146-181 of 3,000 calls at 896 KiB-1 MiB. With the pair, 0 (job 3533388).
   • Pipeline-parallel serving puts only send/recv across nodes, which was clean either way, so for serving the pair is defence in depth.
   • Delta was clean without it (GDR 0, 23559724); it carries the pair for uniformity.

2. DeltaAI SLURM_NETWORK=disable_rdzv_get, the site value since 2026-10-22, replaces single_node_vni,disable_rdzv_get.

   • A multi-node step gets one VNI under either value.
   • This reaches every 1-node DeltaAI job too. Without single_node_vni, a 1-node step gets no VNI and creates no CXI service. That is safe because 1-node jobs already run with NCCL_NET_PLUGIN=none and never drive the NIC. The 1-node launch-test after the deploy checks it.

3. A DeltaAI catalog override. config/models-deltaai.yaml is applied by render-models.py through LLMHUB_MODELS_OVERRIDE:

   • five 2-node A100 entries run at 1 node, TP=4, with mem_per_node: 400G. None of the five has been run on GH200 yet;
   • all four Mllama entries (11B and 90B, Vision and Vision-Instruct) and 405B are left out: neither image registers Mllama, and Llama-3.2-11B-Vision-Instruct had resolved to SGLang through vec-inf's packaged catalog, whose image is not on DeltaAI;
   • Qwen3-235B-A22B-Instruct-2507 is added at 2 nodes, TP=4 PP=2, 400G. So far it has served on 2 nodes only from a test harness, with all node memory (3536666, --mem=0, vLLM-step MaxRSS 41 GiB per node). This entry first runs through LLMHub in the acceptance run.

   vec-inf merges a catalog over a base one, its packaged catalog by default, so removed entries would come back at launch. With an override set, the rendered environment.yaml also names the rendered models.yaml as that base (cached_model_config_path, via render-vec-inf-env.py). vec-inf's launch-path load_config() then resolves exactly DeltaAI's 90 entries. Of the entries whose resolution this changes, all but the removed 11B one differ only in sglang_args, which vLLM does not use.

   Override names are checked against the base: remove and models must name an existing entry, and add a new one. Delta renders backend/config/models.yaml as is, with vec-inf's base catalog unchanged.

4. launch-test --nodes N. At N > 1 it also reads the job's own files, as the backend's logs endpoint does:

   • one rank per node;
   • the model loaded on every Worker_PP<k> stage;
   • NCCL on CXI in every worker's debug file (the files Multinode vLLM in one srun step, without Ray (#173) #177 writes), with GDRDMA required on DeltaAI;
   • the pair in every rank's container;
   • 0 libfabric lines before vLLM's ready line.

   After DELETE:

   • sacct says CANCELLED (a kit-side scancel counts as a failure);
   • no node is left down, drained or completing.

   Slurm reads after "ready" can't abort it before the DELETE: an impersonated job is the backend's to cancel.

   The ready budget (the job's walltime) counts only while Slurm runs the job, so a long queue can't use up a big model's load time; pending has its own 4 h cap. The shutdown is timestamped: DELETE, sacct Start and End, and the node states.

   --client SCRIPT runs an acceptance client against the live server before that shutdown. It gets the endpoint and deployment key in its environment, never argv, and its exit status counts.

Tests (at 631f143): 98 passed (T1-T5 + test_kit_multinode_config.py); full suite 285 passed, 1 skipped; lint clean

• T2 now uses each cluster's vec-inf-config, rendered by the kit's own scripts:
  • DeltaAI: 89 one-node entries identical, 1 multinode entry different;
  • Delta: 86 identical, 7 different.
• test_kit_multinode_config.py checks:
  • the pair on both clusters;
  • DeltaAI's SLURM_NETWORK;
  • every catalog line above;
  • that vec-inf 0.9.0's load_config() loads DeltaAI's 90 entries, on the launch path too, where none of the removed names comes back;
  • that a name typo in the override is an error.

LLMHub CI runs only on PRs into main, so it runs here once #177 merges and this PR is retargeted.

On hardware (DeltaAI, 2 × GH200, Qwen2.5-7B-Instruct, TP=4 PP=2)

Job 3541508 passed 14/14. It was launched from a dev session with this branch's rendered DeltaAI config and env, shaped as launch-test --nodes 2 requests it.

• Status: vec-inf RUNNING; all 8 workers.
• Fabric: CXI on 8/8 ranks, GDRDMA.
• Env: the pair in both containers; 0 libfabric lines before ready.
• Auth: 401 without the key, 200 with it.
• Shutdown: after scancel, CANCELLED, with both nodes freed.

The base-catalog rule leaves that job's generated script byte-identical; it changes only which catalog vec-inf can resolve.

launch-test's multinode checks, run on that job's files: all pass. On a copy with the NCCL files removed and a libfabric line planted, 3 fail, as they should.

launch-test --nodes 2 and a 1-node launch-test through the DeltaAI dev stack follow once both branches are deployed there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HnSD7f83LXZjJVqJ3RLAAD