[dadamsncsa]

Closes #173. Part of #60. Upstream: VectorInstitute/vector-inference#333.

Why

vec-inf 0.9.0 launches a num_nodes > 1 vLLM job as a Ray cluster: one srun step for the head, one per worker, and vllm serve outside all of them. On DeltaAI that fails four ways, each found by fixing the one before it:

1. hostname --ip-address returns 18 addresses, link-local IPv6 first (3286271).
2. The --containall PID namespace hides the head's raylet from the driver (3286313).
3. The head port lands in Ray's reserved range about 1 launch in 6 (3286364).
4. Every step gets its own Slingshot VNI, so ranks in different steps can't reach each other over CXI (3286451, 3286467).

On Delta, (2) fails too (21500825), so the Ray path serves on neither cluster, and nothing that works is replaced here.

What

For multinode vLLM only, the call site (llm_inference.py) now builds MultinodeLauncher, a ModelLauncher subclass whose generator replaces vec-inf's Ray branch (app/utils/vec_inf_multinode.py):

• One step, one task per node. One srun spanning every node, one task per node, --kill-on-bad-exit=1.
• The rank script, written into the job directory, does four things on each node:
  • finds the node's hsn<digits> address (never DNS);
  • waits on a barrier for every rank;
  • sets VLLM_HOST_IP, NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME for that rank;
  • execs the same container vec-inf would.
• Ranks and rendezvous. Rank 0 runs vllm serve with the API; the other ranks run --headless. They join through vLLM's own --nnodes/--node-rank/--master-addr/--master-port with the mp executor.
• vLLM runs under env -u FI_LOG_LEVEL. The image bakes warn, and libfabric's benign Failed to … lines made vec-inf mark a serving 2-node job FAILED (3533520: 1,284 lines).
• What stays vec-inf's. The route is chosen on vec-inf's resolved params, so the catalog's node count decides. Single-node and non-vLLM launches go to client.launch_model unchanged. Status, shutdown, the job JSON and the log layout are all vec-inf's.
• Fails closed. A multinode launch on any vec-inf other than 0.9.0 is refused; it never falls back to Ray.

The adapter is temporary. The same launcher is proposed upstream (vector-inference#333), and the adapter is deleted at the pin bump once a vec-inf release ships it. Until then, T1 keeps a bump from silently routing multinode back to Ray, and T5 names every vec-inf member the adapter depends on.

Behaviour changes for multinode jobs:

• NCCL INFO (INIT,NET) is written to nccl-debug/<host>.<pid> in the job directory, never to stderr. This records each rank's transport. It applies unless the job env or the host sets a level, and a file they name is kept.
• VLLM_HOST_IP, NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME and VLLM_PORT from the deployment or request env are left out of the containers' env, and the job log says so. Each rank sets its own; as --env flags they would override that on every rank.

Scope of the evidence: hardware-tested on DeltaAI only. The adapter also takes Delta's multinode vLLM entries: six at 2 nodes, and Llama-3.1-405B at 8. Delta's 2-node run is milestone 2 of #60. Until then a Delta multinode launch uses code that has run only on GH200, in place of a Ray path that fails there.

Contract tests (tests/test_multinode_t*.py): 84 passed at 93aeeeb

Test	Checks	Fix removed → result
T1	version guard fails closed; nothing submitted	check_vec_inf_version() deleted → 2 failed
T2	every 1-node catalog entry renders the same script with and without the adapter (86 per cluster), and every multinode vLLM entry does not (7 per cluster: six 2-node entries and Llama-3.1-405B at 8 nodes). The eighth multinode entry, Llama-3.2-90B-Vision-Instruct, resolves to SGLang through vec-inf's packaged catalog and stays on vec-inf's path	route everything to the adapter → 2 failed; route nothing → 2 failed
T3	one srun, --kill-on-bad-exit=1, no ray, mp, native flags, --headless, env -u FI_LOG_LEVEL, rank script in the job dir, deployment env on every rank, NCCL debug never on stderr	--kill-on-bad-exit=1 dropped → 1 failed
T4	the rank script run with stubbed ip/hostname/apptainer: rank → flags, master from hsn<digits>, refusal without a fabric IPv4, barrier timeout reported FAILED by vec-inf's own scan	hsn matched by prefix → 15 failed
T5	every vec-inf member the adapter uses or overrides still exists and is still called	no fix to remove: a pytest plugin renames _generate_launch_cmd, as a vec-inf bump would → 1 failed

Full suite: 271 passed, 1 skipped. black, isort and flake8 at CI's pins: clean. Removal records: builds/llmhub/multinode-build-2026-11/test-results/deltaai/pr1-gate/ in llm-service-software.

Removal records (93aeeeb)

T1 @ 93aeeeb: MultinodeLauncher no longer checks the vec-inf version

@@ -384,7 +384,6 @@
 
     def __init__(self, model_name: str, kwargs: Optional[Dict[str, Any]]):
         # The generator refuses anything but multinode vLLM, before sbatch.
-        check_vec_inf_version()
         super().__init__(model_name, kwargs)
 
     def _build_launch_command(self) -> str:

pytest tests/test_multinode_t1_version_guard.py 
FAILED tests/test_multinode_t1_version_guard.py::test_unknown_version_submits_nothing_for_multinode
FAILED tests/test_multinode_t1_version_guard.py::test_launcher_itself_refuses_an_unknown_version
2 failed, 9 passed in 5.01s

T2-routing @ 93aeeeb: the dispatch sends every launch to the adapter, single-node included

@@ -409,6 +409,6 @@
     # consumes (env, vllm_args), so each launcher gets its own copy.
     kwargs = {k: v for k, v in vars(options).items() if v is not None}
     resolved = ModelLauncher(model_name, dict(kwargs)).params
-    if not is_multinode_vllm(resolved):
+    if False:
         return client.launch_model(model_name, options=options)
     return MultinodeLauncher(model_name, dict(kwargs)).launch()

pytest tests/test_multinode_t2_single_node_untouched.py 
FAILED tests/test_multinode_t2_single_node_untouched.py::test_every_single_node_entry_renders_the_same_script[delta]
FAILED tests/test_multinode_t2_single_node_untouched.py::test_every_single_node_entry_renders_the_same_script[deltaai]
2 failed, 2 passed in 55.40s

T2-no-adapter @ 93aeeeb: the dispatch sends every launch to stock vec-inf, multinode included

@@ -409,6 +409,6 @@
     # consumes (env, vllm_args), so each launcher gets its own copy.
     kwargs = {k: v for k, v in vars(options).items() if v is not None}
     resolved = ModelLauncher(model_name, dict(kwargs)).params
-    if not is_multinode_vllm(resolved):
+    if True:
         return client.launch_model(model_name, options=options)
     return MultinodeLauncher(model_name, dict(kwargs)).launch()

pytest tests/test_multinode_t2_single_node_untouched.py 
FAILED tests/test_multinode_t2_single_node_untouched.py::test_every_multinode_vllm_entry_renders_the_adapter[delta]
FAILED tests/test_multinode_t2_single_node_untouched.py::test_every_multinode_vllm_entry_renders_the_adapter[deltaai]
2 failed, 2 passed in 52.62s

T3 @ 93aeeeb: srun loses --kill-on-bad-exit=1

@@ -369,7 +369,7 @@
                 'export LLMHUB_MN_API_PORT="$server_port_number"',
                 f"export LLMHUB_MN_SERVED_NAME={shlex.quote(self.params['model_name'])}",
                 'srun --nodes="$SLURM_JOB_NUM_NODES" --ntasks="$SLURM_JOB_NUM_NODES" \\',
-                "    --ntasks-per-node=1 --kill-on-bad-exit=1 \\",
+                "    --ntasks-per-node=1 \\",
                 '    bash "$rank_script"',
             ]
         )

pytest tests/test_multinode_t3_shape.py 
FAILED tests/test_multinode_t3_shape.py::test_one_srun_step_spans_every_node
1 failed, 11 passed in 2.10s

T4 @ 93aeeeb: fabric interfaces matched by prefix instead of exact hsn

@@ -112,8 +112,8 @@
 
 # The first hsn<digits> IPv4, matched exactly: never DNS, which returns the VLAN
 # address on Delta, several records on DeltaAI and link-local for the FQDN.
-fabric_ipv4() { ip -4 -o addr show | awk '$2 ~ /^hsn[0-9]+$/ {split($4, a, "/"); print a[1]; exit}'; }
-fabric_ifaces() { ip -4 -o addr show | awk '$2 ~ /^hsn[0-9]+$/ {printf "%s%s", s, $2; s=","}'; }
+fabric_ipv4() { ip -4 -o addr show | awk '$2 ~ /^hsn/ {split($4, a, "/"); print a[1]; exit}'; }
+fabric_ifaces() { ip -4 -o addr show | awk '$2 ~ /^hsn/ {printf "%s%s", s, $2; s=","}'; }
 
 if [ "${1:-}" = "--master" ]; then
     addr="$(fabric_ipv4)"

pytest tests/test_multinode_t4_rank_script.py 
FAILED tests/test_multinode_t4_rank_script.py::test_rank_flags - AssertionErr...
FAILED tests/test_multinode_t4_rank_script.py::test_each_rank_gets_its_own_fabric_identity
FAILED tests/test_multinode_t4_rank_script.py::test_deployment_env_and_devices_reach_every_rank
FAILED tests/test_multinode_t4_rank_script.py::test_master_is_the_first_hsn_digits_address
FAILED tests/test_multinode_t4_rank_script.py::test_refuses_without_a_fabric_ipv4[args0-None]
FAILED tests/test_multinode_t4_rank_script.py::test_refuses_without_a_fabric_ipv4[args1-0]
FAILED tests/test_multinode_t4_rank_script.py::test_rank_zero_must_hold_the_master_address
FAILED tests/test_multinode_t4_rank_script.py::test_barrier_gives_up_and_vec_inf_reports_it
FAILED tests/test_multinode_t4_rank_script.py::test_nccl_debug_goes_to_a_file[-host0-INFO-INIT,NET-job]
FAILED tests/test_multinode_t4_rank_script.py::test_nccl_debug_goes_to_a_file[-host1-WARN-None-job]
FAILED tests/test_multinode_t4_rank_script.py::test_nccl_debug_goes_to_a_file[,NCCL_DEBUG=WARN-host2-WARN-None-job]
FAILED tests/test_multinode_t4_rank_script.py::test_nccl_debug_goes_to_a_file[-host3-INFO-INIT,NET-/x/%h]
FAILED tests/test_multinode_t4_rank_script.py::test_nccl_debug_goes_to_a_file[,NCCL_DEBUG_FILE=/x/%h-host4-INFO-INIT,NET-/x/%h]
FAILED tests/test_multinode_t4_rank_script.py::test_nccl_debug_goes_to_a_file[,NCCL_DEBUG=WARN,NCCL_DEBUG_FILE=/x/%h-host5-WARN-None-/x/%h]
FAILED tests/test_multinode_t4_rank_script.py::test_refuses_a_step_without_gpus
15 failed, 1 passed in 17.45s

T5 @ 93aeeeb: no code change: vec-inf renames SlurmScriptGenerator._generate_launch_cmd (pytest plugin seam_renamed.py)

pytest tests/test_multinode_t5_seam.py -p seam_renamed
FAILED tests/test_multinode_t5_seam.py::test_member_exists[SlurmScriptGenerator._generate_launch_cmd]
1 failed, 40 passed in 0.62s

On hardware (DeltaAI, 2 × GH200, Qwen2.5-7B-Instruct TP=4 PP=2)

Submitted through LLMInferenceDirectClient.launch_model from a dev session; Slurm's stored batch script is byte-identical to the generated one.

Job 3541300, at this PR's last code commit (93aeeeb), passed 14/14 on gh[102,127]:

• Status: vec-inf RUNNING; LLMHub status READY.
• Workers: all 8, Worker_PP0_TP0..3 and Worker_PP1_TP0..3.
• Fabric: Selected provider is cxi on all 8 ranks; via NET/Libfabric/0..3/GDRDMA.
• Env: the GAP-098 pair in both ranks' containers; 0 libfabric: lines before ready (3533520, with the image's level, had 1,284). The pair came from the run's own VEC_INF_ENV: the kit adds it to every job in infra/delta: multinode config, DeltaAI catalog, launch-test --nodes (#174) #178, and this PR alone doesn't.
• Auth: 401 without the key, 200 with it.
• Shutdown: after scancel, CANCELLED. Both nodes were released: another user's jobs started on gh102 39 s and on gh127 98 s after this one ended (sacct).

Job 3541160, at 8b2d695, passed the same 14 checks before the last review fixes. Evidence: builds/llmhub/multinode-build-2026-11/ in llm-service-software.

CI on this PR, at e0be734: backend 271 passed, 1 skipped; changelog and every other check pass (run 42292508778, backend job 128149342014).

Not in this PR

• Kit config, DeltaAI catalog and launch-test --nodes (Kit: multinode config for Delta and DeltaAI (fabric env, DeltaAI catalog, 2-node launch-test) #174, stacked on this branch).
• Request validation (An explicit num_gpus/num_nodes request replaces the model's parallel layout from models.yaml #112).
• DeltaAI multi-GPU launches falsely reported failed, and then cannot be shut down #101's shutdown half.
• The images (infra(delta): vLLM image recipes for Delta and DeltaAI #163).

🤖 Generated with Claude Code

https://claude.ai/code/session_01HnSD7f83LXZjJVqJ3RLAAD