Repository navigation
feat(ER-ORCH-005): Complete IPoIB IPv6 pipeline — DNS-free Slurm IB, minimal discovery mode - #5457
Open
sujit-jadhav wants to merge 8 commits into
Open
sujit-jadhav wants to merge 8 commits into
sujit-jadhav wants to merge 8 commits into
Conversation
sujit-jadhav
requested review from
RvishankarOMnia,
VenkateswaraVatam,
abhishek-sa1 and
snarthan
as code owners
October 6, 2026 08:21
… compat, 221 tests Implements the full IPoIB IPv6 configuration pipeline for Omnia orchestrator, covering allocation validation, NM config rendering, cloud-init publication, Slurm NodeAddr injection, and release evidence. Core pipeline (Stories 1-3): - Versioned allocation export JSON Schema (16 fields, dual-stack) - Allocation validator with prefix, duplicate, and atomicity checks - NM renderer producing nmcli scripts with infiniband.mtu property - SMD component registration and BSS cloud-init publication - Managed /etc/hosts block with IPoIB entries in cloud-init templates - IB interface auto-discovery (predictable naming: ibp10s0, ibp181s0) Slurm integration (Story 3): - inject_ib_nodeaddr.yml reads ib_addr_mode and slurm_preferred_addr_family - Dual-stack requires explicit preference; mismatch emits warning - CommunicationParameters EnableIPv6=yes for ipv6-only/dual-stack Release evidence and NFT (Stories 4-5): - collect_ib_ipv6_evidence module for HCA, switch, OS, IPoIB info - benchmark_ib_ipv6 module for throughput parity measurement - verify_ib_ipv6_state module for address, route, privacy checks Backward compatibility: - Legacy IB_IP CSV header auto-detected and normalized to IB_IPV4 - Legacy subnet/netmask_bits accepted alongside ipv4_subnet/ipv4_prefix - ib_addr_mode defaults to ipv4-only when unset Test automation (438 UT + 40 FVT + 20 NFT): - 6 UT modules covering backward compat, diagnostics, docs quality, NM publication, release evidence, and Slurm NodeAddr - 2 FVT modules for allocation validation and NM publication - 1 NFT module for performance benchmarks Signed-off-by: Sujit Jadhav <sujit_jadhav@dell.com> Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
sujit-jadhav
force-pushed
the
feature/ipv6-ib-config-clean
branch
from
October 6, 2026 08:33
bcb9b82 to
d06bb33
Compare
abhishek-sa1
reviewed
Oct 6, 2026
abhishek-sa1
reviewed
Oct 6, 2026
abhishek-sa1
reviewed
Oct 6, 2026
abhishek-sa1
reviewed
Oct 6, 2026
sujit-jadhav
changed the title
…convenience partitions
Major changes on top of the initial IPoIB IPv6 pipeline:
1. Eliminate /etc/hosts and DNS dependency for Slurm IB communication
- Disable ib_ipv6_manage_hosts (default false)
- Remove IPoIB hosts-block injection from all cloud-init templates
(7 ms-group-*.yaml.j2 files)
- Remove hosts-block loading from configure_metadata_svc.yml
- Slurm uses NodeAddr in slurm.conf for all inter-daemon communication
2. Add dynamic node discovery mode (node_discovery_mode: dynamic)
- Skip iDRAC discovery entirely — generate minimal NodeName entries
- Add dynamic to omnia_config.json schema enum
- Add dynamic mode logic in confs.yml
- Best for large clusters (2000+ nodes) where iDRAC is slow/unnecessary
3. PXE mapping fallback for NodeAddr injection
- inject_ib_nodeaddr.yml now falls back to PXE mapping CSV
(IB_IPV6/IB_IPV4 columns) when IB allocation file is absent
- Enables NodeAddr without running the full ib_ipv6_config pipeline
- Auto-enables CommunicationParameters=EnableIPv6 when IPv6 found
4. Login/compiler-login nodes now get NodeAddr
- build_slurm_conf.yml looks up NodeAddr from PXE mapping or
IB allocation when appending login and compiler-login nodes
- Previously login nodes were appended as bare NodeName entries
5. Fix SIGPIPE bug in configure-ib-network.sh.j2
- set -euo pipefail + awk exit in a dmidecode pipeline caused
silent script death via SIGPIPE (exit code 141)
- Fix: capture dmidecode output into variable before piping to awk
6. Add gpu and cpu convenience partitions
- build_slurm_conf.yml now creates gpu and cpu partitions alongside
the existing normal partition for user convenience
Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
sujit-jadhav
force-pushed
the
feature/ipv6-ib-config-clean
branch
from
October 8, 2026 03:08
de54adb to
f15650a
Compare
1. Revert input-contract.md to upstream (contract override reverted) 2. Move CHANGELOG entries under 2.3.0 (remove separate 2.3.1 section) 3. Restore removed error messages in orchestrator_setup/vars/main.yml 4. Replace oim with localhost in validate_ib_ipv6_release.yml Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
Collaborator
Author
|
All 4 review comments from @abhishek-sa1 addressed in commit 583910f:
|
… source_loader - test_docs_ipv6_quality: Update CHANGELOG version assertion from 2.3.1 to 2.3.0 (entries moved per review), replace Known Limitations check with ConnectX check - test_nm_config_publication_ipv6: Invert ib_hosts_entries assertions to verify templates do NOT contain hosts injection (DNS-free: Slurm uses NodeAddr) - Restore missing ut/source_loader.py and ut/__init__.py for test_address_verifier and test_allocation_validation imports Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
… evidence collection IB interface names use predictable naming based on PCI topology (e.g., ibp10s0, ibp181s0, ibp47s0) not generic ib0/ib1. The validate_ib_ipv6_release.yml playbook now runs discover_ib_interfaces.sh before benchmarks and evidence collection to find actual interface names. - Add discover_ib_interfaces.sh: lists /sys/class/net/ib* with ib0 fallback - validate_ib_ipv6_release.yml: use ansible.builtin.script for discovery before throughput parity benchmark and evidence collection tasks Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
…ork_spec.yml Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
… IB to os/default groups - Rename node_discovery_mode: dynamic -> minimal in omnia_config.yml, omnia_config.json schema, and confs.yml runtime logic to avoid confusion with Slurm native dynamic nodes (slurmd -Z) - Add configure-ib-network.sh to os_x86_64, os_aarch64, and default_x86_64 cloud-init templates so all functional groups get IB IPv4/IPv6 assigned during PXE boot Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
sujit-jadhav
changed the title
…eading claims - Replace hardcoded ib0 with dynamic discovery () in ipv6-infiniband-configuration.md and troubleshooting.md - Add predictable interface naming note (ibp161s0, not ib0) - Remove misleading claim about Slurm/MPI dependency on IB IPv6 - Add Known Limitation item for predictable interface naming - Update Last updated to October 2026 Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(ER-ORCH-005): Complete IPoIB IPv6 pipeline — DNS-free Slurm IB, minimal discovery mode
Author: Sujit Jadhav (sujit.jadhav@dell.com)
Summary
Complete IPoIB IPv6 pipeline with minimal node discovery mode, IB IP assignment via cloud-init, and documentation updates.
Changes
1. IB IPv6/IPv4 assignment via cloud-init
configure-ib-network.sh.j2assigns IB IPv4 and IPv6 addresses on all functional groupsset -euo pipefail+awk '{ exit }'in dmidecode pipeline caused silent script deathos_x86_64,os_aarch64, anddefault_x86_64templates (previously only Slurm/K8s groups)2. Minimal node discovery mode (
node_discovery_mode: minimal)dynamictominimalto avoid confusion with Slurm native dynamic nodes (slurmd -Z)NodeNameentriesAutoDetect: nvml3. NodeAddr injection from PXE mapping
inject_ib_nodeaddr.ymlfalls back to PXE mapping CSV (IB_IPV6/IB_IPV4columns) when IB allocation file is absentCommunicationParameters=EnableIPv6when IPv6 addresses are found4. IB interface auto-discovery
discover_ib_interfaces.shto detect actual IB interface names (ibp161s0,ibp181s0, etc.)ib05. Documentation updates
ipv6-infiniband-configuration.mdwith predictable interface naming notestroubleshooting.mdto use dynamic IB interface discovery instead of hardcodedib06. Other changes
slurm_preferred_addr_family: "ipv6"innetwork_spec.ymlgpuandcpuconvenience partitions inbuild_slurm_conf.ymlib_ipv6_manage_hosts(defaultfalse) — removed hosts-block injectionFiles Changed
input/omnia_config.ymlminimaldiscovery mode docs + setnode_discovery_mode: minimalinput/network_spec.ymlslurm_preferred_addr_family: "ipv6"schema/omnia_config.json"minimal"to enum (renamed from"dynamic")ib_ipv6_config/defaults/main.ymlib_ipv6_manage_hosts: falseib_ipv6_config/tasks/main.yml/etc/hostsupdate when disabledprovision_common/tasks/configure_metadata_svc.ymldoca-ofed/configure-ib-network.sh.j2ms-group-*.yaml.j2ib_hosts_entries, add IB script to os/default groupsslurm_config/tasks/build_slurm_conf.ymlslurm_config/tasks/confs.ymlslurm_config/tasks/inject_ib_nodeaddr.ymlscripts/discover_ib_interfaces.shvalidate_ib_ipv6_release.ymloimwithlocalhost, use discovered interfacesdocs/ipv6-infiniband-configuration.mddocs/troubleshooting.mdTest Validation
Test Plan
normal,gpu,cpupartitionsCommunicationParameters=EnableIPv6active in slurmctldNodeAddruses IB IPv6 addresses in slurm.conf