Repository navigation
Upstream contrib/opx 12.4 - #8
Closed
avinciguerra07 wants to merge 39 commits into
Closed
avinciguerra07 wants to merge 39 commits into
avinciguerra07 wants to merge 39 commits into
Conversation
Add the FI_PEER owner infrastructure that lets OPX import the shm (smr)
provider as an intra-node peer, modeled on the EFA provider. This commit
covers the complete owner-side receive path: peer setup, matching,
completion delivery, peek/claim/discard, and FI_MULTI_RECV.
At endpoint enable OPX opens a nested shm fabric/domain/endpoint and
imports it as a peer:
- A hand-rolled owner shared receive context (struct opx_srx_ctx /
opx_srx_rx_entry) provides the fi_ops_srx_owner callbacks
(get_msg/get_tag/queue_msg/queue_tag/free_entry/foreach_unspec_addr).
Incoming shm sends are matched directly against the posted receives
OPX already keeps on opx_ep->rx->queue[kind].mq using the real
fi_addr (shm reports it via FI_AV_USER_ID); unmatched sends park on
a per-kind unexpected list until a later receive drains them.
- A hand-rolled owner peer CQ (struct fid_peer_cq with owner_ops
write/writeerr) deposits shm completions into OPX's own fi_opx_cq.
The shm CQ is opened FI_PEER with an fi_peer_cq_context, and the shm
peer is progressed by reading it with count 0 from the OPX receive
poll.
Peek/claim/discard also consult the shm peer. The owner-side
FI_PEEK / FI_CLAIM / FI_DISCARD receive semantics previously only scanned
OPX's native unexpected packet queue, so messages the shm peer parked on
the owner SRX unexpected list were invisible to a peeking or claiming
receive:
- fi_opx_ep_peek_shm_unexp() mirrors the fabric peek: on an FI_PEEK it
reports the queued message (len/tag/data) without consuming it; with
FI_CLAIM it removes the wrapper and stashes it in the fi_context for a
later stand-alone claim; with FI_DISCARD it calls the shm peer's
discard_{tag,msg} to release the command context, frees the owner
wrapper, and writes the completion. It is called on a fabric-queue
miss before the FI_ENOMSG error path.
- The stand-alone FI_CLAIM path distinguishes a shm-claimed entry from a
fabric unexpected packet via fi_context internal[1] (set to the
srx_ctx for shm, NULL for fabric) and dispatches a claimed shm entry
to the shm peer's start_{tag,msg}; the claimed entry always requests
FI_COMPLETION so shm's smr_complete_rx() does not drop it.
FI_MULTI_RECV is honored on the shm peer. Because the owner performs the
matching, a posted FI_MULTI_RECV buffer would otherwise have been consumed
by a single node-local message:
- opx_srx_get_match() carves one message out of a posted FI_MULTI_RECV
buffer, handing the shm peer an iov at the current buffer offset sized
to the incoming message, completing through a freshly allocated
per-message context whose multi_recv_context points back at the
original posted buffer, and advancing the offset synchronously. A
send that does not fit the remaining space is reported unmatched so
the shm peer queues it unexpected rather than overrunning the buffer.
- Once the remaining space drops below min_multi_recv the original
buffer context is unlinked and marked so opx_peer_cq_write() emits the
trailing FI_MULTI_RECV release completion after the per-message
completion, preserving message-then-release ordering. Messages that
arrived unexpected before the buffer was posted are carved out of the
unexpected queue with the same logic, covering both orderings.
This is owner-side plumbing only: matching and completion delivery run
through OPX while the shm peer owns the intra-node data transport.
Signed-off-by: Ken Raffenetti <ken.raffenetti@cornelisnetworks.com>
When a node-local operation is offloaded to the shm (smr) peer, the peer cannot interpret an OPX descriptor (a struct fi_opx_mr *) or resolve an OPX memory-region key. Mirror the EFA owner model and give shm its own descriptor and key registered against its own domain. This is shared infrastructure used by the node-local RMA, atomic, and HMEM offload that follows. Add a shm_mr field to struct fi_opx_mr, reusing the spare cacheline-4 qword so the struct stays exactly five cachelines (compile-time asserts unchanged). Register a parallel shm-domain MR at MR-registration time on both the HMEM-cache and non-cache paths, and close it at MR-close time (cache and non-cache). A failed shm registration is non-fatal and simply falls back to a NULL descriptor. opx_get_desc_for_shm() translates an OPX descriptor to the parallel shm_mr descriptor for use by the offload paths. Signed-off-by: Ken Raffenetti <ken.raffenetti@cornelisnetworks.com>
The FI_PEER owner import matched and completed receives from the shm
('smr') peer, but nothing routed transmits to the shm endpoint, so
node-local sends still used OPX's native shared-memory ring and the
imported peer only handled receive matching. Add the transmit-side
offload for msg/tagged sends, one-sided RMA, and atomics so all
node-local operations flow through the shm peer.
Send/tagged offload:
- struct fi_opx_av gains a shm_fi_addr[] forward map (OPX fi_addr ->
shm fi_addr, FI_ADDR_NOTAVAIL when the peer is not shm-reachable),
allocated and grown in lockstep with table_addr and freed in
fi_opx_close_av. fi_opx_av_insert populates it for each node-local
peer (opx_lid_is_shm) by inserting the peer's shm endpoint name into
the shm AV with FI_AV_USER_ID set to the OPX fi_addr, so shm reports
the OPX fi_addr back on completions.
- opx_shm_tx_send() plus interception in fi_opx_ep_tx_send_internal()
and fi_opx_ep_tx_inject_internal() route node-local sends into the
shm endpoint. Because OPX is built with FABRIC_DIRECT, the shm ops
vtable is invoked directly rather than through the fi_tsend/fi_tinject
macros (which resolve back into OPX). Sends that need a completion
carry an OPX-allocated opx_context wrapper holding the application
context.
- opx_peer_cq_write()/writeerr() recover the owning opx_ep from
fid.context and route completions to tx->cq for sends and rx->cq for
receives, since the two may be distinct completion queues.
- The owner get_tag/get_msg match path and the unexpected-message fill
path propagate FI_COMPLETION/FI_REMOTE_CQ_DATA from the posted
receive so shm's smr_complete_rx() does not silently drop the
receive completion.
RMA and atomic offload:
- Intercept the seven RMA internal cores (inject_write, write, writev,
writemsg, read, readv, readmsg) and each atomic core (write, fetch,
compare, and their msg/inject variants). When the destination
resolves to a node-local peer (shm_fi_addr[] != FI_ADDR_NOTAVAIL),
route the operation to the shm endpoint's rma/atomic vtable directly
(FABRIC_DIRECT, so the fi_write/fi_read/fi_atomic macros cannot be
used), with descriptors and the target address translated to the
parallel shm_mr through opx_get_desc_for_shm().
- Reuse OPX's existing completion-counter machinery instead of a new
peer counter: the shm op carries the OPX fi_opx_completion_counter as
its context and is issued with FI_COMPLETION, so shm's tx completion
lands in opx_peer_cq_write(), where a branch recognizing FI_RMA and
FI_ATOMIC drives cc->hit_zero() once for the whole transfer --
incrementing read_cntr/write_cntr and enqueuing the CQ completion
exactly as the native path does. Because the shm route bypasses
fi_opx_atomic_internal, each atomic branch sets cc->cntr (write_cntr
for writes, read_cntr for fetch/compare) that fi_opx_atomic_internal
would otherwise assign.
- Register host memory regions against the shm peer domain as well (not
just HMEM buffers) so shm can resolve the RMA/atomic target key, and
open the shm peer domain with mr_mode 0 (no FI_MR_VIRT_ADDR) to match
OPX's offset-based addressing.
Signed-off-by: Ken Raffenetti <ken.raffenetti@cornelisnetworks.com>
When a node-local operation is offloaded to the shm (smr) peer, the peer needs a descriptor it can interpret for device (HMEM) buffers. Building on the shm-peer MR registration and opx_get_desc_for_shm() translator, hand shm the translated descriptor on the HMEM data paths. The TX path (opx_shm_tx_send) and the owner receive matchers (opx_srx_get_match, the multi-recv carve, and the unexpected-queue drain) now translate the OPX descriptor for the shm peer instead of passing NULL. The receive-side translation covers both the dmabuf and general GPU paths: the fi_opx_hmem_info union is collapsed to a single opx_mr pointer so every device receive preserves the application MR pointer, and native receive copies derive the device id on demand from the MR attributes. Signed-off-by: Ken Raffenetti <ken.raffenetti@cornelisnetworks.com>
opx_hfi1_tx_rzv_rts_hfisvc() takes src_iface and src_device_id but not src_handle, so it cannot call opx_copy_from_hmem() on the device-registered path. Both callers already hold src_handle; pass it through. No functional change: the callee does not yet use the parameter. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
TID writes the destination only on a qword boundary, so sub-qword bytes at either end of an unaligned receive buffer arrive out of band. The kernel sources them in _tx_tracker_latch_xfer_fields(), reading the first and last qword of the send buffer. For a dma-buf source each of those reads maps and unmaps a GPU BAR page, a fixed per-transfer cost that dominates small and medium GPU sends. Read them on the sender instead, through the device-registered host aperture it already has mapped, and carry them in the RTS. The receiver places them and hands the DMA engine only the aligned interior. Bit 31 of the niov field flags an RTS carrying the 16-byte slot; a flag rather than a new BTH opcode because all four hfisvc opcodes are taken. The slot rides after the last transmitted iov, not at a fixed struct offset. 16B spends payload qword 0 on the tag and needs a third SCB and credit for the spill; 9B has two free payload qwords and needs neither. Both add 4 dws to pbc_dws. Both ends must be rebuilt together. The sender and the receiver apply the same eligibility rule: the edge is dropped unless a non-empty aligned interior remains, and dropped if either aperture copy fails. The kernel latch then handles those bytes. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
The first receive into a dma-buf MR defers the rdma_read until the kernel MR handle exists. That path passed 0,0 edge counts and submitted the untrimmed transfer, so the kernel latched the edges for exactly the first-use case. Nothing in the placement needs the MR open: it reads dmabuf.base_addr, set at registration, and hmem_dev_reg_handle, set at recv post. Place immediately and defer only the read. Extract the placement into a helper called from both paths rather than duplicating the offset arithmetic. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
Lowering the default for HMEM after initial performance tuning. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
Add a self-contained multi-test binary (opx_regression) for OPX provider
regression tests, driven by a function-pointer table and selectable by a
leading positional test name or the -f filter. It follows the prov/lpp
directory and build-wiring layout but reuses the shared fabtests
infrastructure (common/shared.c ft_* and common/hmem.c ft_hmem_* via
libfabtests.la) instead of carrying private HMEM plumbing, so host-memory
and device-memory tests share one client-server harness.
Two tests are included:
- sizes - a send data-integrity sweep across the canonical boundary set.
- sel - send-protocol selection across the OPX payload-size thresholds
(eager-SDMA floor, RZV-min, HFISVC-min), asserting the device
transport (eager-SDMA, RZV SDMA/TID DPUT, HFISVC RTS) or the PIO
fallback taken when dma-buf is unavailable.
Build wiring:
- configure.ac: add --enable-opx-tests / AM_CONDITIONAL([ENABLE_OPX])
(distinct from libfabric's --enable-opx provider flag)
- Makefile.am: include prov/opx/Makefile.include
- prov/opx/Makefile.include: build prov/opx/src/opx_regression against
libfabtests.la, with -DUSE_ROCM/-DUSE_CUDA/-DUSE_HMEM gated on the
existing HAVE_ROCR/HAVE_CUDA/HAVE_HMEM conditionals
Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
A device destination staged the eager payload through a host buffer: the leading sub-qword bytes travel in the header and the qword-aligned bulk travels in the payload, so both were copied into hmem_copy_buf to make the message contiguous before one copy to the device. Write each fragment to the device where it belongs instead. The header bytes go to the start of the receive buffer and the payload follows at its own offset, which removes the staging pass. Host destinations and the multi-packet eager paths are unchanged. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
The device-registered small-copy path reads the source through the GPU's host-visible aperture. On a discrete GPU that mapping is write-combining, where an ordinary load is much more expensive than a streaming one. Use a streaming load for the whole 16-byte units of the copy, keeping ordinary copies for the leading and trailing partial units. The source is never rounded down and the length never rounded up, so no byte outside the caller's range is read. The instruction needs SSE4.1, so probe for it once at init and fall back to the previous copy when it is absent or the request is too short to contain a full unit. The write direction is unchanged: aperture stores are posted and already cheap. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
Carrying the rendezvous edge bytes inline in the RTS lets the receiver place those bytes itself, but the sender has to read them out of the source buffer before it can build the RTS. On device memory each of those reads is a synchronous, non-posted read through the host aperture, and both sit on the critical path of every transfer. Add FI_OPX_HFISVC_INLINE_EDGE to select the behavior, defaulting off. The receiver needs no change: it keys off the edge flag in the RTS header, which the sender simply stops setting, so a sender with either setting interoperates with an unmodified peer. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
* prov/opx: Plumb a const flow-control mode through the TX and CQ paths Thread a compile-time-constant flow-control mode parameter through the TX, reliability, and CQ inline paths so a future flow-control algorithm can specialize them without duplicating function bodies. The mode is always NONE in this change: every RCCC branch is unreachable, the RCCC catalog rows bind to the NONE bodies through the mode parameter's constant fold, and no flow-control runtime exists yet. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Split sendv and sendmsg out of the base msg specialization macro Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Address flow-control stack review feedback - Remove the duplicate rzv_comp->byte_counter assignment in the TID eligible branch; the common-path assignment covers all paths. - Replace the opx_sdma_request memset with explicit first-cacheline field initialization and fold the NULL check into an else. - Propagate opx_sdma_request allocation failure from the enqueue-dput paths and mark the work entry COMP_ERROR so the completion reaper retires it; reliability retransmits the registered replays. - Return the packet count directly from opx_hfi1_dput_packet_count (0 on malformed range or overflow) instead of a bool out-parameter. - Restore full sendv/sendmsg specializations (multi-iov and FI_REMOTE_CQ_DATA) so no mid-stack regression window exists for consumers such as NCCL/RCCL. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> --------- Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
* prov/opx: Add the flow-control core and RCCC engine Add the flow-control subsystem: process-wide mode selection, the RCCC credit engine, and the dedicated control-message wire transport. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Address flow-control core review feedback - Size the RX flow pool chunk count with OPX_RELIABILITY_RX_FLOW_CHUNK_COUNT so the flow-key and RX-flow pools grow in lockstep. - Emit FI_WARN with the failing return code and sizing inputs before each flow-pool geometry abort so the crash is diagnosable. - Name the rejected _FI_OPX_FLOW_CONTROL_RCCC_* environment variable and value in FI_WARN before failing RCCC profile configuration; the result is cached process-wide, so an undiagnosed failure poisons every subsequent endpoint initialization. - Clarify that the flow-control QW5/QW6 field layout uses the unified qw_16B[] indexing valid for both wire formats. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Accept RCCC control packets arriving on a secondary plane A tx_index=1 flow's CREDIT_REQUEST egresses the secondary plane, so the arriving LRH slid is the sender's secondary-plane LID while the flow key holds primary LIDs. The receive validation required the key source LID to equal both the LRH slid and the QW[3] primary LID, so every dual-plane control packet failed closed: the request was dropped silently, no grant was ever produced, and the sender spun in -FI_EAGAIN forever. IMB-MPI1 PingPong hung at any scale with FI_OPX_FLOW_CONTROL=rccc and _FI_OPX_DUAL_PLANE_=1. Validate the key source against the primary LID baked into QW[3] (the mechanism reliability ping responses already use for dual-plane routing) and the key destination against the LRH dlid, which is always a primary LID because control senders address the peer's primary plane. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> --------- Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
* prov/opx: Wire RCCC admission into replay acquisition and registration Connect the RCCC engine to the reliability TX seams so every governed packet charges the credit ledger at replay acquisition and settles it at ACK. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Hook RCCC into the send and receive paths Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Govern RMA and atomic operations under RCCC Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Enable RCCC flow-control selection Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Admit all traffic classes under RCCC flow control Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Address RCCC enablement review feedback - Attempt the ECOMM/EINTR/EFAULT link-down recovery check before the RCCC fail-stop in the SDMA writev error handler: a transient link bounce recovers the same way it does under mode none, with the registered replays retransmitted once the link returns. The abort is now scoped to unexplained writev failures after replay registration. - Adopt the count-returning opx_hfi1_dput_packet_count contract at the RCCC demand-estimate call site. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> --------- Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
* prov/opx: Cap per-visit SHM poll drain under RCCC flow control Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> * prov/opx: Address SHM poll cap review feedback - Gate the per-visit SHM drain cap on the endpoint's cached flow_control_mode instead of the reservation-pool pointer proxy, so the RCCC coupling is explicit rather than commented. - Enforce the drain cap with a single post-advance bound instead of the redundant double gate, making the 256-packet per-visit limit exact and unambiguous. Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> --------- Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
Both thresholds are declared FI_PARAM_INT but were read with fi_param_get_size_t. The core writes the parsed value through an int pointer for an int-typed parameter, so only the low half of the caller's size_t was initialized and the upper half held whatever was on the stack. The resulting value almost always exceeded the upper bound, so the range check rejected it and the default was used instead, with a warning reporting an implausible number. Setting either variable therefore appeared to work but silently took effect only sometimes, differing between processes in the same job. Read both as int, and check the lower bound as well as the upper one now that a negative value is representable. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
Updating the default based on internal performance testing. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
The streaming aperture read covers only a source that is 16-byte aligned with a length that is a whole number of 16-byte units. Anything else uses an ordinary copy, because reading a leading or trailing fragment with an ordinary load costs about as much as the streaming loop itself. Widen the read to the enclosing 16-byte units and discard the surplus, so any source and length is read with streaming loads. Partial units go through an aligned scratch buffer. That reads bytes the caller did not ask for, so record the host bounds of the registration and widen only within them. The bounds are left empty when they cannot be established, which keeps the ordinary copy. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
When FI_OPX_HFISVC is enabled, a memory region registered against ROCr device memory is opened and closed asynchronously through the hfisvc command queue. The provider advances the MR lifecycle from the completion-queue entries returned by opx_domain_hfisvc_poll(). The error path of that poll only logged the failure and bumped the MR error counter; it never advanced the MR state. As a result a hard CQ error (status != SUCCESS) on a terminal MR command left the MR stuck in a pending state forever. opx_hfisvc_mr_deferred_open()/deferred_close() only make forward progress from a successful completion, so the deferred worker returned -FI_EAGAIN indefinitely, opx_domain_drain_deferred_work() timed out and returned -FI_EBUSY, fi_opx_close_domain() bailed out early (skipping the debug-counter dump), and the MR was leaked. A deferred-open work item may still reference an MR that fails open, opx_domain_deferred_work_neutralize_mr() clears the back-pointer of any queued work item for that MR before the state is changed upon failure and opx_hfisvc_mr_deferred_open() retires a neutralized (NULL) item without dereferencing it. This closes the use-after-free window where the application could fi_close() and free the MR before the orphaned open worker ran again. Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
Add HFISVC-accelerated one-sided RMA write (fi_write / fi_writev / fi_writemsg / write-with-CQ-data) over CN5000, extending the Phase-1 HFISVC read path. Origin gate (fi_opx_rma.h): route eligible off-SHM SYSTEM writes at or above the SDMA floor through opx_hfisvc_rma_invoke_send_rts. Plain writes ride DPUT_OPCODE_PUT; writedata (FI_REMOTE_CQ_DATA) rides DPUT_OPCODE_PUT_CQ carrying the immediate. inject bypasses HFISVC (no bounce buffer). No new BTH opcode. Signed-off-by: Armando Vinciguerra <armando.vinciguerra@cornelisnetworks.com>
Add a pytest suite exercising the OPX provider over the shared
ClientServerTest harness, covering host and device (ROCr/CUDA) memory
paths.
New files under fabtests/pytest/opx/:
- conftest.py: module-scoped memory_type fixture parametrizing
host_to_host plus device legs (host/rocr, rocr/host, rocr/rocr and
the CUDA equivalents). Device legs carry rocr_memory / cuda_memory
marks so they can be selected with -m, and auto-skip when the peer
reports no matching HMEM device (num_hmem_devices == 0).
- opx_common.py: opx_run_client_server_test() wrapper that drives
ClientServerTest with datacheck_type="with_datacheck" so every leg
byte-verifies the payload across the device<->host boundary, which
is the point of exercising the FI_HMEM path.
- test_rdm.py: fi_rdm_pingpong, fi_rdm_tagged_pingpong and
fi_rdm_tagged_bw across short/standard iteration counts and the
queue/counter completion-type cross-product.
- test_msg.py: fi_msg_pingpong and fi_msg_bw (MSG path; deselected on
OPX via the exclude file).
- test_rma_bw.py: fi_rma_bw for read/write/writedata over the RMA
path.
OPX two-node runs require OOB address exchange; a bare hostname is rejected by fi_opx_fillinfo
with -FI_ENODATA. The suite relies on the harness -E OOB wiring for
addressing.
Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
Under FI_OPX_HFISVC=1, a device-memory (GPU) SDMA dput requires an exported dma-buf fd in the SDMA request meminfo. When the fd is absent the kernel path falls back to the legacy per-vendor ordinal pin, which the hfi1 driver rejects with EINVAL(22) for an HFISVC-held buffer. Force PIO for device-memory transfers under HFISVC whenever a usable dma-buf cannot be used, and only allow SDMA when the driver advertises HFI1_CAP_DMABUF and the MR actually exported a valid fd. Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
A zero-length HMEM eager send crashed with SIGSEGV. fabtests calls ft_sync() at the start of every benchmark, which issues an inband size=0 fi_sendmsg. On OPX the eager path unconditionally routed a non-system iface through opx_copy_from_hmem(), which called ofi_copy_from_hmem() and dispatched on hmem_ops[iface]. For a zero-length payload the source hmem descriptor / iface was never initialized to a valid value, so the dispatch indexed hmem_ops[] with a garbage iface and dereferenced an invalid copy_from_hmem function pointer. Guard the HMEM copy on a non-zero len in all four eager-send transport helpers (9B and 16B, SHM and HFI). With len=0 there is nothing to stage into the bounce buffer, so short-circuiting before the copy avoids the dispatch on the uninitialized iface; the packet-header path already handles a zero-length payload. Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
This branch is based on the state prior to removing the native opx_shm ring, so both intra-node transports are present: the native OPX shared-memory ring and the fi_shm provider imported as a FI_PEER. Add a boolean parameter, FI_OPX_SHM_USE_PEER, that selects which one is used at runtime. When FALSE (the default) the shm peer domain is not created, so opx_domain->shm_domain stays NULL. That single condition cascades: the shm peer endpoint is never opened, fi_opx_enable_shm_peer is skipped, and every av entry's shm_fi_addr stays FI_ADDR_NOTAVAIL, so the FI_PEER send/inject/RMA/atomic interceptions are bypassed and node-local traffic flows through the native ring (today's shipped behavior). When TRUE the shm peer domain is created and node-local traffic is routed through fi_shm, leaving the native ring unused. The default preserves the current shipped behavior until a decision is made to retire the native intra-node transport. Signed-off-by: Ken Raffenetti <ken.raffenetti@cornelisnetworks.com>
* prov/opx: one-sided RMA atomic + fetch over HFI Service Add HFISVC-accelerated one-sided atomic operations on top of the HFISVC RMA write path. Atomic-write: fi_atomic / fi_inject_atomic ride the existing origin write gate (PUT/PUT_CQ carrying a real op field). The target discriminates an atomic PUT from a plain PUT on op != FI_NOOP-1, stages the operand into a local VA via cmd_rdma_read_va (no MR registration), and applies the read-modify-write with OPX_HMEM_ATOMIC_DISPATCH on completion. The single-threaded per-EP poller serializes the apply, so no new lock is needed. fi_atomic_read rides the existing GET path unchanged. Fetch-atomic: a serial two-completion, push-first state machine. The target first pushes the pre-op MR value to the origin result buffer (iov[1]) so a failure never leaves the MR modified, then pulls the operand into staging and applies. Compare-fetch stays on the two-sided path: its compare operand is a separate buffer, not contiguous after the swap. Signed-off-by: Armando Vinciguerra <armando.vinciguerra@cornelisnetworks.com>
The documented default did not match the code, which has used 16384 for ROCr builds since the threshold was lowered. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
…limit A device send at or below the PIO flow eager threshold can use eager SDMA, which is faster than rendezvous. Above that threshold the only eager form left is multi-packet, which is slower than rendezvous. The default rendezvous threshold was a constant that fell inside the eager SDMA range, so lengths that eager SDMA could carry went to rendezvous instead. Place the default one byte past the flow eager threshold, so each length takes the faster of the two. That threshold is computed from the PIO credit count and the packet size, so it cannot be a constant and the resulting default is hardware dependent. An explicitly set FI_OPX_HMEM_RZV_MIN_PAYLOAD_BYTES is unchanged. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com> Assisted-by: developer-opus-5
…M builds The libfabric contract makes fi_msg.desc / fi_msg_tagged.desc a per-iov array of descriptors (void**), but the OPX internal send path expects a single descriptor (struct fi_opx_mr* or NULL). The send-msg entry points passed msg->desc (the array pointer) unchanged, so opx_hmem_get_mr_iface read a garbage iface from an unrelated offset. On FI_HMEM builds that value indexed hmem_ops[iface] in ofi_copy_from_hmem, causing a wild function-pointer dereference and a SIGSEGV. The receive side already dereferences msg->desc[0]; the send side now matches it. Pass msg->desc[0] (NULL-guarded) in fi_opx_sendmsg, fi_opx_sendv, fi_opx_tsendmsg, and the sendmsg/sendv variants of the FI_OPX_MSG_OPS_SPECIALIZED_FUNC_ macro. Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
When configured with --with-rocr, the ROCr detection block checked for <hsa/hsa.h> but its success branch only defined the HAVE_ROCR_RUNTIME_H preprocessor macro; it never set the have_rocr shell variable. As a result have_rocr stayed 0 and both the HAVE_ROCR and HAVE_HMEM automake conditionals evaluated false even when ROCr was present, so no fabtests binary was ever compiled with -DUSE_ROCM or -DUSE_HMEM. Set have_rocr=1 in the header-found branch, mirroring the existing CUDA detection block, so --with-rocr correctly enables the HAVE_ROCR and HAVE_HMEM conditionals. Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
The streaming aperture read helpers call _mm_stream_load_si128(), whose intrinsic is always_inline and requires an SSE4.1 target. Only a runtime probe guards them, so on a compiler defaulting to the baseline x86-64 target the build fails with "inlining failed in call to always_inline '_mm_stream_load_si128': target specific option mismatch". It happens to compile where the default target already includes SSE4.1. Name the target on the two helpers. Keeping it off the translation unit leaves the runtime probe authoritative, so no SSE4.1 is emitted into a path the probe does not guard. Drop inline, which the target attribute prevents in a baseline caller anyway. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com>
When forwarding a packet into a peer subctxt's software RHQ or eager buffer, opx_forward_shared_context() and opx_write_eager_pkt_to_subctxt() drop the packet if that queue is full, but neither path advances the shared hardware RHQ head/RHF sequence on the drop. Since the hardware RHQ is shared across the whole context-sharing group, whichever rank holds the context lock keeps re-polling and re-dropping the same stuck entry on every poll, forever, until the destination drains its queue. If the destination is itself stalled, this is a permanent livelock that blocks delivery to every other rank in the sharing group behind that entry -- and it's invisible to the OPX reliability layer, since the packet never reaches the point where PSN tracking would notice it missing. Bound the retry instead of spinning indefinitely: track consecutive full-queue observations per subctxt in a new forward_stall_count[] array (process-shared). Retry as before until OPX_FORWARD_STALL_MAX_RETRIES consecutive stalls, then advance past the stuck hardware entry and count the drop in forward_drop_count[], so recovery falls to the reliability layer's existing NACK/replay path. This keeps the common case (destination momentarily full, drains on its own) free, while bounding the worst-case livelock to a fixed number of poll iterations. Also surface forward_stall_count[self]/forward_drop_count[self] in opx_debug_dump_endpoint()'s existing SIGUSR2 dump for visibility into how close a rank is running to the retry ceiling. Signed-off-by: Ken Raffenetti <ken.raffenetti@cornelisnetworks.com>
The ROCr monitor calls hsa_amd_pointer_info(), which takes the hsakmt FMM lock. Subscribing under mm_lock deadlocks when an FMM-lock holder reenters libfabric through the munmap intercept and waits for mm_lock. Both the Open MPI patcher and libfabric memhooks can supply the intercept. Add an opt-in subscribe_unlocked capability. Subscribe before taking mm_lock, then recheck subscription_live() under mm_lock before publishing the cache entry. Leave entries uncached if the subscription is no longer live. Apply this sequence to the utility and OPX TID caches; other monitors retain their existing behavior. Protect the ROCr region tree with a short-held lock and serialize subscription and teardown with a separate registration lock that cache publication never takes. Keep HSA calls, allocation, deallocation, and tree destruction outside the tree lock, preventing an indirect tree-lock/FMM/mm_lock cycle. Insert pending entries before registering callbacks so a callback racing registration can invalidate them without freeing state still owned by the registering thread. Pending entries cannot authorize cache publication. ROCr callbacks detach under the tree lock, then notify caches under mm_lock and free entries after releasing both locks. Signed-off-by: Mike Wilkins <mwilkins@cornelisnetworks.com>
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com> Co-authored-by: Ben Lynam <ben.lynam@cornelisnetworks.com> Signed-off-by: Armando Vinciguerra <armando.vinciguerra@cornelisnetworks.com>
Embed a struct util_domain in struct fi_opx_domain and initialize it with ofi_domain_init() so OPX can use the core domain helpers. This follows the pattern OPX already uses for its TID and HMEM sub-domains. ofi_domain_init()/ofi_domain_close() already maintain the fabric reference count (they insert/remove the domain on the fabric's domain_list and inc/dec fabric->ref), so drop the hand-rolled ofi_atomic_inc32/dec32(&util_fabric.ref) calls that OPX used to keep the fabric busy while a domain is open. The util_domain also caches the threading and mr_mode attributes (both derived from info->domain_attr), so drop the duplicate threading and mr_mode fields from struct fi_opx_domain and read them from the embedded util_domain instead. Signed-off-by: Ken Raffenetti <ken.raffenetti@cornelisnetworks.com>
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
* prov/opx: remove hfisvc GET completion watchdog The kernel already guarantees GET termination independently: hfi1_dms_poll_stale() ages a stuck tracker out to -ETIMEDOUT via the same completion callback. The userspace watchdog is therefore redundant with a stronger backstop. Remove the watchdog entirely: the five watch helpers, the per-endpoint watch list and window fields, the completion-counter watch state and flags, the FI_OPX_HFISVC_RMA_COMPLETION_TIMEOUT_SEC parameter, the sweep hook, the debug counter, and all arm/settle call sites. Watched-cc terminals collapse back to the plain cc->hit_zero() completion. Signed-off-by: Armando Vinciguerra <armando.vinciguerra@cornelisnetworks.com>
opx_ep_hfisvc_poll_one_queue() and opx_cq_hfisvc_poll() read up to 64 entries into a struct hfisvc_client_cq_entry[64] but passed buf_size_bytes = sizeof(*hfisvc_out) (one entry) while requesting count = 64. Pass sizeof(hfisvc_out) (the whole array) so the library is told the true buffer capacity, matching the correct usage in opx_domain_hfisvc_poll(). Signed-off-by: Armando Vinciguerra <armando.vinciguerra@cornelisnetworks.com>
avinciguerra07
force-pushed
the
upstream-contrib/opx-12.4
branch
from
October 20, 2026 18:05
d10aa17 to
5c0d7f2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.