Add GPU-direct S3-over-RDMA support via NVIDIA cuObject (--cuobj)

[harshavardhana]

Adds --cuobj, the S3 counterpart to --cufile: GPU-Direct S3 over RDMA via NVIDIA's cuObject (cuObjClient) API. Single-part GET/PUT move the payload out-of-band over RDMA — to/from GPU memory with --gpuids, or host/CPU memory otherwise — while a small body-less HTTP request carries the x-amz-rdma-* headers. Works against RDMA-capable S3 endpoints.

Example:

CUFILE_ENV_PATH_JSON=/path/to/cuobj.json \
  elbencho --s3endpoints https://S3SERVER:9000 --s3key KEY --s3secret SECRET \
    --cuobj -w -r -t 16 -s 8m -b 8m s3://mybucket

Build picks it up automatically with make S3_SUPPORT=1 when the cuObject SDK is present (also links libibverbs/librdmacm), gated behind CUOBJ_SUPPORT exactly the way --cufile is gated behind CUFILE_SUPPORT — so default builds are unchanged. See the new GPU-Direct S3-over-RDMA (cuObject) section in the README for the build/runtime requirements.

Caveats:

• Single-part only: --iodepth=1 and block size == object size.
• No HTTP fallback — an RDMA decline/failure is a hard error.
• The linked libcuobjclient must be version-compatible with the server's cuObject. The build uses the libcuobjclient from your CUDA/cuObject install, so use one whose version matches your server — a mismatch shows up as RDMA buffer-registration / retry exceeded errors even though the HTTP control request succeeds. The cuObject runtime config (CUFILE_ENV_PATH_JSON) needs rdma_dev_addr_list + use_pci_p2pdma + rdma_peer_type.

Testing

Built and validated end-to-end against a multi-node MinIO AIStor cluster over 400Gb RoCEv2 (client used host/CPU buffers, no GPU):

• Correctness: single-part PUT/GET with --verify (data integrity) passes.
• At scale: 32 threads, 1 GiB objects, ~2 TiB transferred each way across the cluster endpoints — WRITE ≈ 30 GB/s, READ ≈ 40 GB/s (read ≈ 345 Gb/s, near the client NIC's line rate; raw ib_write_bw on the same link measured ≈ 310 Gb/s single-stream).
• Both host-memory (CPU) and GPU-direct (VRAM) buffer-registration paths were exercised; the control-plane SigV4/x-amz-rdma-* signing and the server's RDMA reply handling work as implemented.

Note: GPU-direct on the client additionally requires GPUDirect RDMA to be working between the GPU and NIC (e.g. PCIe ACS redirect disabled on the data-path bridges); host/CPU buffers have no such requirement.

[breuner]

Thanks a lot for this, @harshavardhana . Indeed I am also looking forward to having S3-over-RDMA support integrated and have also been working on cuObject based patches. But the preferable way would of course be if the AWS SDK has the RDMA support integrated instead of patching each invidual application that uses an AWS SDK. There are efforts in this direction ongoing, so I have reached out and am waiting for a status update to see if this is a possibility for the near future.