[unsettledroell404]

Context
I have access to an ancient K80 GPU, which has two separate GPU cores on a single board and each core can only talk to 12GB of VRAM. It acts essentially like a dual GPU setup with a fast link between then so I wanted to try and get this to work. The work done here should allow running several (modern) GPUs as well. The K80 is very slow, only outpacing my Laptop's GPU by 60%, but it gives me access to 24GB of VRAM which is great for large models.

I know the maintainer is busy with the Kernels and GPU support. This does not change the kernels, but I will keep it as WIP anyway, since so far some things don't work well and I still need to understand the code. Just wanted to show the possibility here. I'd love to see someone test this further.

(end human written part..)

---

The GPU engine uses one device, so that device's memory limits the model size. This PR lets the HIP backend split the grid along x over several GPUs:

openEMS model.xml --engine=gpu --gpu-devices=0,1

or FDTD.Run(..., engine='gpu', gpu_devices=[0, 1]) from Python. --gpu-device=N is unchanged, and a run on one device takes the same code path as before.

How it works

• Each device gets a slab of x lines plus one copy of the neighbouring line at each cut. The update kernels are unchanged; they see a smaller grid.
• Before each voltage or current update the boundary lines are copied device to device, ordered by stream events, so the host does not wait in between.
• Every HIP extension is set up per part for the nodes that part owns. Field snapshots and the frequency domain sums of the dumps are taken per part and assembled on the host.
• No change to Engine_GPU, the GPU_Backend interface, the update kernels, or the Metal and reference backends.

Limits with a split grid

• No fused voltage/current step, and the UPML uses its separate kernels. A split run on a single device is therefore slower than an unsplit one.
• A Mur boundary or absorbing sheet normal to x that falls on two parts is not split; the extensions then run on the host.
• Cylindrical multi-grid meshes are refused. Each part needs at least 3 mesh lines along x.

Results (two GPUs of a Tesla K80, CUDA 11.4)

All probe files and field dumps are bit-identical to a one-GPU run, for 2 to 13 parts, with each extension.

Model, 100 M cells	One GPU	Two GPUs
Cavity	1667 MC/s (1339 without the fused step)	2368 MC/s
Cavity, PML on all sides	1310 MC/s	1955 MC/s

Device memory for the PML model: 5329 MiB on one GPU, 1515 + 1509 MiB on two.

Tests

Engine_Compare.py now runs every case that can be split a second time with the grid in two parts on one device (OPENEMS_GPU_DEVICES=0,0) and requires results bit-identical to the
unsplit run. This needs only one GPU. On the K80 the new check passes in all 19 cases it applies to.

On this K80, five UPML cases of Engine_Compare.py are not bit-identical to the basic engine (4–5e-6) and dumps fails on the reference backend. I see the same without this change.

Not tested: AMD GPUs (only HIP compiled as CUDA), and GPUs of different types in one run.

Generated with Claude Fable 5.2 (Claude Code)

[unsettledroell404]

tbd if something like #252 should be considered together with this idea.

[td-krzysiek]

Thanks for sharing this! We've also been running single simulations across mixed RTX 3080 and RTX 4090 cards in an experimental implementation, including configurations with up to six GPUs.

Two things have been useful for our setup:

• Explicit host-staged halo exchange, since our machine reports no P2P support between any GPU pair.
• Configurable unequal X partitions, to accommodate different device speeds and VRAM capacities.

I noticed your implementation currently divides the X lines equally. Have you tested the device-to-device copy path on separate cards without P2P support? I understand the runtime may stage those transfers internally.

Our implementation is here if useful: td-krzysiek#25. It keeps partitioning and communication separate from the backend, with HIP and Metal adapters. It is still experimental.

Your event-based halo scheduling and per-part DFT accumulation look particularly useful — there may be opportunities to combine ideas from both approaches.

[unsettledroell404]

Thanks for sharing this! We've also been running single simulations across mixed RTX 3080 and RTX 4090 cards in an experimental implementation, including configurations with up to six GPUs.

Two things have been useful for our setup:

• Explicit host-staged halo exchange, since our machine reports no P2P support between any GPU pair.
• Configurable unequal X partitions, to accommodate different device speeds and VRAM capacities.

I noticed your implementation currently divides the X lines equally. Have you tested the device-to-device copy path on separate cards without P2P support? I understand the runtime may stage those transfers internally.

Our implementation is here if useful: td-krzysiek#25. It keeps partitioning and communication separate from the backend, with HIP and Metal adapters. It is still experimental.

Your event-based halo scheduling and per-part DFT accumulation look particularly useful — there may be opportunities to combine ideas from both approaches.

Thanks, useful to see. No, I have only run it on the two GPUs of one Tesla K80. The code never enables peer access and uses a plain device-to-device hipMemcpyAsync, so I expect the runtime to stage the copy itself on a machine without P2P, but I have not measured that.

If you have time, a run of this branch on your 3080/4090 machine would settle it:

  openEMS model.xml --engine=gpu --gpu-devices=0,1 --verbose=1

Results should be bit-identical to a one-GPU run of the same branch. If the direct copy fails or is slow there, a host-staged path like yours should be added.

As for the unequal split; I think that is something I can implement here as well.