GPU: New SYCL (Intel) backend by lucaci32u4 · Pull Request #243 · thliebig/openEMS

lucaci32u4 · 2026-10-26T02:52:02Z

Generated-by: Claude (Claude Code).
I do not have much background in EM, but in electronics design and software.

SYCL backend for Intel GPUs

SYCL backend supports all extensions and corodinate modes of openEMS. It is created by porting each kernel from HIP to SYCL. Few differences from the HIP algorithms:

Mur, UPML are walking the axis with the smaller stride (HIP modifications may come in a separate PR later - I have achieved tremendous speedups there by leveraging GPU cache semantics and a few other tricks. 23000MCells/s on a 5090)
The voltage and current were fused in one step where not required to be separate by extensions.

Build

Assumes fparser and CSXCAD are already installed in $PREFIX and oneAPI is installed at its default path.

PREFIX=$HOME/Projects/openEMS
source /opt/intel/oneapi/setvars.sh
git clone -b contrib/sycl https://github.com/Alacrity-Education/openEMS.git
cmake -S openEMS -B openEMS/build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PREFIX -DFPARSER_ROOT_DIR=$PREFIX -DCSXCAD_ROOT_DIR=$PREFIX
cmake --build openEMS/build -j && cmake --install openEMS/build
$PREFIX/bin/openEMS openEMS/.github/smoketests/gpu/gpu_engine.xml --engine=gpu -v 2

If icpx is not in path, add -DOPENEMS_ICPX=/opt/intel/oneapi/compiler/latest/bin/icpx to the cmake command.

Last command will print:

Create FDTD engine (GPU, backend: SYCL (...))

The flag -DENABLE_SYCL=OFF can be passed to cmake to build without SYCL.
If the flag is not specified, the cmake script looks for an installation of icpx. If it is found, the flag is automatically turned on (when unspecified).
Pass the ON value to always build with SYCL and print a warning if icpx is not found.

Tests

Our test setup is an Intel Arc B580, on oneAPI 2026.0, Arch Linux.

python/Tests/GPU_Engine.py passes 28/28.
38-case integration test suite covering all of the bench.py cases and some more cases built to provide code coverage over all extensions. To pass, results are required to be identical to upstream on CPU solder (and stop at the same iteration).

Comparation between CPU (Intel Ultra 9 285k, Dual channel DDR5 @ 6400) and GPU (B580):
All numbers in MCells/s.

case	CPU	SYCL
ideal M	703	13374
pml M	305	8013
dispersive M	154	5454
cylindrical M	588	9095

Our LTE Antenna feedline simulation went from 209 MCells/s to 1950 MCells/s (28.7 MCells mesh).

Interesting note: Memory bandwidth is still the bottleneck on B580, yielding about 2000MCells/s on our board. We are seeing dramatic speedups, equivalent to 3x memory bandwidth, on steps of the iterations where most of the data is 0. This lines up with memory bandwidth compression that is done by both NVIDIA and Intel.
Intel's compression is always on. NVIDIA requires explicit turning on and is supported for compute only on Ampere or newer. In my test, on NVIDIA gains were small, in the neighbourhood of 5-8%.

Maintenance

Our organization, Alacrity Education, is open to maintaining the SYCL backend to keep it up to date with upcoming algorithm changes that may appear in openEMS.

Please squash the commits when merging.

Alexandru Iercosan-Lucaci alex.lucaci@alacrity.ro

A SYCL (oneAPI DPC++) backend of the GPU engine, built as the plugin openEMS_gpu_sycl when icpx is the C++ compiler (ENABLE_SYCL). It enumerates the Level Zero GPUs, keeps the fields and the full coefficient arrays in device memory and runs the main voltage and current updates as the two unfused kernels of the HIP backend, ported operation by operation, with 32 bit indices while the fields fit (OPENEMS_SYCL_INDEX overrides). DownloadRange, Synchronize and the device energy sum are included; the extensions are stubs that leave the engine on the host fallback, so every case is correct but slow. The kernels are compiled without FMA contraction, with correctly rounded division and square root and with flush-to-zero following ENABLE_FLUSH_TO_ZERO, which makes them bit-exact with the CPU engines on the cases tested so far. OPENEMS_SYCL_AOT_DEVICES (default bmg-g21) names the GPUs compiled ahead of time; a SPIR-V image serves the rest. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

The SYCL extensions read the operator extensions the way the HIP and Metal ones do. Generated-by: Claude Fable 5.1 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Init() stores every distinct set of the 12 update coefficients once and a 16 or 32 bit set index per node, as the HIP backend does, and keeps the full arrays only when the sets would not pay off. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP excitation kernels: one work-item per excitation point, or one work-item applying them in host order when points share an edge. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP UPML kernels and of their fusion with the main updates: the regions along z are updated by the main kernels, the others by one fused kernel per region and half step, or by separate pre/post kernels with OPENEMS_SYCL_UPML_FUSION=0. The coefficients are stored as sets of 18 where that pays off. The fused-step branches wait for the fused step. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP Mur ABC kernels, one work-item per node of the plane. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP absorbing BC kernels, including the super-absorption current update. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP total-field/scattered-field kernel, one launch per active plane. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP record kernel; the records are copied to the host once per period for the check of the host extension. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP kernels for the closed alpha direction and the r=0 axis; the axis sum runs serially per z line in the host order. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP seam copies and interpolation between a multi-grid level and its sub-grid; both grids share the in-order compute queue. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP lumped RLC kernels with the history slots rotated on the host and the serial variant for elements sharing an edge. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP dispersive material kernels; the conducting sheet derives from the Lorentz operator extension and is covered with it. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP Debye kernels. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP snapshot path (the raw values of the dump regions are gathered on the device and downloaded on the second in-order queue while the next timesteps run; the dump thread waits on the event) and of the frequency domain sums accumulated on the device. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Port of the HIP fused step: one kernel per timestep computes the new voltages of a tile into local memory and the currents from them, into the second pair of field buffers; the currents next to voltages the extensions change are recomputed afterwards. The UPML regions can be folded into the kernel, but on an Arc B580 that is slower than the separate updates with the z slabs in the main kernels (1M cells: 4.5 vs 5.9 GCells/s), so the fused step is used by default only on grids without UPML, where it gains up to 17 % (8M cells). Knobs: OPENEMS_SYCL_FUSED_STEP=1 forces it, =0 disables it, OPENEMS_SYCL_FUSED_REGIONS=0 keeps the regions in their own kernels. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

FDTD/sycl is now a CMake project of its own: with icpx as the compiler of the whole build it is added as a subdirectory, otherwise the top level drives it as an external project compiled by the icpx found (OPENEMS_ICPX) against the libopenEMS just built, with the same include directories, definitions and libstdc++ (--gcc-install-dir). The plugin carries the oneAPI runtime directory in its RPATH, so it loads without the oneAPI environment. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

GPU_Engine.py expects every extension and the frequency domain sums on the device for the SYCL backend; changelog, README and AGENTS.md name the backend and its build options. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

One line per grid with the nodes, the coefficient storage mode, the number of sets and the bytes of coefficient traffic per node, and a line for every outcome of the fused-step decision, as a yardstick for measuring the optimizations that follow. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

The fused step folds only the two UPML regions along z into its kernel, with the tiles aligned to the slab boundaries and the slab rows packed so that no sub-group runs both code paths; the x/y regions run as one grouped launch per half step. On a UPML benchmark the fused step is now 6 % (1M cells) to 11 % (8M cells) faster than the separate kernels, but still 7 % slower on a graded inhomogeneous case at 1M cells, so the default policy is unchanged: OPENEMS_SYCL_FUSED_STEP=1 selects it. OPENEMS_SYCL_FUSED_REGIONS=0 keeps the slabs with the region kernels, 2 uses one tiling for measurement. Results are bit-identical to the separate kernels. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

An optional third form of the update coefficients for Cartesian meshes: vv and ii as stored, vi and iv as a material factor times the edge geometry rebuilt in the kernel from 1-D tables (a double-float product, rounded once), deduplicated into sets on the normalized values; nodes the reconstruction misses by more than 2 ulp keep their exact values. On a PCB model mesh this turns 1.23 million exact sets (59 MB, 32 bit index) into 26524 sets (1.3 MB, 16 bit index). Results stay within 7.5e-6 of the peak on the 38 test cases; the default (exact sets, falling back to full arrays) is unchanged and bit-exact. Opt-in: no speed gain was measured. OPENEMS_SYCL_COEFF=sets|full|factor selects it. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

The two orders of a conducting sheet (and any dispersion orders of one side on the same cells) ran as one launch each, every launch reading and writing all three field components of every cell, the normal one included, which has no coefficient and a zero ADE state. They now run in one launch per side that reads each field value once, subtracts the corrections in the order of the orders as before, and skips the directions no order has a coefficient for. The arithmetic is the same, so the results are bit-identical. On a PCB model with its copper as conducting sheets (921x708x44 cells, Mur ABC) the sheet kernels take 899 us instead of 2x1195 (apply) and 641 instead of 2x654 (pre) per step; the model settles at about 1.95 GCells/s instead of 1.6. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

The Mur ABC, absorbing sheet and TF/SF kernels put the second axis of their plane on the contiguous work-item dimension. On a plane normal to y that is x, the axis with the largest memory stride, while z, with stride 1, ran across the sub-groups. The contiguous dimension now takes the axis with the smaller stride. Planes normal to z have no unit-stride axis and are unchanged. Results are bit-identical. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

These extensions register the voltages they change after the voltage update, so the fused step can recompute the currents next to them with its fix-up kernel, as it does for the excitation and the lumped elements; their own kernels run unchanged before and after the fused kernel. An extension that changes currents on the last mesh lines keeps the separate kernels, as do cylindrical and multi-grid meshes. Measured, the fused step is slower than the separate kernels on every case this makes eligible (the fix-up list is a fifth to half of the nodes), so the default policy is unchanged: these grids take the separate kernels unless OPENEMS_SYCL_FUSED_STEP=1, and the verbose output names the reason. Results are bit-identical. Also fixes a device fault of the forced fused step with a UPML region on the upper z end only: the kernel got the pointers of the absent lower slab. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Generated-by: Claude Fable 5.1 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

The factorized-coefficient commit made main_coeff() load all six coefficients up front and put the factor-mode reconstruction behind a runtime branch, in every main update kernel. In the default modes that cost 40 to 70 % more instructions per kernel, spills in the 64-bit index fused kernels, and 7 to 9 % of the speed of the ideal and cylindrical benchmarks at 8M cells. The coefficient form is now a template parameter: the kernels of the default modes are the code from before that commit again (pointers, read where used), the factor kernels are separate instantiations chosen at launch. Interleaved A/B against the build before it: ideal 1.000, cylindrical 0.994; against the previous tip: ideal +8 %, cylindrical +6 %. Results are unchanged. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

Heroiu-Justinian · 2026-10-26T14:58:36Z

Really needed this one last week when I was trying to run on my B580

lucaci32u4 · 2026-10-26T14:59:47Z

I see that the pipelines are probably not having the intel SYCL backend installed. Do you want me to edit those, or maybe add another specific stage to test the build with SYCL?

…check The sub-build that compiles the SYCL plugin with icpx in a tree built by GCC passed --gcc-install-dir to the plugin's sources only. CMake's compiler identification and check in project() ran without it, so icpx took the newest GCC it found. On Ubuntu 26.04 that is GCC 16, installed as a runtime without C++ headers: the identification failed ("C++ header location not resolved"), CMake no longer knew icpx as IntelLLVM, compiled the plugin without -fPIC and its link failed. The flag now goes into CXXFLAGS before project() as well. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

The backend takes Level Zero GPUs only, so it cannot be tested on a machine without an Intel GPU, such as a CI runner. With OPENEMS_SYCL_CPU=1 it also lists the OpenCL CPU devices of the oneAPI runtime, after the GPUs; the kernels compile from the SPIR-V image and run on the CPU. Off by default, nothing changes for the GPUs. python/Tests/GPU_Engine.py passes with it on the CPU device of oneAPI 2026.2, bit-identical to the basic engine in 24 of 28 cases and within 4.2e-7 of the peak in the others. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

A Linux matrix entry "Ubuntu + oneAPI" (gpu: "sycl"), next to the ROCm and CUDA ones: it installs icpx from Intel's oneAPI apt repository (intel-oneapi-dpcpp-cpp-2026.2, which brings the SYCL runtime and the OpenCL CPU runtime but no GPU driver), builds openEMS with GCC and the SYCL plugin with icpx, SPIR-V only since the runner has no ocloc for the ahead-of-time GPU code. The plugin check of the GPU entries also runs for it: the plugin loads, finds no Intel GPU, and the GPU engine fails cleanly. Then python/Tests/GPU_Engine.py runs with the SYCL kernels on the OpenCL CPU device (OPENEMS_SYCL_CPU=1) and compares them with the basic engine. The step makes sure the SYCL backend ran, since the test passes without a device as well. Generated-by: Claude Opus 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

MrHighVoltage · 2026-11-05T21:49:03Z

Two bugs of master are copied here; #248 and #249 fix them for the other backends:

sycl_ext_cylinder.cpp, cyl_r0_ez: the loop should be a<N.ny-2, not a<N.ny-1 (alpha line 0 is counted twice).
sycl_ext_excitation.cpp and sycl_ext_tfsf.cpp: the period should come from GetSignalPeriodTS(), not int(GetSignalPeriod()/GetTimestep()).

lucaci32u4 · 2026-11-05T22:56:54Z

Interesting, I will look into it and push new commits.

Alexandru Iercosan-Lucaci added 26 commits October 26, 2026 04:46

extensions: friend declarations for the SYCL backend

d3aef97

The SYCL extensions read the operator extensions the way the HIP and Metal ones do. Generated-by: Claude Fable 5.1 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

sycl: Mur ABC extension on the device

dbcd234

Port of the HIP Mur ABC kernels, one work-item per node of the plane. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

sycl: absorbing boundary condition extension on the device

fb8f074

Port of the HIP absorbing BC kernels, including the super-absorption current update. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

sycl: TF/SF extension on the device

d7ea956

Port of the HIP total-field/scattered-field kernel, one launch per active plane. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

sycl: steady-state detection extension on the device

25ffa4a

Port of the HIP record kernel; the records are copied to the host once per period for the check of the host extension. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

sycl: lumped RLC extension on the device

5ded0f2

Port of the HIP lumped RLC kernels with the history slots rotated on the host and the serial variant for elements sharing an edge. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

sycl: Debye material extension on the device

304e764

Port of the HIP Debye kernels. Generated-by: Claude Sonnet 5.5 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

sycl: changelog for the fused-step policy and the coefficient modes

f0c570d

Generated-by: Claude Fable 5.1 (Claude Code) Signed-off-by: Alexandru Iercosan-Lucaci <alex.lucaci@alacrity.ro>

lucaci32u4 closed this Oct 26, 2026

lucaci32u4 changed the title ~~Contrib/sycl~~ GPU: New SYCL (Intel) backend Oct 26, 2026

lucaci32u4 reopened this Oct 26, 2026

Alexandru Iercosan-Lucaci added 3 commits October 29, 2026 14:20

lucaci32u4 marked this pull request as draft October 29, 2026 17:22

Merge branch 'master' into contrib/sycl

2561d94

Navigation Menu

Repository navigation

GPU: New SYCL (Intel) backend - #243

GPU: New SYCL (Intel) backend#243
lucaci32u4 wants to merge 30 commits into
thliebig:masterfrom
Alacrity-Education:contrib/sycl

lucaci32u4 commented Oct 26, 2026 •

edited

Loading

Uh oh!

Heroiu-Justinian commented Oct 26, 2026

Uh oh!

lucaci32u4 commented Oct 26, 2026

Uh oh!

MrHighVoltage commented Nov 5, 2026

Uh oh!

lucaci32u4 commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lucaci32u4 commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SYCL backend for Intel GPUs

Build

Tests

Maintenance

Uh oh!

Heroiu-Justinian commented Oct 26, 2026

Uh oh!

lucaci32u4 commented Oct 26, 2026

Uh oh!

MrHighVoltage commented Nov 5, 2026

Uh oh!

lucaci32u4 commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lucaci32u4 commented Oct 26, 2026 •

edited

Loading