Repository navigation
Assertion failure in device_kernel_info.cpp:67 while executing on any SYCL device #20132
Description
Describe the bug
This is related to IntelPython/dpctl#2156
Executing Python test suite for dpctl Python project compiled with SYCL nightly build, and using its SYCL runtime library, and running on AMD Epyc CPU for which CPU compiler identifies ARCH as core-avx2 an assertion assert(Info == *this); at https://github.com/intel/llvm/blob/sycl/sycl/source/detail/device_kernel_info.cpp#L71
To reproduce it on Intel CPU one needs to either set CL_CONFIG_CPU_VECTORIZER_MODE=8 or CL_CONFIG_CPU_TARGET_ARCH=core-avx2.
To reproduce
Do not have a simple reproducible.
- Start with SYCL nightly bundle, and TBB 2022.2.0 and OCLCPU=2025.20.6.0.04_224945 dependencies, so that
sycl-lssees the CPU using this OCLCPU driver. Set environment variableDPCPP_HOMEto that${DPCPP_HOME}/bin/clang++locates C++ clang compiler. - Download Miniforge Python package manager:
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh - Install it locally:
chmod 755 ./Miniforge3-Linux-x86_64.sh && ./Miniforge3-Linux-x86_64.sh -b -p $(pwd)/miniforge - Activate it:
source $(pwd)/miniforge/etc/profile.d/conda.sh - Create Python environment to build
dpctl:conda create -n dev_dpctl --yes python=3.13 cython numpy scikit-build pytest versioneer cmake ninja. - Clone repository:
git clone https://github.com/IntelPython/dpctl.git - Change to the cloned directory
cd dpctl - Start building
python scripts/build_locally.py --compiler-root=${DPCPP_HOME}--c-compiler=${DPCPP_HOME}/bin/clang --cxx-compiler=${DPCPP_HOME}/bin/clang++ - Execute test that results in assertion failure:
OCL_ICD_FILENAMES=~/sycl-builds/oclcpuexp/x64/libintelocl.so OCL_ICD_VENDORS=. CL_CONFIG_CPU_VECTORIZER_MODE=8 ONEAPI_DEVICE_SELECTOR=opencl:cpu gdb --args python -m pytest -s dpctl/tests
Environment
- OS: Linux, Ubuntu 24.04 for WSL
- Target device and vendor: CPU
- DPC++ version:
clang version 21.0.0git (https://github.com/intel/llvm 3906cea47a1e9e564ea3a10ef21d9b7e65c560c0)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/sycloid/sycl-builds/16-09-2025/bin
Build config: +assertions
- Dependencies version: [e.g. the output of
sycl-ls --verbose]
INFO: Output filtered by ONEAPI_DEVICE_SELECTOR environment variable, which is set to opencl:cpu.
To see device ids, use the --ignore-device-selectors CLI option.
<LOADER>[INFO]: loaded adapter 0x0x55b25b34cd00 (libur_adapter_opencl.so.0) from /home/sycloid/sycl-builds/07-09-2025/lib/libur_adapter_opencl.so.0
[opencl:cpu] Intel(R) OpenCL, Intel(R) Core(TM) Ultra 7 155U OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]
Platforms: 1
Platform [#1]:
Version : OpenCL 3.0 LINUX
Name : Intel(R) OpenCL
Vendor : Intel(R) Corporation
Devices : 1
Type : cpu
Version : OpenCL 3.0 (Build 0)
Name : Intel(R) Core(TM) Ultra 7 155U
Vendor : Intel(R) Corporation
Driver : 2025.20.6.0.04_224945
UUID : 8680a406-0a00-2332-faff-fffb8b1f0000
DeviceID : 657060
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : cpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations usm_system_allocations ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice usm_atomic_host_allocations usm_atomic_shared_allocations atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_legacy_image ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
info::device::sub_group_sizes: 4 8 16 32 64
Architecture: x86_64
default_selector() : cpu, Intel(R) OpenCL, Intel(R) Core(TM) Ultra 7 155U OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]
accelerator_selector() : No device of requested type available.
cpu_selector() : cpu, Intel(R) OpenCL, Intel(R) Core(TM) Ultra 7 155U OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]
gpu_selector() : No device of requested type available.
custom_selector(gpu) : No device of requested type available.
custom_selector(cpu) : cpu, Intel(R) OpenCL, Intel(R) Core(TM) Ultra 7 155U OpenCL 3.0 (Build 0) [2025.20.6.0.04_224945]
custom_selector(acc) : No device of requested type available.
Additional context
The backtrace of crash from GDB:
.............................................................................................python: /__w/llvm/llvm/src/sycl/source/detail/device_kernel_info.cpp:67: void sycl::_V1::detail::DeviceKernelInfo::setCompileTimeInfoIfNeeded(const sycl::_V1::detail::compile_time_kernel_info_v1::CompileTimeKernelInfoTy&): Assertion `Info == *this' failed.
Thread 1 "python" received signal SIGABRT, Aborted.
Download failed: Invalid argument. Continuing without source file ./nptl/./nptl/pthread_kill.c.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
warning: 44 ./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7cef27e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff7cd28ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff7cd281b in __assert_fail_base (fmt=0x7ffff7e7a1e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7ffff67d22fc "Info == *this",
file=file@entry=0x7ffff67d21c8 "/__w/llvm/llvm/src/sycl/source/detail/device_kernel_info.cpp", line=line@entry=67,
function=function@entry=0x7ffff67d2250 "void sycl::_V1::detail::DeviceKernelInfo::setCompileTimeInfoIfNeeded(const sycl::_V1::detail::compile_time_kernel_info_v1::CompileTimeKernelInfoTy&)") at ./assert/assert.c:96
#6 0x00007ffff7ce5517 in __assert_fail (assertion=0x7ffff67d22fc "Info == *this",
file=0x7ffff67d21c8 "/__w/llvm/llvm/src/sycl/source/detail/device_kernel_info.cpp", line=67,
function=0x7ffff67d2250 "void sycl::_V1::detail::DeviceKernelInfo::setCompileTimeInfoIfNeeded(const sycl::_V1::detail::compile_time_kernel_info_v1::CompileTimeKernelInfoTy&)") at ./assert/assert.c:105
#7 0x00007ffff668b270 in sycl::_V1::detail::DeviceKernelInfo::setCompileTimeInfoIfNeeded(sycl::_V1::detail::compile_time_kernel_info_v1::CompileTimeKernelInfoTy const&) () from /home/sycloid/sycl-builds/16-09-2025/lib/libsycl.so.8
#8 0x00007ffff66c499e in sycl::_V1::detail::ProgramManager::getOrCreateDeviceKernelInfo(sycl::_V1::detail::compile_time_kernel_info_v1::CompileTimeKernelInfoTy const&) () from /home/sycloid/sycl-builds/16-09-2025/lib/libsycl.so.8
#9 0x00007fff811050c4 in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#10 0x00007fff81104be0 in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#11 0x00007ffff66e72f1 in sycl::_V1::detail::queue_impl::submit_impl(sycl::_V1::detail::type_erased_cgfo_ty const&, bool, sycl::_V1::detail::code_location const&, bool, sycl::_V1::detail::v1::SubmissionInfo const&) () from /home/sycloid/sycl-builds/16-09-2025/lib/libsycl.so.8
#12 0x00007ffff67aaab6 in sycl::_V1::queue::submit_with_event_impl(sycl::_V1::detail::type_erased_cgfo_ty const&, sycl::_V1::detail::v1::SubmissionInfo const&, sycl::_V1::detail::code_location const&, bool) const () from /home/sycloid/sycl-builds/16-09-2025/lib/libsycl.so.8
#13 0x00007fff810b69f9 in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#14 0x00007fff81101ddd in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#15 0x00007fff810ffef5 in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#16 0x00007fff810ff361 in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#17 0x00007fff810af129 in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#18 0x00007fff810ab20b in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#19 0x00007fff810ab18f in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#20 0x00007fff8107e5b3 in ?? () from /home/sycloid/repos/dpctl/dpctl/tensor/_tensor_impl.cpython-313-x86_64-linux-gnu.so
#21 0x000055555572f644 in cfunction_call (func=0x7fff81f903b0, args=0x7fff19630bc0, kwargs=<optimized out>)
at /usr/local/src/conda/python-3.13.7/Objects/methodobject.c:539
#22 0x00005555556fdddc in _PyObject_MakeTpCall (tstate=0x555555ac45b0 <_PyRuntime+283024>, callable=0x7fff81f903b0, args=0x7ffff7ae7820, nargs=2,
keywords=<optimized out>) at /usr/local/src/conda/python-3.13.7/Objects/call.c:242
#23 0x00005555557176a9 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>)
Activity
sycloid commentedon Sep 18, 2025
Actually, the assertion is tripped even without setting any
CL_CONFIG_*variables:sycloid commentedon Sep 18, 2025
I can reliably reproduce the issue running just two test files out of the entire test suite:
sycloid commentedon Sep 19, 2025
Smaller yet, just two tests:
sycloid commentedon Sep 19, 2025
It appears this issue is not specific to CPU at all. It happens with Level-Zero too:
[-]Assertion failure in device_kernel_info.cpp:67 while executing on CPU[/-][+]Assertion failure in device_kernel_info.cpp:67 while executing any any SYCL device[/+]sycloid commentedon Sep 22, 2025
Having built the
dpctlwith Debug build of DPC++ this is info from GDB:sycloid commentedon Sep 22, 2025
@bratpiorka This is not specific to OpenCL. The assertion is tripped for Level-Zero devices as well as for OpenCL devices (both GPU and CPU)
sycloid commentedon Sep 22, 2025
Judging from the
dpctllogs for workflow building and testing the project with nightly builds of SYCL bundle, the problem was introduced between Sept. 10 and Sept. 11:I can reproduce the problem using installed SYCL nightly from Sept 16, but it is not reproducible using build from Sept. 7
bratpiorka commentedon Sep 23, 2025
@sycloid thanks for the investigation, this would really help to identify the root cause of this failure
sycloid commentedon Sep 23, 2025
@bratpiorka The commit that introduced the issue is cb03a1b that merged #20003
The SYCL built from the previous merge commit e26848d does not exhibit the assertion failure.
sycloid commentedon Sep 23, 2025
The kernel involved when the assertion is tripped is defined via lambda in
dpctl:Question: Should the code in
dpctlbe changed to avoid the assertion?sycloid commentedon Sep 23, 2025
CC: @sergey-semenov as author of #20003 that tripped the assertion failure for
dpctlfor commentssycloid commentedon Sep 24, 2025
Here is the minimal Python reproducer:
changing the order of operation the assertion is still reproducible:
What may be relevant is that the same kernel (same kernel name and same lambda) is instantiated twice in different translation units. They end up being in different shared objects. I think the problem occurs while jitting of the second copy is attempted.
sycloid commentedon Sep 24, 2025
Using the two Python scripts with code given in previous comment I am attaching outputs of execution with
SYCL_UR_TRACE=-1.Execution 1
Execution 2
[-]Assertion failure in device_kernel_info.cpp:67 while executing any any SYCL device[/-][+]Assertion failure in device_kernel_info.cpp:67 while executing on any SYCL device[/+]sycloid commentedon Sep 25, 2025
I can confirm that moving implementations of Python functions that are using the same kernels to a common DSO resolves the crash.
Even though we have a work-around, the question remains as to whether:
sycloid commentedon Sep 27, 2025
I opened IntelPython/dpctl#2157 to implement a work-around in
dpctl.To reproduce the reported behavior please use
dpctlsources at SHA878cc1935c8a9fa514b355f5164539cdd8e1be39sycloid commentedon Sep 29, 2025
The crash reproducible with cb03a1b can no longer be reproduced with 36363e6, so I assume this issue was transient and can now be closed.
bratpiorka commentedon Sep 30, 2025
ok, thanks!
sycloid commentedon Sep 30, 2025
This is a duplicate of #20135, and was fixed in #20135