Repository navigation
[SYCL][E2E] Failures in unrelated tests on Intel Arc GPUs (Level Zero) with parallel E2E runs on Linux (B580) and Windows (A770) #23521
Description
Describe the bug
In one CI run of the E2E suite on PR #23250 (merge commit a6c1caa), unrelated tests failed on both Linux (Arc B580) and Windows (Arc A770). The errors include UR_RESULT_ERROR_DEVICE_LOST, UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY, queue-creation failures, crashes, timeouts and wrong results.
WorkGroupMemory/basic_usage.cpp failed on both platforms.
Re-running the failed CI jobs did not reproduce the failures.
To reproduce
Build the toolchain at a6c1caa.
Configure the E2E tests:
Linux: -DSYCL_TEST_E2E_TARGETS="level_zero_v1:gpu;level_zero_v2:gpu"
Windows: -DSYCL_TEST_E2E_TARGETS="level_zero:gpu"
Run the suite with lit's default parallelism (20 workers on Linux, 24 on Windows):
llvm-lit -sv sycl/test-e2e
Each test runs as:
env [UR_LOADER_USE_LEVEL_ZERO_V2=0|1] ONEAPI_DEVICE_SELECTOR=level_zero:gpu .out
Expected: all tests pass.
Actual: 15 tests failed and 4 timed out on Linux, and 24 tests failed on Windows (lists below).
Linux: Arc B580, 15 failed, 4 timed out
Errors seen:
level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY), from SYCL exception caught! : Enqueue process failed.
WorkGroupMemory/basic_usage.cpp:50: void swap_scalar(T &, T &) [T = char]: Assertion 'a == old_b && b == old_a && "Incorrect swap!"' failed.
Failed / timed-out tests
Failed:Adapters/level_zero/event-leak.cpp
ESIMD/api/functional/ctors/ctor_copy_core.cpp
ESIMD/api/functional/ctors/ctor_vector_core.cpp
Matrix/joint_matrix_half_accumulator.cpp
SYCLBIN/basic_executable_bmg_aot.cpp
SYCLBIN/link_rtc_input.cpp
SubGroup/scan_fp16.cpp
SubGroup/scan_spirv13_fp64.cpp
USM/memops2d/copy2d_dhost_to_shared.cpp
USM/memops2d/copy2d_host_to_shared.cpp
USM/memops2d/copy2d_shared_to_dhost.cpp
USM/memops2d/memcpy2d_host_to_device.cpp
WorkGroupMemory/basic_usage.cpp
bindless_images/array/read_write_unsampled_array.cpp
bindless_images/vulkan_interop/vulkan_sycl_image_interop_read_1d.cpp
Timed out:
ESIMD/api/functional/ctors/ctor_default_core.cpp
InvokeSimd/Feature/ImplicitSubgroup/scale.cpp
bindless_images/array/read_sampled_array.cpp
bindless_images/read_sampled.cpp
Abort was called at 39 line in file:
ur_die: [L0] getZeQueue: failed to create queue
FAIL: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST) (in Basic/large_range_error.cpp)
Assertion failed: a == old_a && b == old_b && "Incorrect swap!", file ...\WorkGroupMemory\basic_usage.cpp, line 125
Assertion failed: Data[I] == Values[I], file ...\Basic\handler\handler_mem_op.cpp, line 292
Unexpected value 8 at index 0 after fill. Expected -16. (in Basic/fill_accessor.cpp)
Assertion failed: !R && "Already initialized!", file ...\sycl\source\detail/adapter_impl.hpp, line 283
Failed tests
BFloat16/bfloat16_builtins.cpp
BFloat16/bfloat16_vec.cpp
BFloat16/bfloat16_vec_builtins.cpp
Basic/fill_accessor.cpp
Basic/handler/handler_mem_op.cpp
Basic/large_range_error.cpp
Basic/vector/bool.cpp
Basic/vector/byte.cpp
DeviceImageDependencies/NewOffloadDriver/dynamic.cpp
DeviceImageDependencies/dynamic.cpp
DeviceImageDependencies/dynamic_compress.cpp
DeviceImageDependencies/dynamic_link_opt_win.cpp
DeviceLib/built-ins/nan.cpp
ESIMD/api/simd_mask.cpp
ESIMD/lsc/lsc_usm_block_load_u8_u16.cpp
ESIMD/lsc/lsc_usm_block_load_u8_u16_64.cpp
ESIMD/lsc/lsc_usm_gather_u32.cpp
FreeFunctionKernels/SeparateCompilation.cpp
Matrix/joint_matrix_bf16_fill_k_cache.cpp
Reduction/reduction_internal_range_2dim.cpp
Reduction/reduction_range_3d_dw.cpp
Reduction/reduction_usm_dw.cpp
SubGroup/scan.cpp
WorkGroupMemory/basic_usage.cpp
Environment
Linux
- OS: Ubuntu 24.04.1 LTS, kernel 6.11.0-29-generic (container ghcr.io/intel/llvm/ubuntu2404_intel_drivers:latest)
- GPU: Intel Arc B580 (intel_gpu_bmg_g21)
- CPU: 12th Gen Intel Core i9-12900
- Target devices: level_zero_v1:gpu, level_zero_v2:gpu
- Level Zero driver: 1.17.39758+10
- OpenCL GPU driver: 26.35.39758.10
- IGC: 2.41.5
- Host compiler: gcc 13.3.0
- Lit workers: 20
- DPC++: a6c1caa (PR [SYCL] Consistent is_device_copyable for sycl::accessor #23250 merge)
Windows
- OS: Windows
- GPU: Intel Arc A770 (intel_gpu_acm_g10)
- CPU: 12th Gen Intel Core i9-12900
- Target devices: level_zero:gpu
- Level Zero driver: 1.17.39500+2
- GPU driver: 32.0.101.9030
- Host compiler: MSVC 19.43.34808
- Lit workers: 24
- DPC++: a6c1caa (PR [SYCL] Consistent is_device_copyable for sycl::accessor #23250 merge)
Additional context
CI jobs (both from PR #23250):
Linux: E2E (Intel / Battlemage Graphics, ["Linux", "bmg"], level_zero_v1:gpu;level_zero_v2:gpu)
Windows: run_prebuilt_e2e_tests (Intel Arc Graphics with Level Zero, ["Windows","arc"])
Activity