[TEST][UR][L0] Disable USM pooling on Xe2+ devices with L0 driver 1.18+ by ldorau · Pull Request #23385 · intel/llvm

ldorau · 2026-10-06T13:28:13Z

Use a proxy pool instead of a disjoint pool in both the L0 v1 and v2
adapters for devices that are Xe2 or newer and have an L0 driver of
version 1.18 or newer. Host pools are not associated with any device,
so pooling of host memory is disabled only if all devices in the context
meet this condition.

Fixes: URT-1298
Blocked-by: NEO-17015

ldorau · 2026-10-09T14:03:16Z

Fixes: URT-1298
Blocked-by: NEO-17015
Waiting-for: NEO-17015

Use a proxy pool instead of a disjoint pool in both the L0 v1 and v2 adapters for devices that are Xe2 or newer and have an L0 driver of at least the minimum version defined by UR_L0_USM_POOLING_DISABLED_MIN_DRIVER_MAJOR and UR_L0_USM_POOLING_DISABLED_MIN_DRIVER_MINOR in common/usm_pooling_disabled.hpp (currently 1.17). Host pools are not associated with any device, so pooling of host memory is disabled only if all devices in the context meet this condition.

When USM pooling is disabled for Xe2+ devices with at least the minimum L0 driver version (see isUsmPoolingDisabled()), the adapter uses a UMF proxy pool instead of a disjoint pool. The proxy pool's malloc_usable_size implementation always returns UMF_RESULT_ERROR_NOT_SUPPORTED, which appendUSMFreeExp() treated as fatal, causing urEnqueueUSMFreeExp to fail with UR_RESULT_ERROR_UNKNOWN. This mirrors the tolerance pattern already present in usm.cpp (v1) and v2/usm.cpp for the same proxy-pool quirk. Also relax enqueue_alloc.cpp's SuccessReuse test: the pointer-reuse assertion doesn't hold when pooling is disabled and a proxy pool (with no reuse semantics) is used, so skip it for such devices via a new isUsmPoolingDisabledForDevice() helper that mirrors the adapter-side check and uses the UR_L0_USM_POOLING_DISABLED_MIN_DRIVER_MAJOR/MINOR constants.

TearDown() called urUSMFree(context, deviceMem) before resetData(), which enqueues a USM fill on deviceMem. This used the pointer after it had already been freed. This was masked when USM pooling kept freed memory mapped/valid for reuse, but is exposed now that pooling is disabled for Xe2+ devices with at least the minimum L0 driver version (a UMF proxy pool returns memory immediately on free), causing urEnqueueUSMFill to fail with UR_RESULT_ERROR_UNKNOWN in TearDown for every test in the urGraphPopulatedExpTest hierarchy on affected devices (seen in CI as widespread exp_graph-test failures on Arc B580). Fix: reset the data while the allocation is still valid, then free it.

Proxy pools (used when USM pooling is disabled, e.g. on Xe2/BMG devices with at least the minimum L0 driver version) always fail malloc_usable_size with UMF_RESULT_ERROR_NOT_SUPPORTED and report size 0. Since AllocationStats relies on that call to track allocation sizes, UR_USM_POOL_INFO_USED_CURRENT_EXP/USED_HIGH_EXP always reported 0 for proxy-pool-backed allocations, even though memory was actually in use. Fix by tracking whether a UsmPool is backed by a proxy pool (IsProxy/isProxy), and for such pools querying the underlying memory provider's own statistics (umf.provider.by_handle.{}.stats. allocated_memory / peak_memory) via umfCtlGet, summing them with the regular AllocationStats-based accounting used for non-proxy pools. This mirrors the existing approach already used for getTotalReservedSize()/getPeakReservedSize(). getTotalUsedSize()/getPeakUsedSize() now return ur_result_t (with the size returned via an output parameter) instead of throwing through the UR C API boundary, so UMF errors are propagated cleanly to urUSMPoolGetInfoExp callers instead of risking an uncaught C++ exception crossing an extern "C" entry point. Also fix USM/usm_leak_check.cpp: direct_usm() only freed the shared allocation (p1) and relied on pooling to implicitly free the host/ device allocations (p2/p3) at process exit. With USM pooling disabled on some devices, this now leaks and fails the test. Explicitly free p2 and p3 as well.

The Level Zero adapter disables USM pooling on Xe2 or newer Intel GPUs (IP version >= 20.1.0) with at least the minimum L0 driver version (UR_L0_USM_POOLING_DISABLED_MIN_DRIVER_MAJOR/MINOR). Detect this condition in lit.cfg.py from sycl-ls output and mark the tests which verify pooling behavior (usm_device_read_only.cpp, usm_pooling.cpp) as UNSUPPORTED on such devices. The minimum driver version is mirrored in lit.cfg.py as L0_USM_POOLING_DISABLED_MIN_DRIVER_MAJOR/MINOR.

The UMF Level Zero provider always passes the relaxed allocation limits descriptor to zeMemAllocHost, so some newer L0 drivers accept absurd sizes like SIZE_MAX. With the disjoint pool this was masked by a failing umfPoolMallocUsableSize() call, but with the proxy pool (USM pooling disabled on Xe2+ with at least the minimum L0 driver version) the allocation succeeds and USM/badmalloc.cpp fails. Reject host allocations larger than maxMemAllocSize unless relaxed allocation limits are enabled, matching the V1 adapter behavior.

Many Graph tests run with UR_L0_LEAKS_DEBUG=1 but never free their USM allocations. This was hidden by the UMF disjoint pool, which releases all of its slabs on destruction. With USM pooling disabled (Xe2+ with at least the minimum L0 driver version) the proxy pool forwards every allocation directly to the driver, so unfreed allocations are reported as leaks. Add the missing free() calls.

The UMF proxy pool (used when USM pooling is disabled) does not support umfPoolMallocUsableSize(). In that case urEnqueueUSMFreeExp (V1) freed the memory synchronously and returned early without signaling the event or executing the command list, causing DEVICE_LOST errors and hangs. V2 appendUSMFreeExp inserted the allocation into the async pool with size 0, so it could never be reused. Query the allocation size with zeMemGetAddressRange instead when umfPoolMallocUsableSize() returns UMF_RESULT_ERROR_NOT_SUPPORTED.

When USM pooling is disabled (Xe2 with at least the minimum L0 driver version), V1 uses UMF proxy pools, so every USM free goes directly to zeMemFree and the memory is unmapped immediately. With the disjoint pool, freed memory stayed mapped, so GPU accesses still in flight were harmless; now they can trigger GPU page faults, engine resets and DEVICE_LOST cascades. The V2 adapter already sets the BLOCKING_FREE policy on its UMF Level Zero provider. Do the same in V1's L0MemoryProvider when pooling is disabled by calling zeMemFreeExt with ZE_DRIVER_MEMORY_FREE_POLICY_EXT_FLAG_BLOCKING_FREE.

ldorau · 2026-10-16T07:10:02Z

Rebased (because of merge conflicts)

- Add isUsmPoolingDisabled(Context, Device) and use it both for pool descriptors and in L0MemoryProvider::initialize() to decide on blocking free, instead of duplicating the all-devices check. - Log the pooling-disabled message at DEBUG level, since it is emitted on every pool creation. - Document that the V2 host allocation size check applies to all devices and uses the largest maxMemAllocSize in the context. - Document why zeMemGetAddressRange gives the correct size in the async free paths (USM free always gets the allocation's base pointer). - List in usm_pooling_disabled.hpp all places duplicating the detection logic that must be kept in sync. Signed-off-by: Lukasz Dorau <lukasz.dorau@intel.com>

ldorau · 2026-10-16T10:50:46Z

@kswiecicki @pbalcer Please review - it is a draft just because it is blocked by NEO-17015 and cannot be merged yet, but it is already ready for review.

kswiecicki · 2026-10-19T09:15:30Z

+  // This applies to all devices, not only to those with USM pooling disabled.
+  // Host memory is accessible from every device in the context, so the
+  // largest maxMemAllocSize among them is used as the limit.
+  if (type == UR_USM_TYPE_HOST) {


Perhaps this could be cached in the context struct somewhere, it's being calculated during every host allocate hot path.

kswiecicki · 2026-10-19T09:52:44Z

  void TearDown() override {
    if (deviceMem) {
-      ASSERT_SUCCESS(urUSMFree(context, deviceMem));
+      // Reset the data while deviceMem is still a valid allocation. Freeing


nit: This comment describes in detail why those 2 calls should be reversed, could you make it simpler?

kswiecicki · 2026-10-19T09:54:44Z

  return os;
 }

+// Mirrors ur_device_handle_t_::isUsmPoolingDisabled() in


nit: This comment goes into details like file paths which could change anytime. You might want to just say that this mirrors how the L0 adapter determines USM pooling availability.

kswiecicki · 2026-10-19T10:02:38Z

    return UR_RESULT_ERROR_INVALID_VALUE;
  }

+  // The UMF Level Zero provider always passes the relaxed allocation limits


nit: This comment seems too verbose.

kswiecicki · 2026-10-19T12:30:08Z

Just to confirm, the L0 driver guarantees pooling for the host USM allocations if any of the available devices is BMG or newer, and for USM shared/device allocations if the particular device used for an allocation is BMG or newer?

Cache the effective host USM allocation limit in the V2 context instead of recomputing it on every allocation. Preserve the existing relaxed-limit behavior and simplify explanatory comments in the allocation and test code.

ldorau · 2026-10-20T07:46:29Z

Just to confirm, the L0 driver guarantees pooling for the host USM allocations if any of the available devices is BMG or newer, and for USM shared/device allocations if the particular device used for an allocation is BMG or newer?

Not quite. For host USM, this PR turns off UR pooling only if all devices in the context are BMG or newer. For device/shared USM, it checks only the device used for that allocation. Both checks require a new enough L0 driver.

This was referenced Oct 6, 2026

[SYCL][libdevice] improve complex trig and pi ulp #23375

Merged

[L0] memory copy tests failing on DG2, when using level_zero v2 #23392

Open

ldorau force-pushed the URL0_Disable_USM_pooling_on_Xe2_devices_with_L0_driver_1.18 branch 4 times, most recently from 8653d12 to cb3cca0 Compare October 9, 2026 09:52

ldorau requested review from kswiecicki and pbalcer October 16, 2026 05:59

ldorau added 9 commits October 16, 2026 07:05

ldorau force-pushed the URL0_Disable_USM_pooling_on_Xe2_devices_with_L0_driver_1.18 branch from cb3cca0 to cdfd8e9 Compare October 16, 2026 07:08

kswiecicki reviewed Oct 19, 2026

View reviewed changes

[UR][L0] Address USM pooling review comments

8e274ef

Cache the effective host USM allocation limit in the V2 context instead of recomputing it on every allocation. Preserve the existing relaxed-limit behavior and simplify explanatory comments in the allocation and test code.

ldorau requested a review from kswiecicki October 20, 2026 07:46

Conversation

ldorau commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldorau commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldorau commented Oct 16, 2026

Uh oh!

ldorau commented Oct 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kswiecicki commented Oct 19, 2026

Uh oh!

ldorau commented Oct 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ldorau commented Oct 6, 2026 •

edited

Loading

ldorau commented Oct 9, 2026 •

edited

Loading