Static backend selection: sumtype DynamicBackend + withbackend

[asinghvi17]

Stacked on #308, actions my review comments there. Companion PR for DiskArrayEngine: meggart/DiskArrayEngine.jl#70.

Changes

• DiskArrayEngineBackend is defined in DiskArrays. A MethodError hint points to using DiskArrayEngine when its methods are missing, or says that DiskArrayEngine does not implement the operation when it is already loaded. This removes the DiskArrayEngine dep and DiskArrayEngineExt, which only imported the type (and made load_backend error for the "DiskArrayEngine" preference, since the extension cannot be loaded yet while DiskArrays precompiles).
• DynamicBackend is a sum type: current_backend::Union{DefaultBackend,DiskArrayEngineBackend}. Dispatch is union-split into an isa branch.
• The global backend is always a DynamicBackend, since the branch costs nothing measurable (below). The preference only selects the backend that is active at load time, so the "dynamic" value is gone. set_dynamic_backend!(backend) switches for the session, set_backend(name) switches and saves the preference, no restart needed.
• withbackend(array, backend) attaches any other backend to an array via a WithBackendDiskArray wrapper. Entry points now call get_backend(array), which walks up parent until it finds the wrapper, stops at parent(a) === a, and falls back to the global backend. BroadcastStyle only sees the type, so it still uses the global backend.
• Entry points specialize on f. sum(f::Function, a; kwargs...) and friends only pass f through, so Julia did not specialize on it and the inner call was a dynamic dispatch. Unrelated to backends, but it was the only overhead left in this path.

Overhead

2-element AccessCountDiskArray, Julia 1.13, minimum times. The global backend is a DynamicBackend in every row but the last two.

call	time	allocs
sum(a)	61 ns	7
diskarrays_sum_impl(identity, a, get_backend(a))	61 ns	7
diskarrays_sum_impl(identity, a, DefaultBackend()) (hard-coded)	62 ns	7
sum(a) before specializing on f	142 ns	11
get_backend(view(withbackend(a, b), 1:2))	1 ns	0

sum, mapreduce and sum(a .+ 1) all infer Float64, typeof(a .+ 1) infers concretely, and sum(a) contains no dynamic calls.

For comparison, the abstractly typed field from #308, with 4 extra backend methods defined to mimic a loaded ecosystem (with a single method the compiler devirtualizes either design), median times:

backend	time	allocs	inferred	dynamic calls
static DefaultBackend	110 ns	7	Float64	0
DynamicBackend (Union, this PR)	126 ns	7	Float64	0
abstract field (#308)	374 ns	12	Any	1

Tests

test_backend_dispatch no longer needs a preference: it uses withbackend with the TestBackend, checks lookup through a view, runtime switching of the global backend, type stability through DynamicBackend, and the error hint.

Note that test/backend_tests.jl already fails on #308 itself, before reaching this testset (flaky @test_broken unique(...) unexpected passes, median(da, dims=2) on a vector, unqualified mapreducedim!, cumsum without dims, exact == on float sums, count on a float array, and mapreduce(...; dims=1, init=0) on float data, which throws the same InexactError for a plain Array in Base on Julia 1.10, 1.12 and 1.13, so that is not a DiskArrays bug). I left those alone. With them bypassed locally, the dispatch testset and the rest of runtests.jl (including Aqua) pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_018ra7EAJeFE1dtekwf4Tzxh

[meggart]

Thanks a lot. I agree on all of them, but am still a bit unsure about

withbackend(array, backend) attaches any other backend to an array via a WithBackendDiskArray wrapper. Entry points now call get_backend(array), which walks up parent until it finds the wrapper, stops at parent(a) === a, and falls back to the global backend. BroadcastStyle only sees the type, so it still uses the global backend.

It just feels a bit un-natural to me to tie the way you do processing to an array. IMO it would feel much more natural to tie processing engine to an operation instead of to the data. So what about just adding a keyword argument to all user-callable functions defined here to set a backend optionally?

Regarding naming, when I saw withbackend I was very much reminded of the python with and my first expectation was that it would allow something like:

withbackend(DiskArrayEngineBackend) do
  # Inside this block, global backend is set do DAE
  b = sum(a)
  c = count(a)
end

which would be very neat to have and which I would prefer over wrapping arrays.

Also, I intended to add support for multi-arg mapreduce later to avoid complicated zipping, which would make backend selection a bit harder in the future if we select compute backends based on the data.