Skip to content

fix[next-dace]: block state fusion creating a WAR hazard on global data - #2817

Open
edopao wants to merge 3 commits into
GridTools:mainfrom
edopao:fix/state-fusion-war-merge
Open

fix[next-dace]: block state fusion creating a WAR hazard on global data#2817
edopao wants to merge 3 commits into
GridTools:mainfrom
edopao:fix/state-fusion-war-merge

Conversation

@edopao

@edopao edopao commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Description

GT4PyStateFusion merged two states whenever the consumer component of the second state exchanged a messenger with the producer component of the first state, assuming the transient dataflow imposes all necessary orderings. However, the messenger only orders the producer's writes before the consumer's accesses; it does not order the producer's reads of global data before the consumer's writes to the same data.

If the first state reads a global field both upstream and downstream of the messenger it produces, the merged state contains an unordered write-after-read pair on that global, and the computation result becomes undefined.

This pattern is generated by in-place update stencils, e.g. update_theta_and_exner in icon4py:

z_theta = theta_v
theta_v = theta_v + (area * z_temp)
exner = exner * (1.0 + rd_o_cvd * (theta_v / z_theta - 1.0))

which the lowering turns into a compute state followed by a write-back state. The compute state reads theta_v twice — upstream to produce the new value (the messenger) and downstream, together with the messenger, in the exner update. When GT4PyStateFusion merged the write-back state into the compute state, the write theta_v = theta_v_new was dataflow-ordered after the first read of theta_v via the messenger, but not after the second read, since the reader of the messenger and the write-back are siblings downstream of it.

Fix

Add a node-level check to GT4PyStateFusion: when the consumer component writes global data that the producer component reads, the fusion is rejected unless every read of the conflicting global in the first state is guaranteed to be dataflow-ordered before the consumer's write through an exchanged messenger — i.e. there is a messenger that (a) is dataflow-upstream of the write in the second state and (b) every read of the global in the first state is dataflow-upstream of that messenger's producing AccessNode.

This keeps allowing the safe patterns covered by the existing tests (test_global_in_both_read_and_write, test_global_merge_2, test_non_concurrent_data_dependency, test_double_producer), where the write is chained through the messenger.

Tests

  • New regression test test_global_read_up_and_downstream_of_messenger: without the fix the fusion is applied and executes with wrong results (verified numerically: u = (g+1)/(g+1) ≡ 1 instead of (g+1)/g); with the fix the fusion is rejected.
  • All 436 tests in tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/ pass.
  • Validated downstream against the icon4py stencil test suite (dycore + diffusion, dace_cpu backend, 177 tests), which was the original motivation.

GT4PyStateFusion merged two states whenever the consumer component of
the second state exchanged a messenger with the producer component of
the first state, assuming the transient dataflow imposes all necessary
orderings. However, the messenger only orders the producer's writes
before the consumer's accesses; it does not order the producer's reads
of global data before the consumer's writes to the same data.

If the first state reads a global field both upstream and downstream of
the messenger it produces - a pattern generated by in-place update
stencils such as update_theta_and_exner in icon4py - the merged state
contains an unordered write-after-read pair on the global and the
computation result becomes undefined.

Add a node-level check to GT4PyStateFusion that rejects the fusion
unless every read of the conflicting global in the first state is
guaranteed to be dataflow-ordered before the consumer's write through
an exchanged messenger, and a regression test.
@edopao edopao changed the title fix[next-dace]: Block state fusion creating a WAR hazard on global data fix[next-dace]: block state fusion creating a WAR hazard on global data Aug 24, 2026
@edopao
edopao force-pushed the fix/state-fusion-war-merge branch from b68728e to dd5890b Compare August 24, 2026 09:57
@edopao
edopao force-pushed the fix/state-fusion-war-merge branch from dd5890b to e76652f Compare August 24, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant