Add -hack-atomic-flag-barrier option

[rjodinchr]

Under the SPIR-V and Vulkan memory model specifications, OpAtomic*
instructions with acquire/release semantics already establish
happens-before ordering for regular memory accesses across threads.
However, some drivers (e.g. under the GLSL450 memory model) fail to
properly enforce this ordering for atomic_flag operations without
explicit memory barriers.

To work around such driver issues, add a new compiler option
-hack-atomic-flag-barrier:

• In replaceAtomicFlagClear: emit an OpMemoryBarrier with Release
  semantics immediately prior to OpAtomicStore when release ordering
  is requested (memory_order_release, memory_order_seq_cst, or default).
• In replaceAtomicFlagTestAndSet:
  • Emit an OpMemoryBarrier with Release semantics immediately prior
    to OpAtomicExchange when release ordering is requested
    (memory_order_release, memory_order_acq_rel, memory_order_seq_cst,
    or default).
  • Emit an OpMemoryBarrier with Acquire semantics immediately after
    OpAtomicExchange when acquire ordering is requested
    (memory_order_acquire, memory_order_acq_rel, memory_order_seq_cst,
    or default).
• Do not emit OpMemoryBarrier when memory_order_relaxed is specified,
  as Vulkan SPIR-V forbids OpMemoryBarrier with semantics 0 (None).

[alan-baker]

Can you link to some spec sections that say this. It's been a long time since I've looked at the OpenCL memory model.

[rjodinchr]

Here are the relevant specification sections covering both the OpenCL memory model requirements and why Vulkan SPIR-V under GLSL450 requires explicit OpMemoryBarrier instructions:

---

1. OpenCL Specification

A. Atomic Memory Orders Govern Regular (Non-Atomic) Memory

In the OpenCL C 3.0 Specification, Section 6.15.2 "Order and Consistency":

"The enumerated type memory_order specifies the detailed regular (non-atomic) memory synchronization operations as defined in section 5.1.2.4 of the C11 Specification, and may provide for operation ordering."

And for atomic_flag operations specifically in Section 6.15.7 "Operations on Atomic Types":

• atomic_flag_test_and_set: "Memory is affected according to the value of order. These operations are atomic read-modify-write operations (as defined by section 5.1.2.4 of the C11 Specification)."
• atomic_flag_clear: "Memory is affected according to the value of order."

B. Happens-Before & Visibility Across Threads

In the OpenCL API Specification, Section 3.3.4 "Memory Consistency Model":

• Section 3.3.4.1 "Overview of Atomic and Fence Operations":

  "An atomic operation on one or more memory locations is either an acquire operation, a release operation, or both an acquire and release operation... Informally, executing a memory_order_release on an atomic object A makes all previous side effects visible to any unit of execution that later executes a memory_order_acquire on A."

• Section 3.3.4.2 "Memory Ordering Rules":

  • Synchronizes-with: "An atomic operation A that performs a release operation on a global object M global-synchronizes-with an atomic operation B that performs an acquire operation on M and reads a value written by any side effect in the release sequence headed by A."
  • Happens-before: *"A global memory action A global-happens-before a global memory action B if:
    1. A is sequenced before B, or
    2. A global-synchronizes-with B, or
    3. For some global memory action C, A global-happens-before C and C global-happens-before B."*
  • Visibility of non-atomic memory: "The value of a non-atomic scalar object M, as determined by evaluation B, shall be the value stored by the visible side effect A."
  • Data race: "The execution of a program contains a data race if it contains two conflicting actions A and B in different units of execution, and (1) at least one of A or B is not atomic... and (2) the actions are global actions unordered by the global-happens-before relation..."

---

2. Vulkan SPIR-V Specification

A. Memory Semantics Definition in SPIR-V

In the SPIR-V Specification, Section 3.25 "Memory Semantics ":

• Acquire (0x2): "All memory operations in this thread before this instruction are sequenced-before this instruction. All memory operations in another thread sequenced-before the release operation that synchronizes with this instruction are synchronized with this instruction."
• Release (0x4): "All memory operations in this thread sequenced-before this instruction are synchronized with the acquire operation in another thread that synchronizes with this instruction."
• UniformMemory (0x40): "Apply the memory-ordering constraints to Uniform storage class and Output storage class..."
• WorkgroupMemory (0x100): "Apply the memory-ordering constraints to Workgroup storage class."

B. Vulkan GLSL450 Memory Model and Cache Coherency

In the Vulkan Specification, Appendix "Vulkan Environment for SPIR-V - Memory Model" and Section 15.6 "Shader Memory Access Ordering":

Under the standard Vulkan GLSL450 memory model (OpMemoryModel Logical GLSL450, without VulkanMemoryModelKHR):

Atomic operations (OpAtomic*) guarantee atomicity only with respect to other atomic accesses to the same memory location, but do not automatically act as memory barriers or perform availability/visibility operations for non-atomic memory accesses across cores/SMs.

On GPU architectures with non-coherent L1 caches and store buffers (such as NVIDIA), atomic instructions bypass L1 and go directly to L2/device memory, whereas ordinary buffer loads/stores go through non-coherent per-SM L1 caches. In the GLSL450 model, drivers only insert the required cache flush/invalidate instructions (e.g., membar.gl / membar.cta) when encountering explicit memory barriers (OpMemoryBarrier).

Therefore, to guarantee that non-atomic global/local memory modifications sequenced before an atomic release are visible to threads after a corresponding atomic acquire (as mandated by the OpenCL C memory model), an explicit OpMemoryBarrier with Release semantics must precede release stores, and an OpMemoryBarrier with Acquire semantics must follow acquire operations.

[alan-baker]

In Vulkan, it should be equivalent to write

atomic_store(x, 0, order=release, scope=workgroup);

Or:

atomic_store(x, 0, order=relaxed, scope=workgroup);
fence(order=release, storage_classes=WorkgroupMemory);

The declared memory model doesn't change Vulkan's behaviour. It changes what synchronizations can be expressed and how some properties are expressed.

Here are two litmus tests I wrote for the Vulkan memory model in Alloy:
Using atomics

NEWWG
NEWSG
NEWTHREAD
st.av.scopewg.sc0 x = 1
st.atom.rel.scopewg.sc0.semsc0 y = 1
NEWSG
NEWTHREAD
ld.atom.acq.scopewg.sc0.semsc0 y = 1
ld.vis.scopewg.sc0 x = 1

SATISFIABLE consistent[X] && #dr=0
NOSOLUTION consistent[X] && #dr>0

The model says this is always race free.

Using fences

NEWWG
NEWSG
NEWTHREAD
st.av.scopewg.sc0 x = 1
membar.rel.scopewg.semsc0
st.atom.scopewg.sc0 y = 1
NEWSG
NEWTHREAD
ld.atom.scopewg.sc0 y = 1
membar.acq.scopewg.semsc0
ld.vis.scopewg.sc0 x = 1

SATISFIABLE consistent[X] && #dr=0
NOSOLUTION consistent[X] && #dr>0

This is also always race free.

When the memory model is GLSL450, coherence takes the place of availability and visibility (and non-private). Workgroup is always coherent and storage buffers must be marked as coherent.

Note: the non-atomic memory accesses could be in a different storage class as long as the atomics/fences include that storage class in their semantics. E.g. put x in sc1 and add semsc1 to the acq and rel instructions.

So it don't really follow why we'd need both an ordered atomic and a fence. Are we not marking the right buffers as coherent? Do we not include the correct semantics on the atomic? Are you certain this is not just a driver bug? Am I misunderstanding your description of the problem?

[rjodinchr]

Thanks for the explanation and Alloy tests! You're completely right that ordered atomics already order regular memory per spec, making the extra barrier redundant.

Since this seems to be a driver bug, I've moved the OpMemoryBarrier generation behind a new opt-in -hack-atomic-flag-barrier option (disabled by default).