Skip to content

Commit 5b45d55

Browse files
author
Cong Zhang
committed
[cl] Port the SM120 FMHA context kernel and clustered FP16/BF16 GEMM ts tutorial from DKG
1 parent c4543df commit 5b45d55

35 files changed

Lines changed: 5852 additions & 571 deletions

‎experimental/cuda-lang/tutorial/fmha_prefill.py‎

Lines changed: 35 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,26 @@
11
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
22
# SPDX-License-Identifier: Apache-2.0
33

4-
"""Blackwell FMHA-prefill port from CUTLASS Python DSL to CUDA Lang.
4+
"""Blackwell FMHA-prefill CUDA Lang tutorial.
55
66
This is a native fused implementation, with no SIMT, PyTorch-SDPA, library,
7-
source-kernel, or precompiled-PTX fallback. It supports the source tutorial's
8-
fixed and variable sequence storage, D=32/64/128, FP16/BF16/E4M3 inputs and
9-
outputs, MHA/GQA, mask variants, LSE, sink logits, scaling controls, and
10-
correction/epilogue options.
7+
or precompiled-PTX fallback. It supports fixed and variable sequence storage,
8+
D=32/64/128, FP16/BF16/E4M3 inputs and outputs, MHA/GQA, mask variants, LSE,
9+
sink logits, scaling controls, and correction/epilogue options.
1110
12-
The device kernel retains the source's structural contract:
11+
The device kernel uses:
1312
1413
* one 512-thread CTA and one CTA per ``(256 Q rows, Q head, batch)`` tile;
1514
* softmax warps 0-7, correction warps 8-11, MMA warp 12, TMA warp 13,
1615
epilogue warp 14, and CLC scheduler warp 15;
1716
* CTA-group-1 tcgen05 QK (128x128x16) and PV (128x64x16);
1817
* two Q stages, three unified K/V stages, two output stages, and all 512
19-
TMEM columns with the source S/P/O aliases;
18+
TMEM columns with S/P/O aliases;
2019
* online softmax, packed-FP16 P in TMEM, FP32 O correction in TMEM, TMA
2120
global/shared transfers, and a one-slot 480-consumer CLC mailbox.
2221
23-
The source-compatible host configuration, chunked reference, correctness
24-
matrix, cold-L2/CUDA-graph benchmark, and CLI share this public entry point.
22+
The host configuration, chunked reference, correctness matrix,
23+
cold-L2/CUDA-graph benchmark, and CLI share this public entry point.
2524
Invalid combinations raise clear diagnostics and are never silently redirected
2625
to another attention implementation.
2726
"""
@@ -376,9 +375,8 @@ def _store_output_values(
376375
word_count = 16
377376
if output_kind == ELEMENT_E4M3:
378377
word_count = 8
379-
# One PTX shared-memory vector store carries four 32-bit registers. The
380-
# swizzle is constant within each 16-byte group, so form the same v4
381-
# transfer units as CUTLASS's store_swizzled lowering.
378+
# One PTX shared-memory vector store carries four 32-bit registers. The
379+
# swizzle is constant within each 16-byte group, so form v4 transfer units.
382380
for group in cl.static_iter(range(word_count // 4)):
383381
byte_offset = (
384382
slice_index * MMA_M * slice_bytes
@@ -421,8 +419,8 @@ def _store_output_global(
421419
alignment=1,
422420
)
423421
else:
424-
# CUTLASS keeps the converted FP16/BF16 lanes scalar on the guarded
425-
# tail path, which lowers to one b16 store per valid element.
422+
# Keep converted FP16/BF16 lanes scalar on the guarded tail path, which
423+
# lowers to one b16 store per valid element.
426424
dst_u16 = cl.bitcast(
427425
dst,
428426
cl.pointer_dtype(cl.uint16, cl.MemorySpace.GLOBAL),
@@ -489,12 +487,12 @@ def _key_trip_context(
489487
window_left,
490488
window_right,
491489
):
492-
"""Return the source super-tile K interval and its Q0 invalid tail.
490+
"""Return the super-tile K interval and its Q0 invalid tail.
493491
494-
CUTLASS prunes whole 128-column K/V tiles using the union of the two
495-
128-row Q halves. Q0 can therefore have one fully-invalid trailing tile
496-
that Q1 still needs. That tile remains in the barrier protocol but does
497-
not issue QK0/PV0 math.
492+
Whole 128-column K/V tiles are pruned using the union of the two 128-row Q
493+
halves. Q0 can therefore have one fully-invalid trailing tile that Q1 still
494+
needs. That tile remains in the barrier protocol but does not issue QK0/PV0
495+
math.
498496
"""
499497
query_begin = seq_tile * CTA_M
500498
offset = cl.int32(0)
@@ -792,8 +790,8 @@ def _fmha_prefill_kernel(
792790
sched_valid = cl.shared_array(1, cl.int32, alignment=4)
793791

794792
tid = cl.thread_index(0)
795-
# Match CUTLASS's make_warp_uniform(warp_idx): broadcasting lane 0's
796-
# value lets the backend treat every role predicate as warp-uniform.
793+
# Broadcasting lane 0's warp index lets the backend treat every role
794+
# predicate as warp-uniform.
797795
warp = cl.shfl_sync(tid // WARP_SIZE, 0)
798796
lane = tid % WARP_SIZE
799797
if variable_length:
@@ -855,9 +853,9 @@ def _fmha_prefill_kernel(
855853
l2_promotion=cl.TensorMapL2Promotion.NONE,
856854
)
857855

858-
# CUTLASS initializes Blackwell pipeline stages cooperatively with warp 0.
859-
# Dynamic lane indexing keeps one static PTX site per pipeline half while
860-
# still initializing every physical barrier exactly once.
856+
# Initialize pipeline stages cooperatively with warp 0. Dynamic lane
857+
# indexing keeps one static PTX site per pipeline half while initializing
858+
# every physical barrier exactly once.
861859
if warp == SOFTMAX0_WARPS[0]:
862860
if lane < Q_STAGES:
863861
cl.mbarrier_initialize(q_full.get_element_pointer(lane), 1)
@@ -1313,8 +1311,8 @@ def _fmha_prefill_kernel(
13131311
kv_index = 0
13141312
kv_full_phase = 0
13151313
while valid:
1316-
# CUTLASS reuses one elected MMA-warp lane for all UMMA math in a
1317-
# work tile. Pipeline commits elect independently in their helper.
1314+
# Reuse one elected MMA-warp lane for all UMMA math in a work tile.
1315+
# Pipeline commits elect independently in their helper.
13181316
mma_elect_one = cl.elect_sync()
13191317
query_base, key_base, task_seq_q, task_seq_k = _sequence_context(
13201318
cumulative_q,
@@ -1364,7 +1362,7 @@ def _fmha_prefill_kernel(
13641362

13651363
# QK0(0), QK1(0): both halves share K0 before its ring slot is
13661364
# released. The acquire mbarrier wait orders the TMA async writes
1367-
# before the tcgen MMA, matching CUTLASS's TMA-to-UMMA pipeline.
1365+
# before the tcgen MMA in the TMA-to-UMMA pipeline.
13681366
k_stage = kv_index
13691367
_wait_mbarrier(kv_full.get_element_pointer(k_stage), kv_full_phase)
13701368
k_desc = _qk_descriptor(
@@ -1842,9 +1840,9 @@ def _fmha_prefill_kernel(
18421840
for chunk in cl.static_iter(range(4))
18431841
)
18441842

1845-
# Match CUTLASS's four 32-value vector reductions:
1846-
# MAXIMUMF propagates NaNs within each chunk, while the
1847-
# outer maxnum combines retain the source's semantics.
1843+
# Use four 32-value vector reductions. MAXIMUMF propagates
1844+
# NaNs within each chunk, while the outer maxnum combines
1845+
# retain the required semantics.
18481846
new_row_max = cl.float32(-float("inf"))
18491847
for chunk in cl.static_iter(score_chunks):
18501848
chunk_max = chunk[:32].reduce(
@@ -1878,18 +1876,16 @@ def _fmha_prefill_kernel(
18781876
stats_vec,
18791877
)
18801878
# Acquire the first P-ready stage while the statistics
1881-
# store is in flight. The second acquire and STTM wait
1882-
# are split into the initial FMA lookahead below, matching
1883-
# CUTLASS's machine schedule.
1879+
# store is in flight. The second acquire and STTM wait are
1880+
# split into the initial FMA lookahead below.
18841881
_wait_mbarrier(
18851882
p_empty.get_element_pointer((qid, 0)),
18861883
1 ^ (pv_event & 1),
18871884
)
18881885

18891886
block_sums = cl.Vector(0.0, 0.0, dtype=cl.float32)
18901887
minus_safe_max_scale = -safe_max * scale_softmax_log2
1891-
# Keep the DKG constexpr-loop topology while using flattened
1892-
# local arrays in place of its mutable Python lists.
1888+
# Use constexpr loops with flattened local arrays.
18931889
if input_kind == ELEMENT_E4M3:
18941890
p_window_elems = 4
18951891
fma_pipe_windows = 3
@@ -3211,10 +3207,10 @@ def benchmark(
32113207
return 0.0
32123208

32133209
if use_cuda_graphs:
3214-
# Match the CUTLASS testing helper's semantics: capture the complete
3215-
# warmup and profiling loops on a non-default stream, replay each graph
3216-
# once, and divide the profiled graph time by its launch count. Each
3217-
# workspace has static graph arguments, so cold-L2 rotation is retained.
3210+
# Capture the complete warmup and profiling loops on a non-default
3211+
# stream, replay each graph once, and divide the profiled graph time by
3212+
# its launch count. Each workspace has static graph arguments, so
3213+
# cold-L2 rotation is retained.
32183214
workspace_index = 0
32193215
if warmup_iterations:
32203216
warmup_graph = torch.cuda.CUDAGraph()

‎experimental/cuda-lang/tutorial/fmha_prefill_helpers.py‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,9 @@
33

44
"""Shared helpers for the CUDA Lang Blackwell FMHA-prefill tutorial.
55
6-
This module mirrors DKG's single fmha_prefill_helpers.py dependency. It
7-
contains the pure-Python mask/work-tile formulas, specialization traits,
8-
CUDA Lang leaf device helpers, and host-side tensor/reference utilities used
9-
by fmha_prefill.py and its focused tests.
6+
This module contains the pure-Python mask/work-tile formulas, specialization
7+
traits, CUDA Lang leaf device helpers, and host-side tensor/reference utilities
8+
used by fmha_prefill.py and its focused tests.
109
"""
1110

1211
from __future__ import annotations
@@ -1218,9 +1217,9 @@ def _validate_output_traits(traits: OutputTraits) -> None:
12181217
# Leaf CUDA Lang device helpers.
12191218
# ---------------------------------------------------------------------------
12201219

1221-
# CUTLASS Swizzle<B, 4, 3> uses B={1,2,3} for 32/64/128-byte TMA
1222-
# swizzles. The XOR source bits start at byte-address bit 7 and the
1223-
# destination bits start at bit 4.
1220+
# Swizzle<B, 4, 3> uses B={1,2,3} for 32/64/128-byte TMA swizzles. The XOR
1221+
# source bits start at byte-address bit 7 and the destination bits start at
1222+
# bit 4.
12241223
SWIZZLE_32B_BITS = 1
12251224
SWIZZLE_64B_BITS = 2
12261225
SWIZZLE_128B_BITS = 3
@@ -1312,7 +1311,7 @@ def unpack_e4m3_lane(packed, lane):
13121311

13131312
@cl.function
13141313
def swizzle_byte_offset(byte_offset, swizzle_bits):
1315-
"""Apply CUTLASS ``Swizzle<B,4,3>`` to a slice-local byte offset.
1314+
"""Apply ``Swizzle<B,4,3>`` to a slice-local byte offset.
13161315
13171316
The operation is its own inverse because it XORs bits 4..6 using bits
13181317
7..9, which are not modified. ``swizzle_bits`` must be 1, 2, or 3.
@@ -1906,8 +1905,8 @@ def prepare_tensors(
19061905
"""Allocate fixed or flattened-varlen tensors on ``device``.
19071906
19081907
Custom Q/K/V inputs are interpreted in storage order and must be supplied
1909-
together. Reference inputs retain the original FP32 source values, matching
1910-
the CUTLASS example's ``create_and_pad_tensor`` contract.
1908+
together. Reference inputs retain the original FP32 values used for the
1909+
correctness calculation.
19111910
"""
19121911

19131912
torch_module = _require_torch()

‎experimental/cuda-lang/tutorial/fp16_gemm_0.py‎

Lines changed: 10 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,57 +1,15 @@
11
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
22
# SPDX-License-Identifier: Apache-2.0
33

4-
"""CUDA Lang port of the CuTe DSL ``fp16_gemm_0.py`` tutorial.
5-
6-
Computes ``C = A @ B.T`` with an optional FP16 row bias. The kernel retains
7-
the source tutorial's 128x128x64 single-CTA baseline: one shared-memory stage,
8-
explicit TMA and mbarrier operations, tcgen05 MMA into TMEM, FP32 accumulation,
9-
and an FP16 epilogue.
10-
11-
CuTe DSL -> CUDA Lang API mapping used here:
12-
13-
* ``@cute.kernel`` maps to ``@cl.kernel``. CuTe's ``warp_idx``, thread/block
14-
indices, ``ceil_div``, and ``range_constexpr`` map to the corresponding
15-
``cl`` index operations, ``cl.cdiv``, and ``cl.static_iter``.
16-
* ``cutlass.Array(..., space=cutlass.AddressSpace.smem)`` maps to
17-
``cl.shared_array``. The A/B arrays are flattened in CUDA Lang, but retain
18-
the source sizes, 128-byte alignment, and physical shared-memory layout.
19-
* ``cuda.create_tensor_map_tiled_from_view`` maps to ``cl.tensor_map_tiled``.
20-
A and B remain row-major, K-contiguous 2-D tensors. CUDA Lang expresses the
21-
TMA coordinate order explicitly as ``(K, M)`` / ``(K, N)`` using tile shapes
22-
``(BLOCK_K, BLOCK_M)`` / ``(BLOCK_K, BLOCK_N)`` and ``order="F"``; this is
23-
equivalent to CuTe DSL automatically selecting K as the leading mode.
24-
* ``prims.cp_async_bulk_tensor_shared_cta_global`` maps to
25-
``cl.copy_async_bulk_tensor_global_to_shared``. The expected transaction
26-
byte count and the TMA-completion mbarrier are unchanged.
27-
* ``prims.mbarrier_init``, ``mbarrier_arrive_expect_tx``, and
28-
``mbarrier_try_wait_parity`` map to ``cl.mbarrier_initialize``,
29-
``cl.mbarrier_arrive_expect_transaction``, and
30-
``cl.mbarrier_wait_parity``. The full/empty/accumulator barrier topology
31-
and phase progression are preserved.
32-
* ``prims.Tcgen05InstrDesc.build`` and ``Tcgen05SmemDesc.build`` map to
33-
``cl.Tcgen05InstructionDescriptor`` and
34-
``cl.Tcgen05SharedMemoryDescriptor``, each followed by ``encode()``. Both
35-
builders take byte-valued leading/stride offsets. Once encoded, the per-MMA
36-
K-step is added in descriptor units of 16 bytes, hence ``2 * kk`` for each
37-
32-byte F16 MMA K-step.
38-
* ``prims.tcgen05_alloc``, ``tcgen05_mma``, ``tcgen05_commit``,
39-
``tcgen05_ld``, and ``tcgen05_dealloc`` map directly to the corresponding
40-
``cl.tcgen05_*`` operations. Allocation size, MMA accumulation predicate,
41-
commit barriers, 32-column TMEM loads, and deallocation size match the
42-
source.
43-
* The CuTe ``@cute.jit`` host function builds tensor maps and launches the
44-
device kernel. CUDA Lang instead declares tensor maps in ``@cl.kernel``;
45-
compilation hoists their construction into the generated host program, so
46-
both public launch interfaces still accept ordinary 2-D tensors.
47-
* ``cute.compile`` has no explicit CUDA Lang counterpart here. ``run()`` calls
48-
``cl.launch`` directly; the first launch compiles the kernel and later
49-
launches reuse CUDA Lang's internal cache. K and ``has_bias`` are
50-
``cl.Constant`` specialization keys, while M and N remain runtime dimensions
51-
of C.
52-
53-
The CLI retains the source tutorial's correctness-checking interface. CUDA
54-
Lang-specific restrictions are validated explicitly.
4+
"""Single-CTA CUDA Lang FP16 GEMM tutorial.
5+
6+
Computes ``C = A @ B.T`` with an optional FP16 row bias using a 128x128x64
7+
tile, one shared-memory stage, explicit TMA and mbarrier operations, tcgen05
8+
MMA into TMEM, FP32 accumulation, and an FP16 epilogue. A and B are row-major,
9+
K-contiguous tensors; ``order="F"`` expresses K as the leading TMA coordinate.
10+
Tensor maps declared in ``@cl.kernel`` are hoisted into the generated host
11+
program. K and ``has_bias`` are specialization keys, while M and N remain
12+
runtime dimensions of C. The CLI validates inputs and checks correctness.
5513
"""
5614

5715
from __future__ import annotations
@@ -100,8 +58,7 @@ def _kernel(
10058

10159
# A/B are ordinary row-major (rows, K) arrays. ``order="F"`` reverses the
10260
# tensor-map modes to (K, rows), making K the contiguous TMA coordinate.
103-
# This is the explicit CUDA Lang equivalent of CuTe DSL's leading-mode
104-
# detection in create_tensor_map_tiled_from_view.
61+
# The explicit mode order makes K the leading tensor-map dimension.
10562
a_tmap = cl.tensor_map_tiled(
10663
a,
10764
(BLOCK_K, BLOCK_M),

‎experimental/cuda-lang/tutorial/fp16_gemm_1.py‎

Lines changed: 5 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,17 @@
11
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
22
# SPDX-License-Identifier: Apache-2.0
33

4-
"""CUDA Lang port of the CuTe DSL ``fp16_gemm_1.py`` tutorial.
4+
"""Two-CTA clustered CUDA Lang FP16 GEMM tutorial.
55
66
Computes ``C = A @ B.T`` with an optional FP16 row bias. Two CTAs collaborate
77
on each 256x256 output tile: each CTA loads one 128-row A slice and one
88
128-column B slice, the leader issues CTA_2 tcgen05 MMA instructions, and both
99
CTAs store their 128 output rows.
1010
11-
Additional CuTe DSL -> CUDA Lang notes beyond ``fp16_gemm_0.py``:
12-
13-
* CuTe DSL differentiates CTA-local and cluster TMA with different primitive
14-
names: ``cp_async_bulk_tensor_shared_cta_global`` in ``fp16_gemm_0.py`` and
15-
``cp_async_bulk_tensor_shared_cluster_global`` here. CUDA Lang uses the same
16-
``cl.copy_async_bulk_tensor_global_to_shared`` API for both; cluster TMA is
17-
selected by passing a destination pointer in
18-
``cl.MemorySpace.SHARED_CLUSTER`` plus the CTA_2 multicast arguments.
19-
Completion still has to arrive on the leader CTA's mbarrier, so this sample
20-
maps the local ``ab_full`` pointer with ``cl.map_shared_to_leader_block``.
21-
* The source kernel takes K as a runtime argument. This sample keeps M, N, and K
22-
runtime dimensions; only the optional-bias path is specialized.
23-
24-
The implementation otherwise retains the source's cluster shape, work
25-
decomposition, data movement, barrier phases, CTA_2 MMA sequence, TMEM
26-
epilogue, and vector stores.
11+
Cluster TMA uses a destination pointer in ``cl.MemorySpace.SHARED_CLUSTER``
12+
with CTA_2 multicast arguments. Completion arrives on the leader CTA's
13+
mbarrier, mapped with ``cl.map_shared_to_leader_block``. M, N, and K remain
14+
runtime dimensions; only the optional-bias path is specialized.
2715
"""
2816

2917
from __future__ import annotations

‎experimental/cuda-lang/tutorial/fp16_gemm_2.py‎

Lines changed: 4 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,16 @@
11
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
22
# SPDX-License-Identifier: Apache-2.0
33

4-
"""CUDA Lang port of the CuTe DSL ``fp16_gemm_2.py`` tutorial.
4+
"""Two-CTA clustered CUDA Lang FP16 GEMM tutorial with FP32 output.
55
66
Computes ``C = A @ B.T`` with an optional FP32 row bias. Two CTAs collaborate
77
on each 256x128 output tile: each CTA loads one 128-row A slice and one
88
64-column B slice, the leader issues CTA_2 tcgen05 MMA instructions, and both
99
CTAs store their 128 output rows.
1010
11-
Additional CuTe DSL -> CUDA Lang notes beyond ``fp16_gemm_1.py``:
12-
13-
* The collective N tile is 128 instead of 256. CTA_2 still splits the tile
14-
across the two CTA ranks, so each CTA loads a 64-column B slice.
15-
* C and the optional row bias are FP32. The epilogue stores FP32 accumulator
16-
vectors directly instead of converting them to FP16.
17-
18-
The implementation otherwise retains the source's cluster shape, work
19-
decomposition, data movement, barrier phases, CTA_2 MMA sequence, TMEM
20-
epilogue, and vector stores.
11+
The collective N tile is 128 and is split across the two CTA ranks, so each CTA
12+
loads a 64-column B slice. C and the optional row bias are FP32; the epilogue
13+
stores accumulator vectors directly without an FP16 conversion.
2114
"""
2215

2316
from __future__ import annotations

0 commit comments

Comments
 (0)