Skip to content

CUDA kernels: profile the misaligned (atomicAdd) kernels and tune their split separately #91

Closed as not planned

Description

@aarmey

Motivation

#85 profiled only the aligned kernels (matmul_delta_csr, rmatmul_delta_csc) with ncu. The misaligned, atomicAdd kernels are unprofiled, and they are now the slower pair relative to cuSPARSE (kernel time over cuSPARSE, benchmarks.run --set cuda):

case ratio
VCSC @ B (atomics) 0.700
B @ VCSR (atomics, unchanged by #85) 0.202

Each call issues nnz x width atomic adds (~96M at width 16). The chunk split from #85 helped VCSC @ B at widths 16 to 32 (0.91x and 0.85x of the previous time) but was neutral at 64 and above (1.01 to 1.02x) and measured 1.07x slower at width 1. The last one is not explained, since width 1 does not split. I tuned one split policy for all four kernels; the misaligned ones may want their own.

To investigate

  • ncu on matmul_delta_csc and rmatmul_delta_csr: atomic throughput, L2 contention, where the time actually goes.
  • Warp-level aggregation when several lanes in a lane-group target the same output element (a scattered cell id rarely repeats, but a scattered gene id does, in B @ VCSR).
  • Vector atomics (sm_90+ only, see the float4 issue CUDA kernels: vectorized (float4) loads of dense-operand rows #90).
  • Whether the misaligned kernels should split at all above some width, and the width-1 slowdown.

Activity

  1. self-assigned this
    on Nov 15, 2026
  2. aarmey commented on Nov 15, 2026

    @aarmey
    MemberAuthor

    Investigated on the RTX 5080 (sm_120), against the kernels after #85/#97/#95. There is no significant speedup left in the misaligned kernels, so I'm closing this.

    Profile (ncu). VCSC @ B, width 16, uniform 60,000 x 2,000 at 5%: 314 us, L2 throughput 93% of peak (L1 hit 37%, L2 hit 97%, DRAM 6%). It is bound by L2 atomic throughput, with nothing left on the SM side to tune. B @ VCSR on a Zipf-skewed matrix (hot genes in most cells): 594 us at only 33% L2 throughput. There the time goes to same-address serialization on the hot genes' output elements, not to bandwidth.

    Warp-level aggregation: not applicable by construction. A warp covers one slice (or one chunk of one), and the minor indices inside a slice are distinct. A cell appears once per gene in VCSC @ B, and a gene once per row in B @ VCSR. So lanes of a warp never target the same output element. The contention in B @ VCSR is across warps (many rows hitting the same hot gene), which warp aggregation cannot reduce.

    Vector atomics (atomicAdd(float4*), sm_90+): implemented and measured, ~0-9% at best. Uniform: 0.95-1.01x at widths 1-64. Skewed: VCSC @ B 0.76x at width 4 and 0.91-0.99x elsewhere. B @ VCSR 1.28x slower at width 4 and 0.96-1.00x elsewhere. This matches the earlier note in _VECTOR_WIDTH: the L2 atomic units are limited by elements updated, not instructions issued. It would also need an sm_90+ gate, so I did not keep it.

    Splitting the misaligned kernels. The chunk split is neutral at width 64+ (VCSC @ B 920 us before and after). The width-1 "1.07x slower" is within run-to-run noise: width 1 does not split, and the whole-slice kernel is unchanged. Re-measured 139 vs 149 us across separate runs. The real problem for these kernels was skew, which is fixed in #100 (#92): skewed VCSC @ B goes 537 -> 113 us at width 1, 1499 -> 151 us at width 4, and 1073 -> 271 us at width 16.

    Not tried: shared-memory privatization of the output for B @ VCSR (output indexed by gene, so hot-gene contention would go away). It only fits when n_genes * width * 4 bytes is under ~48-100 KB (width <= ~6 at 2,000 genes), so it would help narrow widths only. If skewed B @ VCSR at those widths matters downstream, that is the lead to open a new issue for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions