Skip to content

CUDA: float4 gathers of dense-operand rows in the aligned kernels - #97

Merged
aarmey merged 2 commits into
mainfrom
cuda-float4-gather
Nov 12, 2026
Merged

aarmey merged 2 commits into
mainfrom
cuda-float4-gather

Conversation

@aarmey

@aarmey aarmey commented Nov 12, 2026

Copy link
Copy Markdown
Member

Closes #90.

What

The aligned kernels (VCSR @ B, B @ VCSC) now load each dense-operand row with float4 when width % 4 == 0, so a quarter as many load instructions per nonzero. The inner loop is written once over a vector type (VT), compiled as a third variant axis alongside accumulator count and chunked/whole. Other widths keep scalar loads.

  • width / 4 lanes now cover the width, so _split_factor counts dense columns (ww * vec) against _MIN_SPLIT_WW; width 8 still splits.
  • The misaligned atomicAdd kernels stay scalar. atomicAdd(float4*) needs sm_90+ (not the 4090) and gained only 1-8% on a 5080 in a prototype; the atomics, not the B loads, dominate there.

Measured (µs, parafac2, 5% density; VCSR @ B on 60k x 2k / B @ VCSC on 2k x 60k; before -> after)

width 4090 VCSR@B 4090 B@VCSC 5080 VCSR@B 5080 B@VCSC
8 134 -> 102 142 -> 141 174 -> 115 173 -> 152
16 198 -> 131 197 -> 147 279 -> 155 256 -> 170
32 309 -> 200 311 -> 240 443 -> 216 406 -> 252
64 426 -> 377 451 -> 447 542 -> 380 583 -> 457
128 722 -> 713 865 -> 873 803 -> 721 1194 -> 894
256 1445 -> 1407 1888 -> 1898 1652 -> 1466 2527 -> 1960

Whole @ calls, including the rank-1 correction and scratch reduction. Largest gains at widths 8-64 (up to ~2x on the 5080); on the 4090 B @ VCSC gains little outside widths 16-32.

Testing

  • tests/test_cuda.py passes on both an RTX 4090 and an RTX 5080 (355 tests); added widths 4, 12 and 256 to the split test, plus launch-config tests.
  • ruff, vulture, radon/xenon and jscpd clean for _cuda.py.
  • Not run: ncu profile of the new kernels.

Note: cuda-narrow-indices also touches the kernel signatures, so expect a mechanical conflict with whichever merges second.

…width % 4 == 0

Gathers one B row per nonzero with 128-bit loads instead of four 32-bit
ones. Written once over a vector type; the split threshold counts dense
columns so width 8 still splits. Misaligned atomic kernels stay scalar:
float4 atomicAdd needs sm_90 and gained <10%.
@aarmey
aarmey merged commit 6864bb9 into main Nov 12, 2026
8 checks passed
@aarmey
aarmey deleted the cuda-float4-gather branch November 12, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CUDA kernels: vectorized (float4) loads of dense-operand rows

1 participant