Skip to content

perf(prover): default the cuda table scheduler to K = num_airs - #911

Open
MauroToscano wants to merge 2 commits into
mainfrom
perf/table-parallelism-num-airs
Open

perf(prover): default the cuda table scheduler to K = num_airs#911
MauroToscano wants to merge 2 commits into
mainfrom
perf/table-parallelism-num-airs

Conversation

@MauroToscano

Copy link
Copy Markdown
Contributor

The cuda arm of table_parallelism() scales K by available_parallelism()
(cores * 2 / 3). Measured over 881 runs on two RTX 5090 boxes, that is the wrong
shape — not the wrong constant.

All eight core-count curves fit T(K) = S + max(Tmax, W/K) within run-to-run noise,
and the work K divides (W ≈ 5.3–8.0 s) is invariant to host core count over an 8×
range, to CPU model, and to rayon pool width. The decisive one: cutting
RAYON_NUM_THREADS 32 → 4 leaves W alone and merely doubles S, with the best K still
num_airs at every pool width. available_parallelism() sizes precisely that rayon
pool, so it is the wrong quantity to scale K by — K is not a thread count, each
table's work runs on the one global pool.

Worst case against the best measured K, over four core counts on both boxes:

rule worst case
cores/3 +30.2%
cores*2/3 (what this replaces) +13.0%
constant 12 +7.0%
num_airs +1.6%

cores*2/3 fails where predicted: low core counts, K=2 at 4 cores (+13.0%), K=5 at 8
cores (+8.1%). No K below num_airs is significantly faster than num_airs anywhere
on either box (every such cell p ≥ 0.38 at n=8).

Taking the ceiling rather than solving for an optimum is right in both regimes of the
fit: if W/num_airs > Tmax more K strictly helps; if W/num_airs < Tmax the extra
drivers are floor-limited and cost nothing — the one staging slab is held 56% of wall
at K=31 and wall time still improves.

The old doc comment's mechanism ("in-flight tables mostly sit in GPU waits") is not
what happens — mean GPU utilisation never exceeded ~38% at any K — so it is rewritten
rather than re-tuned. What is meant to bound concurrency is memory admission rather
than a count: that is what VramGate is for.

auto_storage is held bounded

table_parallelism now takes num_airs and clamps to it, replacing the .min(num_airs)
the call site applied. auto_storage::decide keeps a bounded figure through the new
storage_estimate_parallelism(): peak_bytes sums the transient bytes of the top-k
tables, so an unbounded k there sums every table — measured +27% at 128 PAGE tables,
+44% at 512 — and would spill proofs to disk that fit in RAM. Its value is unchanged,
so no storage decision moves. Two tests pin both halves.

The CPU arm is untouched

cores / 3 stays. The sweep ran only on cuda builds, where the parallelized work is
device-bound; on a CPU-only build every table is pure host work and none of this
evidence transfers.

Evidence

scripts/profiling/table-parallelism-sweep/ — both rounds' write-ups, result CSVs and
re-runnable harnesses, so every number above is checkable.

Honest limits

  • One GPU model (RTX 5090) and one workload throughout: 6.8M cycles, 4 epochs at
    epoch-size-log2 21, ~31 tables. Production tables are much larger, so memory per
    concurrently-admitted table is larger and the auto_storage interaction is unverified
    at that scale — a big-block check is a merge gate, not something this PR establishes.
  • Low-core legs are cache-confounded against round 1 (this box's taskset -c 0-7 is one
    CCD with 32 MB L3 vs the 3D part's much larger cache).
  • W is inferred from T(1) − T(best), not instrumented per phase.
  • Round 1's headline "knee K≈8" should not be quoted: the curve is S + W/K with no
    saturating resource, so a tolerance-knee is derived and reads 8 on one box and 16 on
    the other at the same 5% tolerance.

Verified

cargo fmt --check clean; cargo clippy --all-targets -p stark -p lambda-vm-prover
clean. auto_storage_tests 8 passed (needs --features disk-spill);
table_parallelism_stays_within_one_and_num_airs passes. The cuda-gated test and the
cuda arm itself cannot compile here — no local CUDA toolchain — so both were verified by
reading plus a non-cuda build.

`table_parallelism()`'s cuda arm scaled K by `available_parallelism()`
(`cores * 2 / 3`). Measured over 881 runs on two RTX 5090 boxes, that is the
wrong shape. All eight core-count curves fit `T(K) = S + max(Tmax, W/K)` within
run-to-run noise, and the work K divides — W ≈ 5.3-8.0 s — is invariant to host
core count over an 8x range, to CPU model, and to rayon pool width: cutting
RAYON_NUM_THREADS 32 -> 4 leaves W alone and merely doubles S, with the best K
still num_airs at every pool width. `available_parallelism()` sizes precisely
that rayon pool, so it is the wrong quantity to scale K by. K is not a thread
count; each table's work runs on the one global pool.

Worst case against the best measured K, over four core counts on both boxes:

  cores/3      +30.2 %
  cores*2/3    +13.0 %   (what this replaces)
  constant 12   +7.0 %
  num_airs      +1.6 %   (both non-zero cells inside noise, p = 0.88 / 0.80)

`cores*2/3` fails where it was predicted to: low core counts, K=2 at 4 cores
(+13.0 %) and K=5 at 8 cores (+8.1 %).

Taking the ceiling rather than solving for an optimum is right in both regimes
of the fit: if W/num_airs > Tmax more K strictly helps, and if W/num_airs < Tmax
the extra drivers are floor-limited and cost nothing — the one staging slab is
held 56 % of wall at K=31 and wall time still improves. The old doc comment's
mechanism ("in-flight tables mostly sit in GPU waits") is not what happens —
mean GPU utilisation never exceeded ~38 % at any K — so it is rewritten rather
than re-tuned. What is meant to bound concurrency is memory admission rather
than a count: that is what VramGate is for, and it never binds at the default
budget.

`table_parallelism` now takes `num_airs` and clamps to it, replacing the
`.min(num_airs)` the call site applied. `auto_storage::decide` keeps a bounded
figure through the new `storage_estimate_parallelism()`: `peak_bytes` sums the
transient bytes of the top-k tables, so an unbounded k there sums every table —
measured +27 % at 128 PAGE tables, +44 % at 512 — and would spill proofs to disk
that fit in RAM. Its value is unchanged, so no storage decision moves.

The CPU arm keeps `cores / 3`. The sweep ran only on cuda builds, where the
parallelized work is device-bound; on a CPU-only build every table is pure host
work and none of this evidence transfers.
The measurement behind the previous commit, under
`scripts/profiling/table-parallelism-sweep/`: both write-ups, every timing CSV
(881 runs over two RTX 5090 boxes, one row per run), the analysis scripts, the
server-side sweep harnesses, the box/toolchain fingerprints, the frozen job
orders and the diffs for the three experimental builds. `rules2.py` reproduces
the rule-cost table from the committed CSVs; `amdahl.py` reproduces the fits.

`thoughts/` is gitignored, so this lands next to the profiling tooling instead.
Build logs, the per-run logs for the stages whose hypotheses were refuted, and
the Stage 1a stack samples (gdb could not attach — the rented container dropped
cap_sys_ptrace) are left out; the README says what is included and what is not.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant