bench(zero-cache): N-group wave-replay harness and scaling baseline

[Karavil]

Benchmark tooling: a concurrent client-group wave-replay harness for zero-cache, a committed synthetic fixture, temporary stage probes on the sync path, and a measured baseline with raw results in the tree.

What it measures. N fresh client groups each register the same six query roots (a classroom app's assignment-detail page) in one same-tick batch, against one sync worker, and the harness records how the query wave scales with N. The reproduced result is a scaling law, not a time: query-wave wall clock grows near-linearly with concurrent client groups while the CPU-ish per-group query time stays flat, because every ViewSyncer on a worker shares one cooperative time-slice queue and TimeSliceTimer stops while a ViewSyncer yields. Optimization branches should compare ratio columns; the milliseconds belong to whichever machine produced them.

The workload is entirely synthetic: emu-* identifiers, Emulation Student NNN display names, synthetic addresses. No production data. It is production-shaped -- the fixture (136 students, 5 classes, 24 problems, 973 problem trackers, 973 conversations, 957 mastery assessments) and the six roots come from a real educator-facing page in the Goblins app, whose query mix motivated this work. This branch exists so that the perf branches built against this workload (#6432, #6433, #6434, #6435, #6436, #6437 all cite it) have a committed workload definition, fixture, and rerun recipe in the tree, instead of numbers pinned to uncommitted scripts on one machine: the profiling investigation that established the scaling law ran exactly that way. That investigation is referred to as "the diagnosis" in the committed results notes and in the probe event names below; its scripts and replica are not committed anywhere, and its numbers appear here only as comparison columns. The rerun recipe has one external dependency this repo cannot supply -- the source application's transform server and schema -- spelled out under "Reruns, and an honest dependency" below.

The commits

1. perf(zero-cache): add temporary stage probes to the assignment-wave hot path. Measurement scaffolding, not a behavior change: every addition is a performance.now() delta and an lc.info line, and nothing on the request path is reordered. Current main already reports zero.sync.view_syncer_hydration and zero.sync.lock-wait-time, but neither can answer the question here: hydration covers transform through pokeEnd as one number with no group or query attribution, and lock-wait records how long a caller waited to acquire the client-group lock rather than how long the lock stayed held while its work was descheduled. Without splitting the wave into stages you cannot tell an expensive query on an idle worker from an ordinary query queued behind sixteen peers, which is exactly the distinction the scaling law rests on.

The probes emit four zeroEvent payloads, keyed by clientGroupID so the harness can join them to a client sample:

• diagnosis-view-syncer-lock-acquired / -lock-released wrap #runInLockWithCVR and carry both lockWaitMs and lockWorkMs. The released probe fires in the finally block, before auth-maintenance scheduling and the actual unlock, so it slightly understates hold time.
• diagnosis-config-update-stages splits #updateCVRConfig into bookkeeping, CVR flush, config poke, and the query-sync branch.
• diagnosis-query-sync-stages reports transform time accumulated across every custom query in #syncQueryPipelineSet, plus the CVR/custom/add/remove counts.
• diagnosis-query-wave-stages splits #addAndRemoveQueries into prepare, TimeSliceTimer processing, hydration and bookkeeping, unreferenced-row deletion, CVR flush, client catch-up, and pokeEnd, alongside the wave's wall time and query count.

The separation between queryProcessingCpuMs and totalMs is the whole point: TimeSliceTimer stops while a ViewSyncer yields to the shared time-slice queue, so the first approximates CPU spent on this group and the second includes the scheduler wait. Their ratio is what makes queueing visible. (#6437 later promotes this distinction to standing OTel metrics; the log probes here are what the recorded baseline was extracted from.)

2. test(zero-cache): add the concurrent assignment-wave replay baseline harness. The harness, the fixture, and the recorded baseline, under packages/zero-cache/bench/wave-replay/.

3. test(zero-cache): settle the tracker divergence with an ac5eb231 A/B. See "The tracker divergence" below.

Measured baseline

3 runs per N, pooled median over all client samples, 8 vCPU. A fresh client group also hydrates Zero's internal lmids and mutationResults, so the server sees eight pipeline additions per six-root client wave.

N	settle	query wave	per-group cpu	wave / N=1	prior investigation wave / N=1
1	998 ms	496.8 ms	409.9 ms	1.00x	1.00x
2	1,448 ms	882.4 ms	380.0 ms	1.78x	1.75x
4	2,504 ms	1,653.1 ms	373.2 ms	3.33x	3.40x
8	5,076 ms	3,735.5 ms	395.4 ms	7.52x	6.86x
16	9,592 ms	7,362.1 ms	395.3 ms	14.82x	14.76x

Per-group CPU varies +4.9% to -4.5% around its mean across a 16x load increase. Row counts match the prior investigation exactly (roster 1,118, with_problems 53, trackers 7,768), and problem_trackers.for_assignment carries the same transformationHash 216mzlo8m5qcv. Raw per-client samples, per-run summaries and the pooled medians are committed under results/.

The tracker divergence, and what it teaches about portability

The baseline reproduced the earlier investigation's scaling ratios and row counts, with one divergence recorded in results/baseline-2026-08-28.md: the tracker graph costs 2.4x less here than in that investigation, moving its share of per-group processing from ~73% to ~53%, while the other five roots reproduce within 26%. Two data-shape explanations were tested against the original replica and ruled out. The remaining hypothesis was a code delta, since the investigation ran on upstream ac5eb2311 (#6425), which this branch's base predates.

An A/B settles it, and the code-delta hypothesis is wrong. Side A is this fork's main (16019afa4, as recorded in results/raw/ab/), a merge whose tree differs from upstream 7fb2c78bb (#6423) only by a fork CI workflow file that never runs in the harness. Side B is upstream ac5eb2311 (#6425), whose parent is exactly 7fb2c78bb. The only code that can differ between the two measured servers is therefore #6425: 35 lines in litestream3-prometheus-poller.ts and its test, a metrics poller that never runs in this harness. Both sides were built with the same instrumentation and run side by side, each on its own port, app id and replica against the same database, six measured N=1 replays each, alternating so neither could hold a cache-warmth advantage:

query	A: fork main (16019afa4)	B: ac5eb2311 (#6425)	B/A	prior investigation
assignment.roster	42.5	51.4	1.21x	49.8
problem_trackers.for_assignment	190.4	231.5	1.22x	537.7
per-group cpu	391.2	447.8	1.14x	730.6

The #6425 commit shows no tracker-specific penalty: its 1.22x sits inside the same band as roster's 1.21x, and the whole column is a flat ~1.14x offset from running two zero-cache instances on one 8-core box. What it does show is the tracker query at 231.5 ms on the exact commit the investigation measured at 537.7 ms -- 2.3x cheaper on this machine with the same fixture and the same transformationHash.

The divergence is therefore not explained by any code delta, which leaves the environment: it behaves like a machine-dependent per-row constant. The plausible mechanism is that problem_trackers.for_assignment hydrates 7,768 row events against at most 1,118 for any other root, so it is the only root whose cost is dominated by cache residency and allocation rate rather than fixed per-query overhead, and a different host memory subsystem would move it while leaving the small roots roughly in place. That mechanism is inferred, not isolated; what is established is that the same commit, fixture, and transformationHash price the tracker 2.3x apart on two hosts.

This changes what the harness promises. The scaling law and the ratio columns are the portable result; tracker dominance is not, and branches should re-baseline it on their own hardware rather than carrying ~73% or ~53% forward. Per-client replay output for all twelve A/B runs is under results/raw/ab/.

Shape of the harness

• client/zero-ordered-replay.ts is one client group. It imports the app's Zero client package, so run-replay-concurrency.ts copies it into that app's checkout before spawning it; bun resolves workspace packages by walking up from the script's own directory.
• run-replay-concurrency.ts starts N clients as separate processes, so each gets its own auth token, its own in-memory kv store, and therefore its own client group. It asserts wave integrity on every client: all six roots must complete within 1 ms of each other, because one group poke commits the wave and no root becomes visible before pokers.end(finalVersion). A wider spread means the run is not exercising single-poke semantics, so it exits non-zero rather than reporting a number that looks fine. All 93 samples in the recorded baseline had a 0 ms spread.
• run-matrix.ts drives the matrix and then the two aggregation scripts (extract-stage-events.mjs, summarize-concurrency.mjs).
• seed/apply-seed.sh applies the 3,365-statement idempotent fixture and mints one session row per concurrent client. The driver sends emu-session-token-NNN as its Zero auth token and zero-cache forwards it as a bearer token, so concurrent groups need distinct tokens. An unmatched token fails quietly: the transform endpoint answers 200 with a null userID and the connection is then rejected for a userID mismatch, leaving every root to time out with zero rows.

bench/wave-replay is excluded from the zero-cache tsconfig because the drivers are bun scripts, and typechecking them would pull @types/bun into a package with no other reason to want it.

Reruns, and an honest dependency

README.md in the harness directory carries the full recipe. ZERO_NUM_SYNC_WORKERS=1 is the load-bearing setting: normal multiprocess mode with one syncer child, not SINGLE_PROCESS=1, so the dispatcher's taskID/clientGroupID hash lands every group on the same worker and the same time-slice queue.

One prerequisite an outside runner should know up front: the six roots are named custom queries, so the harness needs the Goblins application's transform server and a Postgres database with that app's schema (the seed SQL is idempotent against it). The README documents the exact setup. The workload data itself is fully committed and synthetic; the transform dependency is what keeps the replayed queries byte-identical to the ones the investigation profiled, rather than an approximation of them.

Validation

pnpm --filter zero-cache check-types passes and git diff --check is clean. The probes are logging-only; the harness and results are new files outside the compiled package.

[vercel]

@Karavil is attempting to deploy a commit to the Rocicorp Team on Vercel.

A member of the Team first needs to authorize it.