perf(concat): plan merges analytically instead of materializing an R x S x P provenance map

[d-laub]

Closes #403. Closes #406.

Why

gvl.concat planned every merge by building a provenance map — one row per
merged (region, sample, ploid) slot — and then coalesce-ing it into runs. The
map is (R*S*P, 2) int64. At an All of Us chr22 grid (R = 3,734 regions,
S = 535,662 samples, ploidy 2) that is 64 GB allocated purely to decide where
bytes go, before a single byte is copied.

What changed

RunPlan derives the same runs analytically from the sorted merge order, never
materializing the map. Planning memory is now O(R + S) instead of O(R·S·P).

The key property that makes this work: the run-break pattern depends only on
order, never on the region index r. On the samples axis the r*S_d terms
cancel, leaving w_{j+1} == w_j + 1, so the break pattern is computed once in
O(R + S) and reused for every region.

provenance and coalesce are not deleted. They are retained as the
equivalence oracle and are now off the production path entirely — zero call sites
remain in _concat.py. test_run_plan_matches_coalesce_provenance_exhaustively
sweeps 4,800 configurations (including 4-dataset merges and non-permutation
orders) asserting RunPlan reproduces coalesce(provenance(...)) exactly. Zero
mismatches. Both functions' docstrings now say why they must not be deleted.

as_plan now raises TypeError on a one-shot generator instead of silently
list()-materializing it — which would have rebuilt the very structure this PR
exists to remove. RunPlan is a re-iterable object, not a generator; consumers
iterate it twice.

What this does not claim

The merged offsets.npy is still one int64 per merged slot — about 16 GB for
per-sample tracks, 32 GB for diploid genotypes at that grid. That is a property of
the ragged on-disk format, not of the merge. Peak resident during the copy is
roughly double that (32 / 64 GB), because the copy also holds each input's own
offsets array for the duration. The docs state this explicitly; no speedup is
claimed, only a memory-scaling fix to planning.

Fixture work (#406)

The svar2_store fixture had one occupied cell in an 18-cell grid, which made
several sparse/dense parity tests structurally vacuous — S1 was indistinguishable
from the deliberately-empty S2 column, and dense_snp_range was all zeros, so the
dense SNP path had no coverage at all. The fixture now has 7 of 18 occupied,
measured rather than reasoned about: every sparse variant carries exactly one call,
which is what keeps genoray's cost model from routing it to the per-region dense
channel.

This made two things possible that were not before:

• A sample-ordering guard that had to be withdrawn as unsatisfiable during svar2_ranges cache is dense over regions × samples × ploidy (64 B/cell, >99% empty at cohort scale): sparse or lazy alternatives #357 is
  reinstated — a transposed sample axis is now detectable.
• test_dense_layout_dataset_still_opens_and_reads is promoted from a smoke test to
  a real dense-vs-sparse parity pin. The original byte-comparison approach was found
  to be structurally incapable of catching a corrupted region index (nested bed
  regions plus position-based variant filtering self-heal it), so the test asserts on
  raw ranges instead.

The duplicated svar2_store fixture is also hoisted into tests/conftest.py.

Testing

• pixi run -e dev pytest tests -q — 1265 passed, 61 skipped, 4 xfailed, 0 failed
• cargo test — 138 passed
• pyrefly — 0 errors; ruff check / ruff format clean across 270 files
• api.md in sync with __all__ (MISSING: none); docs build with no new warnings

Versioning

No feat: commits, so commitizen cuts a PATCH bump (0.43.0 → 0.43.1).
DATASET_FORMAT_VERSION is unchanged at 2.0.0 — the on-disk format does not move.

Follow-ups filed

• Fold the test_svar2_readbound_tracks fixture onto the shared _svar2_slot_src #411 — fold test_svar2_readbound_tracks's byte-identical fixture onto _svar2_slot_src
• Promote svar2_store_2s to session scope now that no test mutates the shared store #412 — promote svar2_store_2s to session scope (unblocked by this PR; 6 store builds to 1)
• Hoist np.unique out of the per-batch loop in copy_runs #413 — hoist np.unique out of copy_runs' per-batch loop

🤖 Generated with Claude Code

https://claude.ai/code/session_015VxRqNngU7Eg1wdb1aEgD6