[joshfactorial]

Closes #876 (close by hand on merge to develop).

The shuffle merged in #875 made writing the final FASTQs ~6× slower than plain concatenation at genome scale. This PR cuts that to ~2.2×, and the output stays byte-identical to #875.

Change (fastq_shuffle.rs)

• Bucket files are kept open whenever 2K + 64 fits the soft open-file limit (read from /proc/self/limits). Otherwise each append reopens the file. Shuffle gen-reads FASTQ output by default (shuffle_fastq, #870) #875 opened, appended to and closed a file per bucket per mate for every full buffer.
• Appends wait their turn, so segments land in fill order and the output does not depend on thread timing. If any thread fails, an abort flag releases the waiting appends.
• Full buffers are compressed on a worker pool of up to 4 threads while the reader continues.
• Bucket files use libdeflate level 1. Shuffle gen-reads FASTQ output by default (shuffle_fastq, #870) #875 used CompressionLvl::fastest(), which is level 0 (stored), so it wrote ~164 GB of uncompressed temp data at this scale. The bug is still on develop until this merges.
• Shuffled buckets go straight into the output through an in-order writer. Pass 2 decompresses each bucket whole.
• Each phase logs its elapsed time (shuffle scatter: …, shuffle and write: …).

Evidence at genome scale: shuffle_bench 23673810

256.6M pairs (T2T-CHM13 at 21×), 64 CPUs, TMPDIR on Lustre scratch, three rounds. Each round runs one task per condition. All 9 tasks COMPLETED with exit 0.

condition	final-FASTQ stage (s)	whole run (s)
shuffle off	406, 393, 394	1119, 1093, 1088
#875 (develop 58eba13)	2348, 2397, 2598	3052, 3097, 3294
this PR (f76f9ca)	875, 864, 884	1580, 1568, 1586

• Output is identical at scale: Shuffle gen-reads FASTQ output by default (shuffle_fastq, #870) #875 and this PR have the same r1 md5 (d486c0023b96) and the same sizes for both mates in all six runs.
• Phases for this PR: scatter took 382–394 s, with 512 buckets, "files kept open" and 58.8 GB of bucket files. Shuffle and write took 481–488 s on 8 threads.
• An earlier draft of this PR used a single scatter log and took 4358–4479 s (job 23620433), from random reads of segments. It is replaced by the design above.

Evidence (local)

• Byte-identical to Shuffle gen-reads FASTQ output by default (shuffle_fastq, #870) #875 on CHM13 chr21 at 20×, both mates.
• 14 unit tests, including:
  • kept-open and reopened files give the same bytes;
  • appends land in fill order;
  • abort releases a waiting append;
  • a cap-path test that re-runs itself under ulimit -n 48 and drives 512 paired buckets through the reopen path;
  • raw gzip bytes match across thread counts (unit tests and tests/fastq_shuffle_output.rs).
• Mutants: ignoring append turns fails 4 tests; ignoring abort hangs the abort test (caught by the timeout).
• The workspace suite (65 test binaries), clippy -D warnings and fmt pass.

Also here

• scripts/delta/shuffle_bench.sbatch, the replicated benchmark above (%2, because three tasks at once exceeded the scratch quota).
• docs-site/src/gen-reads/fastq-shuffling.md now gives the genome-scale cost.

Not verified

• Peak memory at scale: /usr/bin/time read a constant 13.7 GB for every condition, and sacct MaxRSS is noisy (59–112 GB across all conditions). Neither isolates the shuffle.
• The reopen path at genome scale. It is covered locally by the cap-path test only; Delta's limit kept files open.
• Any filesystem other than Delta's Lustre.

🤖 Generated with Claude Code

[joshfactorial]

Benchmark 23620433, and a v3 design (f76f9ca)

Results, 256.6M pairs, Delta, TMPDIR on scratch; stage = "Read generation complete" → "Successfully wrote":

condition	stage (s)
shuffle off	400, 408, 394
merged shuffle (#875)	2530, 2578
this PR as of 2072546 (single scatter log)	4358, 4479

• The single scatter log is slower. Its shuffle-and-write phase took 3982 s and 4090 s: each bucket's ~70–256 KB segments sit scattered across the log, so pass 2 did ~650k small random reads.
• Correctness at scale: Shuffle gen-reads FASTQ output by default (shuffle_fastq, #870) #875 and this PR wrote byte-identical FASTQs (same md5 and sizes).
• The night-of-filesystem-work run (2 h 12 m for Shuffle gen-reads FASTQ output by default (shuffle_fastq, #870) #875) was ~3× slower than this, so the filesystem was a large part of that number.
• Tasks 4 and 5 failed on scratch quota, three tasks at once.

Found while reading the phase logs: bucket files used CompressionLvl::fastest(), which is libdeflate level 0, stored. Both #875 (on develop) and the earlier commits here wrote ~164 GB of uncompressed temp data at this scale.

v3:

• per-bucket files again, so pass 2 reads sequentially;
• files kept open while 2K + 64 fits the soft open-file limit (from /proc/self/limits), reopened per append otherwise;
• appends wait their turn so segments stay in fill order, and an abort flag releases waiters if any thread fails;
• level 1 for bucket files;
• kept from the previous design: the worker pool, the in-order writer straight to the output, and the phase logs.

Local evidence (chr21, 20×):

• output byte-identical to Shuffle gen-reads FASTQ output by default (shuffle_fastq, #870) #875 on both mates;
• bucket files 0.7 GB against 2.2 GB stored;
• 14 unit tests;
• mutants: ignoring append turns fails 4 tests; ignoring abort hangs the abort test. Forcing reopen mode is output-equivalent, and a test pins that equivalence.

The workspace suite (65 test binaries), clippy and fmt all pass. The genome-scale rerun is next, with shuffle_bench now at %2.