Repository navigation
Store lossless qualities in rANS blocks by default - #115
Merged
Merged
Conversation
Lossless qualities were a string column compressed by zstd, 109.3 MiB of the 478.2 MB file on chr20 of NA12878 at 30x; -fqzcomp put them into the blocks of 10,000 records instead, compressed with fqzcomp. On this NovaSeq data, whose qualities are binned to four values, fqzcomp saves 9.1% of the file but takes three times the CPU to read the qualities, half of it in building its 68 MB model for every block. CRAM 3.1's default profile does not use fqzcomp; it compresses its quality series with rANS Nx16 at order 1. A probe over the same blocks gave 72.4 MB for rANS order 1 with RLE and PACK, against 71.6 MB for fqzcomp and 72.7 MB for CRAM 3.1's quality series, and decoded 25 times faster than fqzcomp. Lossless qualities now go into the blocks whatever the quality policy, compressed with rANS order 1 with RLE and PACK. The first byte of a block says its codec; a rANS block holds the lengths of its qualities, as varints compressed the same way (0.03 MB on chr20), then the qualities. -fqzcomp passes the new kFqzcomp in the policy and gets fqzcomp blocks, as before plus the codec byte; kQualInBlock now only says that a record's quality is in its block. The format version becomes 8. On all of chr20 the file goes from 478.2 to 435.7 MB from SAM and from 477.7 to 435.3 MB from BAM, against develop acb8904 built the same way. The md5 of ramdump -h is the input's for SAM and develop's for BAM, and flagstat and idxstats print the same. On the 2 million record subset, medians of five alternating runs on a loaded machine, conversion takes 25.1 against 28.3 s of CPU on one thread, 29.3 against 31.2 s with -threads 4 and 34.5 against 37.4 s from BAM, at the same peak memory. Reading every quality costs CPU in the rANS decoder, built without SIMD: ramdump -h takes 8.0 against 6.9 s of CPU and 6.3 against 5.8 s wall on the subset, and 44.7 against 36.6 s of CPU and 32.5 against 30.0 s wall on chr20. Regions of 500 to 2,000 bp take 0.39 to 0.50 s on both sides, and ramdump -c and flagstat take the same time.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #115 +/- ##
===========================================
- Coverage 88.69% 88.69% -0.01%
===========================================
Files 25 25
Lines 3115 3220 +105
Branches 1050 1103 +53
===========================================
+ Hits 2763 2856 +93
- Misses 188 194 +6
- Partials 164 170 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| // fqzcomp and the name tokeniser take char, rANS unsigned char. | ||
| unsigned char *AsBytes(char *p) | ||
| { | ||
| return static_cast<unsigned char *>(static_cast<void *>(p)); |
There was a problem hiding this comment.
warning: do not cast 'char *' to 'unsigned char *' through 'void *' [bugprone-casting-through-void]
return static_cast<unsigned char *>(static_cast<void *>(p));
^
mvassilev
added a commit
that referenced
this pull request
Oct 18, 2026
The sizes and timings in CHANGELOG.md were measured with scripts that lived on one desktop, on data made by hand, so nobody else could check them or measure them on another machine. bench/ramtools-bench.py does it all from a clone. fetch reads chr20 of NA12878 from the 1000 Genomes 30x CRAM at EBI and chr20 of the GRCh38 reference, checks them against the M5 of chr20 and the md5 of the 17,705,654 records, and writes the SAM, BAM and CRAM 3.1 files and subsets of 50,000 and 2 million records, without @pg lines, so every machine gets the same files. run builds two git revisions, or the working tree, the same way, and measures conversion from SAM on one thread and with -threads, conversion from BAM, the whole file, a large and three small regions, a count, flagstat and idxstats, in alternating order, with wall time, CPU time and peak memory. It does the same reads with samtools on the BAM and the CRAM as the yardstick, checks every output against the input or samtools, and saves the report as Markdown and JSON. --pin keeps a run on one NUMA node. bench/README.md says what the script needs, how to run it and how to read the results. Tried on macOS: check; fetch of an 8 Mb slice of chr20 in 1 min 47 s; and run on 50,000 records, where all seven checks hold for develop before and after #115.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lossless qualities were a string column compressed by zstd, 109.3 MiB of the 478.2 MB file on chr20 of NA12878 at 30x; -fqzcomp put them into the blocks of 10,000 records instead, compressed with fqzcomp. On this NovaSeq data, whose qualities are binned to four values, fqzcomp saves 9.1% of the file but takes three times the CPU to read the qualities, half of it in building its 68 MB model for every block. CRAM 3.1's default profile does not use fqzcomp; it compresses its quality series with rANS Nx16 at order 1. A probe over the same blocks gave 72.4 MB for rANS order 1 with RLE and PACK, against 71.6 MB for fqzcomp and 72.7 MB for CRAM 3.1's quality series, and decoded 25 times faster than fqzcomp.
Lossless qualities now go into the blocks whatever the quality policy, compressed with rANS order 1 with RLE and PACK. The first byte of a block says its codec; a rANS block holds the lengths of its qualities, as varints compressed the same way (0.03 MB on chr20), then the qualities. -fqzcomp passes the new kFqzcomp in the policy and gets fqzcomp blocks, as before plus the codec byte; kQualInBlock now only says that a record's quality is in its block. The format version becomes 8.
On all of chr20 the file goes from 478.2 to 435.7 MB from SAM and from 477.7 to 435.3 MB from BAM, against develop acb8904 built the same way. The md5 of ramdump -h is the input's for SAM and develop's for BAM, and flagstat and idxstats print the same. On the 2 million record subset, medians of five alternating runs on a loaded machine, conversion takes 25.1 against 28.3 s of CPU on one thread, 29.3 against 31.2 s with -threads 4 and 34.5 against 37.4 s from BAM, at the same peak memory. Reading every quality costs CPU in the rANS decoder, built without SIMD: ramdump -h takes 8.0 against 6.9 s of CPU and 6.3 against 5.8 s wall on the subset, and 44.7 against 36.6 s of CPU and 32.5 against 30.0 s wall on chr20. Regions of 500 to 2,000 bp take 0.39 to 0.50 s on both sides, and ramdump -c and flagstat take the same time.