Skip to content

Add a script that reproduces the benchmarks on any machine - #116

Merged
mvassilev merged 1 commit into
developfrom
feature/bench-script
Oct 18, 2026
Merged

mvassilev merged 1 commit into
developfrom
feature/bench-script

Conversation

@mvassilev

Copy link
Copy Markdown
Collaborator

The sizes and timings in CHANGELOG.md were measured with scripts that lived on one desktop, on data made by hand, so nobody else could check them or measure them on another machine.

bench/ramtools-bench.py does it all from a clone. fetch reads chr20 of NA12878 from the 1000 Genomes 30x CRAM at EBI and chr20 of the GRCh38 reference, checks them against the M5 of chr20 and the md5 of the 17,705,654 records, and writes the SAM, BAM and CRAM 3.1 files and subsets of 50,000 and 2 million records, without @pg lines, so every machine gets the same files. run builds two git revisions, or the working tree, the same way, and measures conversion from SAM on one thread and with -threads, conversion from BAM, the whole file, a large and three small regions, a count, flagstat and idxstats, in alternating order, with wall time, CPU time and peak memory. It does the same reads with samtools on the BAM and the CRAM as the yardstick, checks every output against the input or samtools, and saves the report as Markdown and JSON. --pin keeps a run on one NUMA node. bench/README.md says what the script needs, how to run it and how to read the results.

Tried on macOS: check; fetch of an 8 Mb slice of chr20 in 1 min 47 s; and run on 50,000 records, where all seven checks hold for develop before and after #115.

The sizes and timings in CHANGELOG.md were measured with scripts that lived
on one desktop, on data made by hand, so nobody else could check them or
measure them on another machine.

bench/ramtools-bench.py does it all from a clone. fetch reads chr20 of NA12878
from the 1000 Genomes 30x CRAM at EBI and chr20 of the GRCh38 reference,
checks them against the M5 of chr20 and the md5 of the 17,705,654 records, and
writes the SAM, BAM and CRAM 3.1 files and subsets of 50,000 and 2 million
records, without @pg lines, so every machine gets the same files. run builds
two git revisions, or the working tree, the same way, and measures conversion
from SAM on one thread and with -threads, conversion from BAM, the whole file,
a large and three small regions, a count, flagstat and idxstats, in
alternating order, with wall time, CPU time and peak memory. It does the same
reads with samtools on the BAM and the CRAM as the yardstick, checks every
output against the input or samtools, and saves the report as Markdown and
JSON. --pin keeps a run on one NUMA node. bench/README.md says what the script
needs, how to run it and how to read the results.

Tried on macOS: check; fetch of an 8 Mb slice of chr20 in 1 min 47 s; and
run on 50,000 records, where all seven checks hold for develop before and
after #115.
@mvassilev
mvassilev merged commit 629b2b4 into develop Oct 18, 2026
1 check passed
@mvassilev
mvassilev deleted the feature/bench-script branch October 18, 2026 09:52
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.69%. Comparing base (5b61106) to head (14a7809).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff            @@
##           develop     #116   +/-   ##
========================================
  Coverage    88.69%   88.69%           
========================================
  Files           25       25           
  Lines         3220     3220           
  Branches      1103     1103           
========================================
  Hits          2856     2856           
  Misses         194      194           
  Partials       170      170           
Flag Coverage Δ
unittests 88.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants