Repository navigation
Add a script that reproduces the benchmarks on any machine - #116
Merged
Merged
Conversation
The sizes and timings in CHANGELOG.md were measured with scripts that lived on one desktop, on data made by hand, so nobody else could check them or measure them on another machine. bench/ramtools-bench.py does it all from a clone. fetch reads chr20 of NA12878 from the 1000 Genomes 30x CRAM at EBI and chr20 of the GRCh38 reference, checks them against the M5 of chr20 and the md5 of the 17,705,654 records, and writes the SAM, BAM and CRAM 3.1 files and subsets of 50,000 and 2 million records, without @pg lines, so every machine gets the same files. run builds two git revisions, or the working tree, the same way, and measures conversion from SAM on one thread and with -threads, conversion from BAM, the whole file, a large and three small regions, a count, flagstat and idxstats, in alternating order, with wall time, CPU time and peak memory. It does the same reads with samtools on the BAM and the CRAM as the yardstick, checks every output against the input or samtools, and saves the report as Markdown and JSON. --pin keeps a run on one NUMA node. bench/README.md says what the script needs, how to run it and how to read the results. Tried on macOS: check; fetch of an 8 Mb slice of chr20 in 1 min 47 s; and run on 50,000 records, where all seven checks hold for develop before and after #115.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #116 +/- ##
========================================
Coverage 88.69% 88.69%
========================================
Files 25 25
Lines 3220 3220
Branches 1103 1103
========================================
Hits 2856 2856
Misses 194 194
Partials 170 170
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sizes and timings in CHANGELOG.md were measured with scripts that lived on one desktop, on data made by hand, so nobody else could check them or measure them on another machine.
bench/ramtools-bench.py does it all from a clone. fetch reads chr20 of NA12878 from the 1000 Genomes 30x CRAM at EBI and chr20 of the GRCh38 reference, checks them against the M5 of chr20 and the md5 of the 17,705,654 records, and writes the SAM, BAM and CRAM 3.1 files and subsets of 50,000 and 2 million records, without @pg lines, so every machine gets the same files. run builds two git revisions, or the working tree, the same way, and measures conversion from SAM on one thread and with -threads, conversion from BAM, the whole file, a large and three small regions, a count, flagstat and idxstats, in alternating order, with wall time, CPU time and peak memory. It does the same reads with samtools on the BAM and the CRAM as the yardstick, checks every output against the input or samtools, and saves the report as Markdown and JSON. --pin keeps a run on one NUMA node. bench/README.md says what the script needs, how to run it and how to read the results.
Tried on macOS: check; fetch of an 8 Mb slice of chr20 in 1 min 47 s; and run on 50,000 records, where all seven checks hold for develop before and after #115.