Skip to content

Add lastz=1.04.52,pigz=2.8,python=3.12 - #4178

Merged
bgruening merged 1 commit into
BioContainers:masterfrom
richard-burhans:lastz-pigz-python
Sep 19, 2026
Merged

Add lastz=1.04.52,pigz=2.8,python=3.12#4178
bgruening merged 1 commit into
BioContainers:masterfrom
richard-burhans:lastz-pigz-python

Conversation

@richard-burhans

@richard-burhans richard-burhans commented Sep 19, 2026

Copy link
Copy Markdown

Requesting mulled-v2-22e8b6fd71e23314d99ed252d036fc77e59901b5.

Why: Batched LASTZ (in richard-burhans/galaxytools) concatenates every per-command alignment into a single gzip stream at the end of its job, and that one write is ~62% of the job's wall clock (measured twice, on inputs differing 9×). Python's gzip module has no parallel mode, so it uses one core while the rest of the node is idle — the compute workers have all finished by the time the collapse runs.

Measured on 1.69 GB of real alignment output, 30 threads:

throughput
gzip -6 18 MB/s
gzip -1 137 MB/s
pigz -1 -p 30 2,456 MB/s

The tool already resolves mulled-v2-ab598cfd8731d8707b79df8e1e97a3113be58a40 (lastz + python), which exists. Adding pigz moves it to mulled-v2-22e8b6fd71e23314d99ed252d036fc77e59901b5, which currently returns 401 from quay.io.

pigz is on conda-forge (2.8); there is no bioconda build.

Richard Burhans
Add lastz=1.04.52,pigz=2.8,python=3.12
42f553f 
Batched LASTZ (richard-burhans/galaxytools) concatenates every
per-command alignment into one gzip stream at the end of its job, and
that single write is ~62% of the job's wall clock. Python's gzip module
has no parallel mode, so it uses one core while the rest of the node
sits idle -- the compute workers have all finished by then.

Measured on 1.69 GB of real alignment output:

  gzip -6          18 MB/s
  gzip -1         137 MB/s
  pigz -1 -p 30 2,456 MB/s

The tool already resolves mulled-v2-ab598cfd8731d8707b79df8e1e97a3113be58a40
(lastz + python); adding pigz moves it to
mulled-v2-22e8b6fd71e23314d99ed252d036fc77e59901b5, which does not exist
yet.
@richard-burhans richard-burhans mentioned this pull request Sep 19, 2026
Merged
@bgruening
bgruening enabled auto-merge September 19, 2026 13:21
@bgruening
bgruening merged commit 2704ab7 into BioContainers:master Sep 19, 2026
2 checks passed
@richard-burhans
richard-burhans deleted the lastz-pigz-python branch September 19, 2026 13:28
richard-burhans pushed a commit to richard-burhans/galaxytools that referenced this pull request Sep 19, 2026
Richard Burhans @claude
batched_lastz: run the collapse through pigz
7402f03 
The collapse -- every per-command alignment concatenated into one gzip
stream -- is ~62% of this job's wall clock, and it was single-threaded
because Python's gzip module has no parallel mode. By the time it runs
the compute workers have all finished, so one core does the work while
the rest of the node is idle.

Measured on 1.69 GB of real Cannabis AXT:

  gzip -6 (before #154)       18 MB/s
  gzip -1 (#154)             137 MB/s
  pigz -1 -p 30            2,456 MB/s

and end-to-end through this function, against the Python gzip it
replaces: 150 MB/s -> 997 MB/s, 6.6x, with the decompressed payload
byte-identical. The gap between 2,456 and 997 is Python feeding the
pipe, not pigz.

⚠ The level stays at 1, which is the opposite of what it looks like.
Threads make level 6 affordable -- 547 MB/s, four times faster than the
single-threaded level 1 it replaces, and 40% smaller. But level 1 with
the same threads is 2,456 MB/s, so against each other level 1 still wins
4.5x on the axis that matters. Level 6 becomes right only if the
bottleneck moves to I/O, which is UNMEASURED: the benchmark read from
page cache.

Three things from pigz's manual that the benchmark would not have told
me, all of them load-bearing:

  -n   NOT OPTIONAL. pigz stores the input's name and mtime in the gzip
       header by default, where gzip.open did not. Without it the output
       stops being a function of its content and two identical runs
       differ -- and nothing downstream would complain.
  -i   deliberately ABSENT. pigz loads the previous block's last 32 KiB
       as a preset dictionary, which is why its output is the same SIZE
       as gzip's at the same level (0.53 GB, 3.17x, both). -i drops that
       for random access and partial error recovery, neither of which a
       single streamed output needs.
  env  pigz reads GZIP and PIGZ from the environment BEFORE its command
       line, so anything set there silently overrides these flags. They
       are cleared for the child rather than trusted.

And one that says what NOT to do: decompression "can't be parallelized",
so the tarball this job unpacks is unaffected. Measured: 380 MB/s with
gzip, 417-448 with unpigz, and thread count changes nothing.

A missing pigz is FATAL, not a fallback. A quiet drop back to Python's
gzip would produce correct output several times slower and hide exactly
the failure the requirement exists to prevent -- a container that
resolves without it.

The copy loop stays text and line-by-line. Rewriting it binary with
shutil.copyfileobj was measured at 826 MB/s against 767, which does not
justify changing the contract on a path where a partial write is already
a known trap.

ONLY batched_lastz moves, which is the first change to benefit from the
suffix decoupling it is stacked on. Rendered through
galaxy.tool_util.loader:

  batched_lastz   384857b08bf2 -> 63d549236b5f   1.04.52+galaxy4 -> +galaxy5
  kegalign        4e1560c2c42c    unchanged      0.3.2+galaxy1
  growler_lastz   2bcdc1669338    unchanged      1.04.52+galaxy2

@PIGZ_VERSION@ lives in batched_lastz.xml rather than macros.xml for the
same reason: only this tool uses pigz, and a token used by one tool in a
shared file is what coupled the three versions in the first place.

The container exists now (BioContainers/multi-package-containers#4178)
and was verified by running it, not by trusting the lint:

  pigz 2.8, lastz 1.04.52, Python 3.12.14, and `pigz -1 -n -p 4 -c`
  round-trips inside mulled-v2-22e8b6fd...:528c639a...-0

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
richard-burhans pushed a commit to richard-burhans/galaxytools that referenced this pull request Sep 19, 2026
Richard Burhans @claude
batched_lastz: run the collapse through pigz
b46ab1a 
The collapse -- every per-command alignment concatenated into one gzip
stream -- is ~62% of this job's wall clock, and it was single-threaded
because Python's gzip module has no parallel mode. By the time it runs
the compute workers have all finished, so one core does the work while
the rest of the node is idle.

Measured on 1.69 GB of real Cannabis AXT:

  gzip -6 (before #154)       18 MB/s
  gzip -1 (#154)             137 MB/s
  pigz -1 -p 30            2,456 MB/s

and end-to-end through this function, against the Python gzip it
replaces: 150 MB/s -> 997 MB/s, 6.6x, with the decompressed payload
byte-identical. The gap between 2,456 and 997 is Python feeding the
pipe, not pigz.

⚠ The level stays at 1, which is the opposite of what it looks like.
Threads make level 6 affordable -- 547 MB/s, four times faster than the
single-threaded level 1 it replaces, and 40% smaller. But level 1 with
the same threads is 2,456 MB/s, so against each other level 1 still wins
4.5x on the axis that matters. Level 6 becomes right only if the
bottleneck moves to I/O, which is UNMEASURED: the benchmark read from
page cache.

Three things from pigz's manual that the benchmark would not have told
me, all of them load-bearing:

  -n   NOT OPTIONAL. pigz stores the input's name and mtime in the gzip
       header by default, where gzip.open did not. Without it the output
       stops being a function of its content and two identical runs
       differ -- and nothing downstream would complain.
  -i   deliberately ABSENT. pigz loads the previous block's last 32 KiB
       as a preset dictionary, which is why its output is the same SIZE
       as gzip's at the same level (0.53 GB, 3.17x, both). -i drops that
       for random access and partial error recovery, neither of which a
       single streamed output needs.
  env  pigz reads GZIP and PIGZ from the environment BEFORE its command
       line, so anything set there silently overrides these flags. They
       are cleared for the child rather than trusted.

And one that says what NOT to do: decompression "can't be parallelized",
so the tarball this job unpacks is unaffected. Measured: 380 MB/s with
gzip, 417-448 with unpigz, and thread count changes nothing.

A missing pigz is FATAL, not a fallback. A quiet drop back to Python's
gzip would produce correct output several times slower and hide exactly
the failure the requirement exists to prevent -- a container that
resolves without it.

The copy loop stays text and line-by-line. Rewriting it binary with
shutil.copyfileobj was measured at 826 MB/s against 767, which does not
justify changing the contract on a path where a partial write is already
a known trap.

ONLY batched_lastz moves, which is the first change to benefit from the
suffix decoupling it is stacked on. Rendered through
galaxy.tool_util.loader:

  batched_lastz   1.04.52+galaxy5 -> +galaxy6   CHANGED
  kegalign        0.3.2+galaxy2               unchanged
  growler_lastz   1.04.52+galaxy3             unchanged

and the change set is batched_lastz.xml plus its script: macros.xml is
not touched at all, which is what the decoupling bought.

@PIGZ_VERSION@ lives in batched_lastz.xml rather than macros.xml for the
same reason: only this tool uses pigz, and a token used by one tool in a
shared file is what coupled the three versions in the first place.

The container exists now (BioContainers/multi-package-containers#4178)
and was verified by running it, not by trusting the lint:

  pigz 2.8, lastz 1.04.52, Python 3.12.14, and `pigz -1 -n -p 4 -c`
  round-trips inside mulled-v2-22e8b6fd...:528c639a...-0

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
richard-burhans pushed a commit to richard-burhans/galaxytools that referenced this pull request Sep 19, 2026
Richard Burhans @claude
batched_lastz: run the collapse through pigz
e01cbe5 
The collapse -- every per-command alignment concatenated into one gzip
stream -- is ~62% of this job's wall clock, and it was single-threaded
because Python's gzip module has no parallel mode. By the time it runs
the compute workers have all finished, so one core does the work while
the rest of the node is idle.

Measured on 1.69 GB of real Cannabis AXT:

  gzip -6 (before #154)       18 MB/s
  gzip -1 (#154)             137 MB/s
  pigz -1 -p 30            2,456 MB/s

and end-to-end through this function, against the Python gzip it
replaces: 150 MB/s -> 997 MB/s, 6.6x, with the decompressed payload
byte-identical. The gap between 2,456 and 997 is Python feeding the
pipe, not pigz.

⚠ The level stays at 1, which is the opposite of what it looks like.
Threads make level 6 affordable -- 547 MB/s, four times faster than the
single-threaded level 1 it replaces, and 40% smaller. But level 1 with
the same threads is 2,456 MB/s, so against each other level 1 still wins
4.5x on the axis that matters. Level 6 becomes right only if the
bottleneck moves to I/O, which is UNMEASURED: the benchmark read from
page cache.

Three things from pigz's manual that the benchmark would not have told
me, all of them load-bearing:

  -n   NOT OPTIONAL. pigz stores the input's name and mtime in the gzip
       header by default, where gzip.open did not. Without it the output
       stops being a function of its content and two identical runs
       differ -- and nothing downstream would complain.
  -i   deliberately ABSENT. pigz loads the previous block's last 32 KiB
       as a preset dictionary, which is why its output is the same SIZE
       as gzip's at the same level (0.53 GB, 3.17x, both). -i drops that
       for random access and partial error recovery, neither of which a
       single streamed output needs.
  env  pigz reads GZIP and PIGZ from the environment BEFORE its command
       line, so anything set there silently overrides these flags. They
       are cleared for the child rather than trusted.

And one that says what NOT to do: decompression "can't be parallelized",
so the tarball this job unpacks is unaffected. Measured: 380 MB/s with
gzip, 417-448 with unpigz, and thread count changes nothing.

A missing pigz is FATAL, not a fallback. A quiet drop back to Python's
gzip would produce correct output several times slower and hide exactly
the failure the requirement exists to prevent -- a container that
resolves without it.

The copy loop stays text and line-by-line. Rewriting it binary with
shutil.copyfileobj was measured at 826 MB/s against 767, which does not
justify changing the contract on a path where a partial write is already
a known trap.

ONLY batched_lastz moves, which is the first change to benefit from the
suffix decoupling it is stacked on. Rendered through
galaxy.tool_util.loader:

  batched_lastz   1.04.52+galaxy5 -> +galaxy6   CHANGED
  kegalign        0.3.2+galaxy2               unchanged
  growler_lastz   1.04.52+galaxy3             unchanged

and the change set is batched_lastz.xml plus its script: macros.xml is
not touched at all, which is what the decoupling bought.

@PIGZ_VERSION@ lives in batched_lastz.xml rather than macros.xml for the
same reason: only this tool uses pigz, and a token used by one tool in a
shared file is what coupled the three versions in the first place.

The container exists now (BioContainers/multi-package-containers#4178)
and was verified by running it, not by trusting the lint:

  pigz 2.8, lastz 1.04.52, Python 3.12.14, and `pigz -1 -n -p 4 -c`
  round-trips inside mulled-v2-22e8b6fd...:528c639a...-0

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
richard-burhans added a commit to richard-burhans/galaxytools that referenced this pull request Sep 19, 2026
@richard-burhans Richard Burhans @claude
batched_lastz: run the collapse through pigz (#157)
3fa8520 
The collapse -- every per-command alignment concatenated into one gzip
stream -- is ~62% of this job's wall clock, and it was single-threaded
because Python's gzip module has no parallel mode. By the time it runs
the compute workers have all finished, so one core does the work while
the rest of the node is idle.

Measured on 1.69 GB of real Cannabis AXT:

  gzip -6 (before #154)       18 MB/s
  gzip -1 (#154)             137 MB/s
  pigz -1 -p 30            2,456 MB/s

and end-to-end through this function, against the Python gzip it
replaces: 150 MB/s -> 997 MB/s, 6.6x, with the decompressed payload
byte-identical. The gap between 2,456 and 997 is Python feeding the
pipe, not pigz.

⚠ The level stays at 1, which is the opposite of what it looks like.
Threads make level 6 affordable -- 547 MB/s, four times faster than the
single-threaded level 1 it replaces, and 40% smaller. But level 1 with
the same threads is 2,456 MB/s, so against each other level 1 still wins
4.5x on the axis that matters. Level 6 becomes right only if the
bottleneck moves to I/O, which is UNMEASURED: the benchmark read from
page cache.

Three things from pigz's manual that the benchmark would not have told
me, all of them load-bearing:

  -n   NOT OPTIONAL. pigz stores the input's name and mtime in the gzip
       header by default, where gzip.open did not. Without it the output
       stops being a function of its content and two identical runs
       differ -- and nothing downstream would complain.
  -i   deliberately ABSENT. pigz loads the previous block's last 32 KiB
       as a preset dictionary, which is why its output is the same SIZE
       as gzip's at the same level (0.53 GB, 3.17x, both). -i drops that
       for random access and partial error recovery, neither of which a
       single streamed output needs.
  env  pigz reads GZIP and PIGZ from the environment BEFORE its command
       line, so anything set there silently overrides these flags. They
       are cleared for the child rather than trusted.

And one that says what NOT to do: decompression "can't be parallelized",
so the tarball this job unpacks is unaffected. Measured: 380 MB/s with
gzip, 417-448 with unpigz, and thread count changes nothing.

A missing pigz is FATAL, not a fallback. A quiet drop back to Python's
gzip would produce correct output several times slower and hide exactly
the failure the requirement exists to prevent -- a container that
resolves without it.

The copy loop stays text and line-by-line. Rewriting it binary with
shutil.copyfileobj was measured at 826 MB/s against 767, which does not
justify changing the contract on a path where a partial write is already
a known trap.

ONLY batched_lastz moves, which is the first change to benefit from the
suffix decoupling it is stacked on. Rendered through
galaxy.tool_util.loader:

  batched_lastz   1.04.52+galaxy5 -> +galaxy6   CHANGED
  kegalign        0.3.2+galaxy2               unchanged
  growler_lastz   1.04.52+galaxy3             unchanged

and the change set is batched_lastz.xml plus its script: macros.xml is
not touched at all, which is what the decoupling bought.

@PIGZ_VERSION@ lives in batched_lastz.xml rather than macros.xml for the
same reason: only this tool uses pigz, and a token used by one tool in a
shared file is what coupled the three versions in the first place.

The container exists now (BioContainers/multi-package-containers#4178)
and was verified by running it, not by trusting the lint:

  pigz 2.8, lastz 1.04.52, Python 3.12.14, and `pigz -1 -n -p 4 -c`
  round-trips inside mulled-v2-22e8b6fd...:528c639a...-0

Co-authored-by: Richard Burhans <richard.burhans@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 19, 2026
Open
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bgruening bgruening bgruening approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@richard-burhans @bgruening