MetaPathways 4.0.1: reliable nested Pathway Tools builds

CITATION.cff

@@ -41,4 +41,4 @@ keywords:
   - functional annotation
   - pathways
   - Nextflow
-version: "4.0.0"
+version: "4.0.1"

MANIFEST.in

@@ -17,6 +17,7 @@ include tests/test_read_mapping.py
 include tests/test_pt_container.py
 include tests/test_nextflow.py
 include docker/Dockerfile.release
+include docker/Dockerfile.ptools-base
 include docker/README.quay.md
 
 recursive-include metapathways/report_assets *.html *.js *.css

README.md

@@ -14,7 +14,7 @@ On Linux x86-64, install MetaPathways and its workflow dependencies with Mamba:
 
 ```bash
 mamba create -n metapathways --override-channels --strict-channel-priority \
-  -c hallamlab -c conda-forge -c bioconda metapathways=4.0.0
+  -c hallamlab -c conda-forge -c bioconda metapathways=4.0.1
 conda activate metapathways
 ```
 

changelog.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 4.0.1
+
+- Bundle an unlicensed Pathway Tools dependency rootfs in release containers so nested licensed builds do not need APT user-switching privileges.
+- Clear inherited outer binds only for the inner image build; retain package provenance and blocking image validation.
+- Retry transient official-patch HTTP failures with bounded delays.
+- Add optional outer-image extraction and isolated configuration to the licensed nested test helper.
+
+
 ## 4.0.0
 
 - Nextflow controllers for local and Slurm workflows, multi-sample manifests, resource budgets, database builds and licensed Pathway Tools container preparation.

docker/Dockerfile.ptools-base

@@ -0,0 +1,20 @@
+# Public system dependencies only. No Pathway Tools installer, software,
+# MetaCyc data, credentials, or vendor patches belong in this image.
+# Keep this package list aligned with metapathways/pt_container.py until the
+# prepared-base build path has passed licensed integration testing.
+FROM ubuntu:22.04
+
+RUN apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        ca-certificates xterm openssl libxml2 xvfb xauth libxm4 libssl-dev \
+        procps bzip2 ncbi-blast+ \
+    && mkdir -p /data /opt/bin \
+    && printf '[ncbi]\nData=/usr/share/ncbi/data\n' > /opt/mp-ncbirc \
+    && test -d /usr/share/ncbi/data \
+    && blastp -version \
+    && makeblastdb -version \
+    && dpkg-query -W ncbi-blast+ > /opt/mp-blast-version.txt \
+    && dpkg-query -W > /opt/mp-base-packages.tsv \
+    && rm -rf /var/lib/apt/lists/*
+
+LABEL org.metapathways.purpose="Pathway Tools system dependencies only"

docker/Dockerfile.release

@@ -4,6 +4,7 @@ USER root
 RUN apt-get update && apt-get upgrade --yes && \
     apt-get install --yes --no-install-recommends procps && \
     rm -rf /var/lib/apt/lists/*
+COPY --from=ptools-base / /opt/metapathways/ptools-base/
 USER $MAMBA_USER
 ARG VERSION
 ARG REVISION

docs/containers.md

@@ -2,7 +2,7 @@
 
 [Home and quick start](installation.md) · [Test dataset](test.md) · [Licensed Pathway Tools](pathway-tools.md)
 
-Use the versioned `quay.io/hallamlab/metapathways:4.0.0` image on Linux x86-64. It includes MP, its workflow dependencies (including MAGSplitter and Camelot), and the three-sample test dataset derived from CAMI II ([Meyer et al., 2022](#cami-references)). Production references and licensed Pathway Tools are supplied separately.
+Use the versioned `quay.io/hallamlab/metapathways:4.0.1` image on Linux x86-64. It includes MP, its workflow dependencies (including MAGSplitter and Camelot), and the three-sample test dataset derived from CAMI II ([Meyer et al., 2022](#cami-references)). Production references and licensed Pathway Tools are supplied separately.
 
 ## Docker three-sample test
 
@@ -11,9 +11,9 @@ Create a working directory and open a shell in the image. The mount keeps inputs
 ```bash
 mkdir -p ~/mp-test-docker
 cd ~/mp-test-docker
-docker pull quay.io/hallamlab/metapathways:4.0.0
+docker pull quay.io/hallamlab/metapathways:4.0.1
 docker run --rm -it --network host --user "$(id -u):$(id -g)" \
-  -v "$PWD:/work" -w /work quay.io/hallamlab/metapathways:4.0.0 bash
+  -v "$PWD:/work" -w /work quay.io/hallamlab/metapathways:4.0.1 bash
 ```
 
 Inside the container, run:
@@ -40,7 +40,7 @@ Create a working directory and pull the image on an internet-connected host:
 ```bash
 mkdir -p ~/mp-test-apptainer
 cd ~/mp-test-apptainer
-apptainer pull metapathways.sif docker://quay.io/hallamlab/metapathways:4.0.0
+apptainer pull metapathways.sif docker://quay.io/hallamlab/metapathways:4.0.1
 apptainer exec --bind "$PWD:/work" --pwd /work metapathways.sif bash
 ```
 
@@ -69,3 +69,73 @@ The public MP image does not include Pathway Tools or MetaCyc. Licensed users sh
 
 ```{include} includes/cami-references.md
 ```
+
+## Nested Pathway Tools containers
+
+MP's release container includes an unlicensed Ubuntu dependency base for
+`build_pt`. When present, MP uses it automatically instead of installing system
+packages inside the nested build. Conda installations without the bundled base
+continue to build from Ubuntu using host Apptainer.
+
+A local nested build passed installation, official patch startup, SIF creation,
+BLAST validation and Pathway Tools readiness. Complete nested PGDB inference and
+portability across HPC installations require separate validation. Use the Mamba
+route when a host restricts nested user namespaces or container execution.
+
+### What the nested checks established
+
+- Java/Nextflow stalled on a FUSE filesystem request when executing from the
+  outer SIF on the test host. Launching the outer image with `--unsquash`
+  resolved this; extraction requires additional temporary space and startup time.
+- Outer bind mounts were inherited by the inner build and targeted directories
+  absent from its writable root. Clearing inherited bind variables inside the
+  outer container let the build proceed. Explicit inner runtime mounts are still
+  required; removing all mounts indiscriminately is not a general solution.
+- The inner build fell back to a root-mapped namespace without full UID mappings
+  or a fakeroot helper. Package installation failed when APT tried to switch to
+  its `_apt` user. Preparing dependencies outside this restricted namespace
+  removed that step; the licensed installer then completed successfully.
+- The prepared-base build completed in approximately six minutes, including
+  official patch startup, SIF packaging, BLAST database/search validation and
+  the Pathway Tools readiness marker. This duration excludes building the base.
+- SRI returned HTTP 503 twice during the experiment. The successful test reused
+  107 unmodified official patch files from an existing private SIF, checking
+  their hashes against its retained manifest. A fresh vendor download was
+  therefore not validated in that successful attempt.
+
+### Build behavior and provenance
+
+The inner build subprocess clears inherited bind variables, while leaving the
+parent environment unchanged. The built image metadata records its installer,
+recipe, image and official-patch hashes, plus a package-manifest hash for the
+bundled dependency base. Validation remains blocking: a failed build is never
+registered. Transient vendor HTTP errors receive at most three attempts; missing
+patches still stop the build.
+
+Use `--unsquash` on the outer Apptainer invocation if Java stalls on its FUSE
+mount. This is a host launch setting; code inside an already-mounted container
+cannot remount that outer image. Keep configuration inside the mounted project
+when building from a container, and pass the container-visible SIF path with
+`--image` when running it. A `/work` registration is meaningful inside that mount,
+not from an unrelated host shell.
+
+Maintainers can repeat the licensed checks on a target host from a source checkout:
+
+```bash
+python scripts/check_nested_ptools.py \
+  --mp-image /absolute/path/metapathways.sif \
+  --pt-image /absolute/path/pathway-tools.sif \
+  --installer /absolute/path/pathway-tools-29.5-linux-64-tier1-install \
+  --output /absolute/path/new-nested-test
+```
+
+Activate the host Apptainer environment first so its helper tools are on `PATH`.
+Omit `--installer` to check only execution of an existing licensed image. The
+output directory must be new; it contains private logs and a machine-readable
+result. Each check has a configurable timeout (`--timeout`, default 900 seconds).
+The test does not upload licensed files. Its configuration directory is isolated
+under the requested output, so it does not replace the normal MP image registry.
+Use `--unsquash` when testing the extraction route described above; the default
+continues to test the mounted SIF.
+A passed startup check verifies Pathway Tools readiness and BLAST integration;
+it does not replace a complete PGDB inference test.

docs/installation.md

@@ -6,7 +6,7 @@ Install MetaPathways, try the included three-sample dataset, then use your own d
 
 ```bash
 mamba create -n metapathways --override-channels --strict-channel-priority \
-  -c hallamlab -c conda-forge -c bioconda metapathways=4.0.0
+  -c hallamlab -c conda-forge -c bioconda metapathways=4.0.1
 conda activate metapathways
 ```
 
@@ -30,10 +30,10 @@ Open the URL printed by the report server. On a remote server, use an [SSH tunne
 
 ```bash
 # Docker
-docker pull quay.io/hallamlab/metapathways:4.0.0
+docker pull quay.io/hallamlab/metapathways:4.0.1
 
 # Or Apptainer
-apptainer pull metapathways.sif docker://quay.io/hallamlab/metapathways:4.0.0
+apptainer pull metapathways.sif docker://quay.io/hallamlab/metapathways:4.0.1
 ```
 
 The image includes the same workflow dependencies and test data. Follow the [Docker three-sample test](containers.md#docker-three-sample-test) or [Apptainer three-sample test](containers.md#apptainer-three-sample-test) to run the commands with your working directory mounted for persistent results. Licensed Pathway Tools is a separate image.

docs/pathway-tools.md

@@ -18,6 +18,15 @@ The compatibility list removes only unsafe **explicit reaction assignments**,
 not genes, annotations or sequences; the same reaction may still be inferred
 later. The sections below explain the evidence, limits and diagnostic records.
 
+## Building from the MP Apptainer container
+
+The release container includes an unlicensed Ubuntu dependency base. `build_pt`
+selects it automatically, avoiding package installation inside the restricted
+nested build. Your licensed installer and official patches are still required
+and remain private. See [nested containers](containers.md#nested-pathway-tools-containers)
+for tested behavior and host limitations. On hosts affected by Java/FUSE stalls,
+launch the outer MP SIF with `--unsquash`; this needs temporary extraction space.
+
 ## Understand the three different databases
 
 | Name | What it contains | How you use it |

docs/releasing.md

@@ -6,8 +6,8 @@ This chapter is for maintainers preparing and publishing packages. Every release
 must pass package and workflow validation before publication.
 
 The source version in `metapathways/_version.py` is authoritative. The current
-release series is **4.0**, with package version **4.0.0** and tag **v4.0.0**.
-Use subsequent patch versions such as **4.0.1**, or **4.0.1rc1** for a preview.
+release series is **4.0**, with package version **4.0.1** and tag **v4.0.1**.
+Use subsequent patch versions such as **4.0.2**, or **4.0.2rc1** for a preview.
 Tags have a leading `v`; package versions do not. Updating a repository does not
 upgrade an already installed environment.
 
@@ -25,10 +25,10 @@ Open feature work as a draft PR targeting `dev` while benchmarks or test checks
 Prepare any version changes on the feature branch before final PR review:
 
 ```bash
-python scripts/release.py prepare 4.0.0
+python scripts/release.py prepare 4.0.1
 git diff
 git add README.md CITATION.cff metapathways/_version.py conda_recipe/meta_template.yaml
-git commit -m "Prepare MetaPathways 4.0.0"
+git commit -m "Prepare MetaPathways 4.0.1"
 ```
 
 Commit the release script, workflow, tests, and other implementation files as well
@@ -281,7 +281,7 @@ The intended sequence is **feature branch → single-server and HPC benchmarks p
 
 Use the [PR tester checklist](pr-testing.md) for independent installation, the single- and two-sample CAMI workflows, restart behavior, optional licensed Pathway Tools, and report exports. Record the tested Git commit. Record the source commit as well as the package version when testing a candidate.
 
-The release version is **4.0.0**. Preparing this version does not publish it; branch and PR testing still happen before a release tag is pushed. Do not reuse 3.5.1 or its existing tags. Change the version with `prepare` before final release validation; it also updates the citation version and resets the Conda build number for a new version. Avoid editing runtime code/version files while a benchmark is executing from an editable checkout.
+The release version is **4.0.1**. Preparing this version does not publish it; branch and PR testing still happen before a release tag is pushed. Do not reuse 3.5.1 or its existing tags. Change the version with `prepare` before final release validation; it also updates the citation version and resets the Conda build number for a new version. Avoid editing runtime code/version files while a benchmark is executing from an editable checkout.
 
 Before merging, manually dispatch the Release workflow on the feature branch with blank publication inputs. Select `test_containers` to test the exact Conda artifact inside Docker and its SIF conversion. Download `release-assets`, `container-assets`, and the security reports from Actions for review. Core integration uses explicit 2 GB task reservations, a 4 GB memory budget, and two CPUs so the small fixture fits CI runners. These limits are not production metagenome sizing recommendations.
 

metapathways/_version.py

@@ -1,5 +1,5 @@
 __author__ = "Ryan J. McLaughlin, Tony X. Liu, Tomer Altman, Aditi N. Nallan, Aria S. Hahn, Julia Anstett, Connor Morgan-Lang, Kishori M. Konwar, Steven J. Hallam"
-__version__ = "4.0.0"
+__version__ = "4.0.1"
 __maintainer__ = "Ryan J. McLaughlin"
 __contact__ = "mcglock@student.ubc.ca"
 __status__ = "Release"

metapathways/pt_container.py

@@ -17,6 +17,7 @@
 from collections import deque
 import uuid
 from urllib.request import build_opener, HTTPRedirectHandler
+from urllib.error import HTTPError, URLError
 
 from metapathways import nextflow
 
@@ -76,6 +77,32 @@
 '''
 
 
+PREPARED_BASE = Path('/opt/metapathways/ptools-base')
+
+
+def build_recipe():
+    """Use the bundled unlicensed rootfs when running the release container."""
+    if not PREPARED_BASE.is_dir():
+        return RECIPE, None
+    manifest = PREPARED_BASE / 'opt/mp-base-packages.tsv'
+    if not manifest.is_file():
+        raise RuntimeError('Bundled Pathway Tools dependency base lacks provenance')
+    recipe = RECIPE.replace('Bootstrap: docker\nFrom: ubuntu:22.04',
+                            'Bootstrap: localimage\nFrom: ' + str(PREPARED_BASE))
+    recipe = recipe.replace('    apt-get update\n', '')
+    recipe = recipe.replace('    apt-get install -y --no-install-recommends ca-certificates xterm openssl libxml2 xvfb xauth libxm4 libssl-dev procps bzip2 ncbi-blast+\n', '')
+    return recipe, dict(kind='bundled-rootfs', packages_sha256=digest(manifest))
+
+
+def inner_build_environment():
+    """Outer binds must not become mounts inside the writable installation root."""
+    env = os.environ.copy()
+    for name in ('APPTAINER_BIND', 'APPTAINER_BINDPATH',
+                 'SINGULARITY_BIND', 'SINGULARITY_BINDPATH'):
+        env.pop(name, None)
+    return env
+
+
 def digest(path):
     h = hashlib.sha256()
     with Path(path).open('rb') as f:
@@ -112,6 +139,21 @@ def installer_version(installer, explicit=None):
     return version
 
 
+def open_vendor_url(opener, url):
+    """Retry transient vendor errors without accepting redirects or partial files."""
+    for attempt in range(3):
+        try:
+            return opener.open(url, timeout=60)
+        except (HTTPError, URLError) as exc:
+            if isinstance(exc, HTTPError) and exc.code not in (429, 500, 502, 503, 504):
+                raise
+            if attempt == 2:
+                raise
+            delay = (5, 15)[attempt]
+            print(f'Official patch download temporarily unavailable; retrying in {delay}s', flush=True)
+            time.sleep(delay)
+
+
 def download_patches(version, destination):
     """Snapshot SRI's official release feed; never silently use partial downloads."""
     if not re.fullmatch(r'\d+\.\d+', version):
@@ -122,7 +164,7 @@ def download_patches(version, destination):
     files.mkdir(parents=True)
     opener = build_opener(_NoRedirect())
     try:
-        with opener.open(url, timeout=60) as response:
+        with open_vendor_url(opener, url) as response:
             listing = response.read()
         links = _PatchLinks()
         links.feed(listing.decode('utf-8'))
@@ -133,7 +175,7 @@ def download_patches(version, destination):
         for name in sorted(links.names):
             print('Downloading official Pathway Tools patch: ' + name, flush=True)
             target = files / name
-            with opener.open(url + name, timeout=60) as response, target.open('wb') as stream:
+            with open_vendor_url(opener, url + name) as response, target.open('wb') as stream:
                 shutil.copyfileobj(response, stream)
             if target.stat().st_size == 0:
                 raise ValueError('Empty patch: ' + name)
@@ -360,18 +402,19 @@ def build(installer, image, installer_sha256, threads, version=None):
         (work / 'installer').symlink_to(installer)
         patches = download_patches(installer_version(installer, version), work / 'official-patches')
         definition = work / 'pathway-tools.def'
-        definition.write_text(RECIPE)
+        recipe, base = build_recipe()
+        definition.write_text(recipe)
         partial = work / 'pathway-tools.sif'
         subprocess.run([executable, 'build', '--fakeroot', '--mksquashfs-args',
                         f'-processors {threads}', str(partial), str(definition)],
-                       cwd=work, check=True)
+                       cwd=work, check=True, env=inner_build_environment())
         validation_log = validate(partial, work / 'validation')
         metadata = dict(installer_sha256=installer_sha256, image=str(image),
-                        image_sha256=digest(partial), recipe_sha256=hashlib.sha256(RECIPE.encode()).hexdigest(),
-                        official_patches=patches, validation_log=validation_log)
+                        image_sha256=digest(partial), recipe_sha256=hashlib.sha256(recipe.encode()).hexdigest(),
+                        dependency_base=base, official_patches=patches, validation_log=validation_log)
         partial.replace(image)
         save_json(str(image) + '.json', metadata)
-        image.with_suffix('.def').write_text(RECIPE)
+        image.with_suffix('.def').write_text(recipe)
 
 
 def parser():
@@ -403,7 +446,8 @@ def main(argv=None):
             if not shutil.which(executable):
                 p.error(f'{executable} must be installed and on PATH')
     checksum = digest(installer)
-    recipe_hash = hashlib.sha256(RECIPE.encode()).hexdigest()
+    recipe, _ = build_recipe()
+    recipe_hash = hashlib.sha256(recipe.encode()).hexdigest()
     try:
         version = installer_version(installer, args.ptools_version)
     except ValueError as exc: