Developer Guide

Developer guide

This page describes the repository layout, process conventions and release checks. User-facing behavior is documented in the other wiki pages.

Repository layout

Path	Purpose
main.nf	Workflow entry point and channel wiring
modules/local/<name>/main.nf	One Nextflow process per stage
bin/	Python and shell helpers used by processes and tests
lib/CLIParamsValidation.groovy	Samplesheet parsing, caller/index/read-group/ploidy validation, merge ulimit check
lib/ResourceUtils.groovy	JVM -Xmx calculation from task memory
conf/base.config	Process labels, resource limits, retry policy, stage CPU precedence
nextflow.config	Parameter defaults, plugins, reports, manifest, process shell
nextflow_schema.json	nf-schema validation contract
assets/templates/config.yaml	Documented configuration template
tests/python/	pytest regression suite
tests/nf/	nf-test module and pipeline suite
test_run/	Small end-to-end dataset
Dockerfile	Relocatable, offline-capable image

Process conventions

• Declare exactly one process_low, process_medium or process_high label.
• Use beforeScript 'export TMPDIR="$(mktemp -d -p "${PWD}")"' when the task needs temporary files.
• Use afterScript 'stage_cleanup.sh "${nxf_main_ret:-0}"' so successful tasks remove staged inputs and failed tasks keep them for debugging.
• Process scripts run under bash -euo pipefail from nextflow.config.
• Stage inputs under tmp/ and write named outputs. Do not rely on host paths at task runtime.
• Preserve caller identity in staged file names (tmp/callers/<caller>.bcf) rather than positional names.
• Use ResourceUtils.javaHeapOption(...) for Java tools and account for parallel JVMs where needed.
• Process-level cpus is used only for single-CPU low-label tasks; stage CPU precedence and resource limits live in conf/base.config.

Helper scripts

Script	Purpose
bin/postprocess_bam.sh	Shared BAM pipeline: collate, fixmate, sort, markdup, MAPQ filter, left-align, CSI index
bin/filter_bed.py	Strip BED headers/comments and validate contigs against .fai
bin/check_bam_reference.py	Compare BAM @SQ names/lengths and M5 checksums with the reference
bin/check_contig_lengths.py	Reject contigs above 2^31-1 bp
bin/mapping_stats.py	Build mapping_summary.tsv from flagstat/stats/idxstats
bin/generate_consensus.py	Consensus caller and writer (pure pysam or Cython accelerator)
bin/build_consensus_accel.py	Build the optional Cython extension
bin/merge_bcf_tree.py	Chunked FIFO merge with bcftools merge --gvcf
bin/merge_outputs.sh	Production merge/export chain
bin/drop_refonly_duplicates.py	Remove redundant reference-only records after split and preserve hom-reference sample values
bin/merge_cho_headers.py	Aggregate ##ChoCallate= metadata for merged output
bin/stage_cleanup.sh	Success-only cleanup of staged inputs and $TMPDIR

Configuration architecture

• nextflow.config defines defaults for every parameter and sets process.shell.
• nextflow_schema.json validates enums, minimums, required files and unknown keys through validateParameters().
• assets/templates/config.yaml is the user-facing template and must document every parameter.
• lib/CLIParamsValidation.groovy handles checks that JSON Schema cannot express: samplesheet rows, caller parsing, BAM/reference compatibility, index prefixes, ploidy arguments, read groups and the merge descriptor limit.
• -params-file values are deep-merged into the defaults.

Adding a caller or mapper

A new caller needs:

1. A module under modules/local/ with process_medium or process_high.
2. An entry in AVAILABLE_CALLERS, the calling schema and the template.
3. Selection and channel wiring in main.nf.
4. A stable output name (<caller>.bcf) and normalization/soft-filter behavior consistent with the existing callers.
5. Tests in tests/nf/modules/local/ and consensus cases if the calling model changes.

A new mapper needs:

1. A module that stages the mapper-specific index set and writes mapping.bam, .csi and mapping.raw.flagstat.
2. An index suffix set in main.nf and the schema enum.
3. Documentation of the index format and prefix requirement.
4. Mapping tests and a pipeline case.

Tests and CI

pytest -q tests/python
nf-test test
ruff check .
nextflow lint .
actionlint

Run python bin/build_consensus_accel.py first to exercise the accelerated consensus path. Snapshot updates use nf-test test --update-snapshot; CI mode uses nf-test test --ci. See Testing and CI for details and the current test matrix.

Release checks

1. Run pytest and nf-test; re-record snapshots only for intentional output changes.
2. Run ruff check ., nextflow lint . and actionlint.
3. Update manifest.version in nextflow.config and the CHANGELOG.md entry.
4. Confirm the default output still contains all covered sites, including monomorphic positions.
5. Build the Docker image and verify the accelerator import and offline plugin resolution.
6. Do not combine output.publish_mode: "link" with cleanup when publishing linked results.