Repository navigation
Conversation
generate_consensus.py no longer drops interval-end bases or crashes on missing genotypes. The accelerator no longer hangs when a BED position exceeds the pileup. PREPARE_BAM declares the BAM it writes and normalises read groups via samtools addreplacerg. MERGE_OUTPUTS keeps monomorphic sites when remove_invariant is false, using a new drop_refonly_duplicates.py to remove ALT="." records that duplicate a real variant at the same position. merge_bcf_tree.py passes --force-single for single-chunk merges. Add pytest suite covering the consensus and merge paths.
…ence depth in the AD slot so `bcftools merge --gvcf` reports honest ALT depth for hom-reference samples instead of a fabricated 0. This removes the need for `fill_missing_ad.py`, which is deleted. Mention the behaviour change for no-call records in the changelog, add a testing section to the README, and add pytest to the environment.
Parse `calling.callers` into a trimmed, deduplicated list in configured order, and use it to drive process selection, threshold validation and the consensus header. Add `calling.allow_caller_failure` (default false); callers are retried twice and the run terminates on failure unless the flag is set. In degraded mode the consensus step rejects a threshold exceeding the surviving callers and records `##callers`/`##callersFailed` provenance in the VCF header. Stage caller BCFs by name so missing callers are detectable, and pass sample ids explicitly to keep merge order stable.
Add two-stage consensus voting: an ALT allele needs at least consensus.threshold supporters, then the majority zygosity among their supporters decides 0/1 vs 1/1. Add consensus.tie_policy (nocall, first, qual) and consensus.absent_vote (ref, abstain) options and record the applied rule in a structured ##ChoCallate header line. Move quality filtering to per-caller SNP/INDEL thresholds and drop the shared calling.min_snp_qual key.
…efore voting Split multi-allelic and atomise complex alleles in every caller BCF before consensus so all callers share one representation, with exact deduplication at chunk boundaries. Consensus voting now produces a separate record per qualifying ALT instead of a single winner; the sample output re-joins them unless split output was requested, and the single-BCF merge path now honours the split option. GATK HaplotypeCaller pads intervals (calling.gatk.interval_padding, default 150) so indels near covered-island edges are assembled.
Add `bin/postprocess_bam.sh` and `bin/merge_outputs.sh` so the production pipeline and the regression tests exercise the same code. Replace the duplicated merge chain in `MERGE_OUTPUTS` and route BAM preparation through the shared helper. Add `SCATTER_INTERVALS` to call the scattered interval chunks and drop the per-process `maxForks 1` limits. Add the `mosdepth` dependency, the `reference_index_prefix` parameter, and tests for both helpers.
Replace ad-hoc CLI checks with nf-schema validation driven by nextflow_schema.json and a full defaults block in nextflow.config. The samplesheet is parsed up front for unique shell-safe ids, exact row widths, optional headers and resolvable read paths. BGZF reference detection is now header-based, removing ref_genome.bgzip, and merge tasks fail early when the file-descriptor limit is too low. Missing read slots stage a bundled NO_FILE placeholder, and the default min_coverage is raised to 5.
Record versions.yml once per run and embed the invocation and caller versions in consensus VCF headers so outputs are traceable to the exact environment. Pin all conda dependencies to exact versions and add a lock file.
Replace exact dependency pins with `>=` lower bounds in environment.yaml and update the reproducibility test to assert the new floating-minimum convention, dropping lock file checks.
- New `compress_reference_blocks` output option writes runs of hom-reference positions as gVCF blocks with INFO/END; FORMAT/DP is the block minimum and FORMAT/AD is omitted on multi-base blocks. - Set rlen on compressed blocks so CSI overlap queries see them. - Validate reference contigs against the 2^31-1 bp limit after generating the .fai. - Expand config template with full parameter documentation. - Remove the versions process/script and the `--help` handling.
Introduce process labels with CPU, memory and time limits read from the params file, and retry transient failures with escalated memory. Stage CPU options remain authoritative for their processes. Fix stage cleanup to run only after successful tasks and to remove a temporary directory only when it lies inside the work directory. Add a publish mode option and derive JVM heap sizes from task memory.
- Correct spelling errors in CONTRIBUTING.md and ROADMAP.md - Make README badges link to their targets - Drop unused RemoveInvariant and SplitMultiallelic header fields from generate_consensus.py and update its tests - Add merge_bcf_tree.py module docstring and tidy shell/import formatting in stage_cleanup.sh and CLIParamsValidation.groovy
Cover edge cases with a configurable scratch table limit, missing qualities, reference skips, indels, and a sparse reference near the 32-bit coordinate limit. Extend the BAM and reference fixtures to support CIGAR, flag, and quality overrides plus CSI indexing.
Introduce a GitHub Actions CI pipeline running ruff, Nextflow lint, actionlint, pytest (with and without the consensus accelerator), and nf-test. Add nf-test configuration with an end-to-end pipeline test, ruff config, and new dev dependencies. Drop the unused QEMU setup step from the Docker workflow and ignore nf-test and ruff cache directories.
Also rename CI matrix accelerator values from boolean to pure/accel for clearer job names and guard against missing setuptools with an explicit error.
Introduce a shared ChoCallateTestUtils helper that builds reference indexes, BAMs, and BCFs, and add per-process tests for indexing, decompression, mapping (bowtie2, bwa, minimap2), coverage, scattering, calling (bcftools, freebayes, gatk), consensus, BAM preparation and output merging. Expand the pipeline test coverage to exercise PE/SE reads, BAM input, each mapper, single vs per-sample output, VCF vs BCF format, invariant retention and multi-sample merging, and disable nf-boost cleanup so work directories survive assertions.
CPU allocation is now controlled solely by resource labels, so the per-process `cpus` overrides are no longer needed.
Add `boost.cleanup` and `mapping.aligner_reserve_mb` pipeline parameters exposed via config.yaml and the params schema. The aligner reserve is subtracted from task memory before sort memory is split, defaulting to the staged index size plus 1 GiB when unset. BAM post-processing rejects a reserve that is not smaller than the task memory. README documents the cleanup/-resume trade-off and large-genome memory sizing.
Seed deletion support from records skipped before a chunk start so reference records are not written over bases deleted in the previous chunk. Deletions below the consensus threshold or without usable ALT support are ignored.
Change output.remove_invariant default to false so all sites, including monomorphic reference records, are exported. Positions with ambiguous IUPAC reference bases or no-call consensus are still omitted. Also fix Docker image permissions and bump version to 4.0.0.
bcftools merge drops generic meta lines, so MERGE_OUTPUTS previously rebuilt ##ChoCallate= from the first sample only, hiding per-sample caller failures in degraded mode. The new merge_cho_headers.py records the union of surviving and failed callers in the main line and emits a
Extend check_bam_reference.py to compare M5 checksums from the BAM header against a supplied sequence dictionary, catching alignments to a different assembly whose contigs share names and lengths. postprocess_bam.sh gains a --fai option that runs the guard on the filtered BAM before left-alignment and indexing; mapping modules pass the reference index and PREPARE_BAM passes the dictionary.
Zero would add monomorphic positions to the coverage BED and consensus, which is not supported. Update the schema, config template docs, README, and add a test for rejection of zero.
Reject intervals on unknown contigs, with negative or reversed coordinates, or extending past the contig end, exiting with a clear error message instead of an IndexError traceback.
Reject --consensus_threshold values below 1 and --min_coverage values below 0 at argument parsing time, so invalid input fails before any files are opened.
The accelerator import now distinguishes a missing extension (expected in a plain checkout, stays silent) from an extension that exists but fails to load (still reported). Since the module is imported once per consensus chunk, warning on every chunk was noisy.
Install the pipeline under /opt/chocallate instead of /tmp, since Apptainer and Singularity bind the host /tmp and would hide it. Pin Nextflow plugins at build time so runs need no network access, and provide a world-writable HOME so the image works under arbitrary UIDs. Update the CI smoke tests accordingly and add a check that plugin resolution works offline.
Index files are staged under a flat index/ directory, so a prefix selecting a subdirectory would resolve during validation but point at a non-existent nested path in the mapping command. Validate the prefix up front and document the plain-file-name requirement.
Restrict mapping.platform and mapping.library to letters, digits, underscore, dot, colon and hyphen, both in the schema and in a full-string check that rejects trailing line terminators.
A deletion is now an observation without an allele, so an overlapping mate's base always wins over it during collapsed support tallying. This keeps DP equal to the mosdepth coverage BED while ensuring reference AD counts the mate's allele. The accelerated pileup tally now tracks query name lengths to detect hash collisions safely.
Copy sample values from the dropped reference-only record to surviving split ALT records so each retains the hom-reference genotype. Add end-to-end tests for multiallelic join/split output and for the enabled nf-boost cleanup observer.
Only treat the first non-comment line as a potential header so that data rows beginning with "id" or "sample" are validated as regular rows. Resolve read paths and the samplesheet file itself via java.nio.file.Path instead of java.io.File.
Add filter_bed.py to strip track, browser and # comment lines from include/exclude BED files before sorting, and reject contigs absent from the reference .fai with an error naming the BED file and the offending contigs.
Use distinct staging directories for the include and exclude BED inputs so files sharing the same basename no longer collide. Sentinel detection now inspects the last path component since stageAs prefixes the input path.
Major update covering parameter validation, the two-stage caller-order-specific consensus model, gVCF-style reference blocks, unified EXPORT_OUTPUT stage, shared BAM post-processing, resource labels with retry escalation, and new test suites and CI.
Defer resolution of these settings until task scheduling so overrides from a later config file or params file are honored.
Replace floating minimum version constraints with exact pins to make the environment reproducible. Also bump nextflow to 26.10.2 and pysam to 0.24.1, and pin python to the GIL-enabled CPython build since pysam has no free-threaded variant. Update the corresponding reproducibility test.
Align the installed plugin version in the Docker image with the nextflow.config declaration and downgrade Nextflow to a compatible release.
Clarify plugin version pinning, reference genome format requirements, and required index files for FASTQ input. Adjust contributing wording and update gitignore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.