Output Structure

Output structure

ChoCallate writes published results into output.directory (default: ChoCallate_output). The files depend on output.type and output.format. Internally, GENERATE_CONSENSUS always writes one BCF per sample and MERGE_OUTPUTS writes an internal merged BCF when needed; EXPORT_OUTPUT applies the requested format and the final record transformations.

Published layout

output.type	output.format	Published consensus
"single"	"bcf"	<outdir>/consensus.bcf
"single"	"vcf"	<outdir>/consensus.vcf.gz
"sample"	"bcf"	<outdir>/per_sample/<sample_id>.bcf
"sample"	"vcf"	<outdir>/per_sample/<sample_id>.vcf.gz

VCF output is bgzip-compressed. A VCF request does not change the internal representation: per-sample and merged files are BCF until EXPORT_OUTPUT.

Always published

• dropped_samples.txt: samples removed before consensus. It is published even when no sample was dropped.
• mapping_stats/: per-sample mapping statistics (see below).

Nextflow also writes pipeline_report.html, timeline_report.html and trace.txt to the paths configured in nextflow.config. Nextflow resolves these paths while parsing that file, before a params-file override of output.directory takes effect, so a custom output directory still leaves them under the default ChoCallate_output in the launch directory.

Output options

Parameter	Default	Effect
output.remove_invariant	false	false keeps monomorphic/reference sites. true drops reference-only records at export.
output.split_multiallelic	false	false re-joins split ALT records at export. true keeps one record per ALT.
output.compress_reference_blocks	true	Collapse consecutive hom-reference positions into multi-base gVCF blocks with INFO/END.
output.publish_mode	"copy"	copy writes independent files; link creates symbolic links into the work directory.

With the default remove_invariant: false, the consensus contains every covered position that has a reference or variant call, including monomorphic sites. Positions whose reference base is an IUPAC ambiguity code (N included) and positions that end as a no-call are not written.

output.publish_mode: "link" requires the work directory to be retained (for example, with boost.cleanup: false). copy is independent of work-directory cleanup.

Reference representation

Hom-reference positions use the gVCF-style symbolic ALT <*> instead of ALT=".":

• GT is 0/0.
• INFO/END marks the block end.
• Single-base records carry FORMAT/AD=(reference depth, other-allele depth).
• Multi-base blocks omit FORMAT/AD; FORMAT/DP is the minimum depth of the block.

The <*> representation lets bcftools merge --gvcf assign a hom-reference sample's non-reference depth to another sample's real ALT. output.compress_reference_blocks: false writes one reference record per covered base instead of blocks, but keeps the same <*> representation.

Consensus rule

The genotype at each position is decided by the two-stage voting model described in Consensus model. That page covers caller normalization, ALT candidacy, zygosity voting, tie policy, absence and filtered-record policies, IUPAC handling and deletions.

Header metadata

Each consensus file carries one structured ##ChoCallate= line with the applied settings, for example:

##ChoCallate=<Version=4.0.0,Threshold=2,Callers="bcftools,gatk",Rule=two-stage,Tie=nocall,AbsentVote=ref,FilteredVote=abstain,ReferenceBlocks=True,Accel=false>

Additional fields:

• CallersFailed="..." in degraded caller mode.
• SamplesDropped="..." in merged output when samples were removed.
• MinCoverage=... when absent_vote is abstain.
• ReferenceBlocks=True|False records whether reference blocks were written; it is False for per-sample export that omits reference records.
• Accel=True|False records whether the Cython accelerator wrote the records.

In a merged multi-sample file, Callers and CallersFailed are the union across samples. ##ChoCallateSamples=<Sample=...,Callers="...",CallersFailed="..."> lines preserve per-sample values when caller sets differ or any caller failed.

FORMAT fields

Tag	Description
GT	Consensus genotype.
AD	Allelic depths for REF and ALT. Counts come from the BAM pileup; overlapping mates are collapsed to one observation per read name. Multi-base reference blocks omit AD.
DP	Number of collapsed observations at the position, excluding reference skips. For multi-base blocks, the minimum depth of the block.

INFO fields

Filled with bcftools +fill-tags after consensus and after merge:

Tag	Description
NS	Number of samples with data.
AN	Total number of alleles in called genotypes.
AC	Allele count in genotypes.
AC_Hom / AC_Het	Allele counts in homozygous / heterozygous genotypes.
AF	Allele frequency.
MAF	Frequency of the second most common allele.
TYPE	Variant type (for example SNP or INDEL).
F_MISSING	Fraction of missing genotypes.
DP	Total depth summed across samples (sum(FORMAT/DP)).

Mapping statistics

Published under <outdir>/mapping_stats/ for every sample that reached mapping:

• <sample_id>.raw.flagstat: samtools flagstat of the aligner output before MAPQ filtering and duplicate handling. For BAM input, the input BAM.
• <sample_id>.final.flagstat: samtools flagstat of the final BAM consumed by coverage and calling.
• <sample_id>.stats: samtools stats of the final BAM.
• <sample_id>.idxstats: samtools idxstats of the final BAM.
• mapping_summary.tsv: one row per samplesheet entry, in samplesheet order, with alignment-record counts, raw mapped and properly paired fractions, duplicate and MQ0 fractions, mean read length and MAPQ, insert size, error rate and contig counts.

Counts are alignment records: a paired-end read pair contributes two records. The raw report is taken before bam_filter.min_map_qual and rmdup.mode are applied, so raw_mapped_pct describes the aligner output (or the input BAM for BAM input) rather than the filtered BAM. Statistics are reported even for samples later dropped by coverage or a failed caller.

Dropped samples

dropped_samples.txt lists, one per line, samples removed from the run:

• samples with an empty coverage BED when coverage.on_empty: "skip";
• samples whose every caller failed.

In a merged output the same ids appear in SamplesDropped in the ##ChoCallate= header.