A local-first pipeline that ingests raw consumer DNA files from multiple tests, reconciles them into a unified variant set, cross-references that set against public annotation databases (ClinVar, GWAS Catalog, SNPedia/Promethease), enriches variants with VEP consequence predictions, assigns Y-chromosome and mitochondrial haplogroups, flags archaic-introgression overlaps, and produces a single self-contained interactive HTML report — all on the user's machine, with raw genome data never leaving it.
This is annotation, not diagnosis.
Consumer DNA arrays measure roughly 600,000–700,000 positions out of ~3 billion in the genome. At clinically relevant positions, array calls have well-documented false-positive rates; the same rsID can be a real pathogenic variant or a strand-flip artefact depending on the platform. Every association in this report is phrased as "studied in / associated with" — never "you have" or "you are safe from." That framing is a correctness requirement, not legal boilerplate: an honest annotation is more useful than a confident wrong one.
- Not a clinical tool. Not a diagnostic tool. Not FDA-cleared or CLIA-regulated. Not a substitute for a clinical genetic test or consultation with a healthcare provider or genetic counsellor.
- Pharmacogenomics entries carry the explicit note "discuss with a pharmacist; confirm with clinical PGx testing." Array data cannot reliably call high-complexity genes like CYP2D6.
- Variants flagged as "dubious" by Promethease (known platform miscalls) are visibly marked in the report and excluded from any finding-level interpretation.
- The
⚠ dubiousflag is non-negotiable. Removing it would make the output less trustworthy, not more.
| § | Module | What it does |
|---|---|---|
| §3 | extract_promethease.py |
Decompresses the Promethease HTML report (zlib+base64 JSON blobs) into a flat table of rsID → magnitude/repute/genosummary/dubious, preserving Promethease's fields verbatim without reinterpretation |
| §4 | parse_arrays.py |
Parses each raw array file, strand-normalises to forward (+), reconciles conflicts across sources by majority vote (2-of-3), keeps measured and imputed calls as separate tagged layers that are never blended |
| §5 | intersect_dbs.py |
Downloads and caches ClinVar (~440 MB+) and GWAS Catalog bulk files locally; intersects the §4 measured set against both; carries full provenance (assembly, review status, phenotype list, effect size, p-value) |
| §6 | enrich_variants.py |
Queries Ensembl VEP in batched POST requests (rsIDs only, never genotypes) for consequence prediction, SIFT/PolyPhen scores, gene symbol, transcript, and gnomAD/1000-Genomes population frequencies; results cached under cache/vep/ |
| §8 | haplogroups.py |
Assigns Y-chromosome haplogroup via yhaplo and mtDNA haplogroup via haplogrep3 in chip mode; cross-source agreement is surfaced but the honest ceiling is broad-branch (fine subclades need Big-Y or WGS) |
| §9 | archaic.py |
Intersects the §4 measured set against the Prüfer 2014 common-archaic catalog (positions where both Altai Neandertal and Denisova carry the derived allele); classifies het/hom; flags EUR-enriched subset; reports overlap counts — not a "% Neanderthal" calculation |
| §10 | report.py |
Generates a single self-contained HTML file with no external dependencies |
Interactive, sortable, client-side — no server required. Open in any browser.
Filter bar: full-text keyword search across all content fields (rsID, gene, trait, ClinVar condition, Promethease summary — "cholesterol" finds it whether it appears in GWAS, ClinVar, or SNPedia); gene prefix search; ClinVar significance dropdown (with "Actionable" shortcut); repute, dubious, imputed, magnitude, and source-platform filters. All filters AND together. Keyword search is debounced at 200ms for responsiveness at 35k+ rows.
Table columns: rsID (linked to dbSNP), locus, gene/consequence, genotype/sources, flags (⚠ dubious, ⊙ imputed, ⚡ conflict), Promethease magnitude, repute, ClinVar top significance, GWAS top trait, gnomAD frequency.
Expanded row detail: per-SNP: all ClinVar entries (not just the most severe), all GWAS associations, VEP consequence + SIFT/PolyPhen scores, full population frequencies, Promethease genosummary, dbSNP ↗ link, SNPedia ↗ link (shown only when a Promethease annotation exists, so the link reliably resolves).
Reference panels: §8 haplogroups and §9 archaic overlap are collapsible panels, collapsed by default — the main table is the primary view. Haplogroup panel: Y and mtDNA with cross-source agreement, confidence scores, and explicit note that array data gives broad-branch calls only. Archaic panel: count table with the two-reasons framing (African ancestry carries less Neandertal introgression + the common-archaic catalog is a conservative floor), EUR-enriched subset, and explicit "Not a % Neandertal measure" label.
Copy-export: "⎘ Copy TSV" and "⎘ Copy MD" buttons export the current filtered set (respecting active search/filters) as clean flat text — one row per variant, no internal newlines, sources joined with "; " — for pasting into spreadsheets or LLM prompts. The export is built from the in-memory data, not from DOM scraping, so stacked multi-line cells never corrupt column alignment.
Theme: light (default) and dark, toggled live via a fixed button; all colours are CSS custom properties; no localStorage.
Non-dismissible uncertainty banner at the top of every rendered report.
Tested:
| Format | File | Notes |
|---|---|---|
| AncestryDNA | AncestryDNA.txt |
V2.0 array; 5-column TSV (rsid chrom pos allele1 allele2); ~650k rsIDs; hg19; forward strand |
| Vitagene | Vitagene.txt |
4-column TSV (rsid chrom pos genotype); ~713k rsIDs; hg19; forward strand; no mtDNA |
| Genes for Good (measured) | GFG_filtered_unphased_genotypes.vcf.gz |
VCF 4.1; ~500k positions; hg19; includes Y and mtDNA |
| Genes for Good (imputed) | GFG_filtered_imputed_genotypes_noY_noMT.vcf.gz |
~7M positions; phased; no Y or mtDNA; treated as separate lower-confidence layer |
Structurally supported but untested: 23andMe, MyHeritage, FTDNA raw downloads likely parse with the existing 4- or 5-column TSV reader, but have not been run through the pipeline. A user with one of these formats may need to add or adjust a parser in parse_arrays.py; the parser layer is modular by design. Contributions welcome.
Build: hg19/GRCh37 expected for all inputs. rsID-keyed annotation lookups (ClinVar, GWAS, VEP) are build-agnostic, so build mismatches between sources matter primarily for the position column, not for annotation.
Python 3.10+
pip install pandas pyarrow pysam requests tqdm # core
pip install git+https://github.com/23andMe/yhaplo.git # §8 Y-haplogroup
On Ubuntu systems that enforce PEP 668: pip install --break-system-packages ... or use a virtual environment.
haplogrep3 (§8 mtDNA) is a separate Java JAR — download from the haplogrep3 releases page and note its path.
dna-annotator/
data/ ← your raw files go here (gitignored — never committed)
AncestryDNA.txt
Vitagene.txt
promethease.html
gfg/
GFG_filtered_unphased_genotypes.vcf.gz(.tbi)
GFG_filtered_imputed_genotypes_noY_noMT.vcf.gz(.tbi)
cache/ ← downloaded databases (gitignored — grows, keep between runs)
out/ ← output tables and HTML report (gitignored)
src/ ← pipeline modules
# §3 — extract Promethease annotations
python src/extract_promethease.py --input data/promethease.html
# §4 — parse and reconcile arrays (adjust paths for your files)
python src/parse_arrays.py \
--ancestry data/AncestryDNA.txt \
--vitagene data/Vitagene.txt \
--gfg-measured data/gfg/GFG_filtered_unphased_genotypes.vcf.gz \
--gfg-imputed data/gfg/GFG_filtered_imputed_genotypes_noY_noMT.vcf.gz
# §5 — ClinVar + GWAS (downloads ~440MB+ on first run into cache/)
python src/intersect_dbs.py
# §6 — VEP enrichment (batched rsID queries; cache/vep/ grows per run)
python src/enrich_variants.py
# §8 — haplogroups
python src/haplogroups.py \
--haplogrep-jar /path/to/haplogrep3.jar
# §9 — archaic introgression (downloads ~88MB catalog on first run into cache/)
python src/archaic.py
# §10 — render HTML report
python src/report.py
# → out/report.htmlFirst run: cache/ will download ClinVar (440 MB+, grows over time), GWAS Catalog, and the archaic catalog (~88 MB). Subsequent runs use the cache. VEP results are cached per-rsID under cache/vep/.
- Raw genome files live in
data/and are gitignored. They never leave the machine. - Annotation lookups send rsIDs only — never genotypes, never the reconciled call table. The VEP batch API receives a list of anonymous SNP identifiers; the ClinVar and GWAS lookups are local file intersections.
- Personal results (
RESULTS.md) and rendered outputs (out/) are gitignored. This repository contains code and method only — no one's genetic data. - The planned §14 superkit exporter (not yet built) is the one module whose output is intended to be uploaded to a third-party site. It will carry an explicit notice to that effect each time it runs, so the privacy exception is always visible.
This is architectural design, not a privacy promise. Audit the code if you need certainty.
Imputation fills the ~99.98% of the genome that consumer arrays do not directly measure, by statistically inferring genotypes from a reference panel. It is a statistical estimate, not a measurement — reliability degrades where the reference panel under-represents the user's ancestry. This pipeline handles three scenarios:
Tier 1 — if you already have an imputed file: Genes for Good and some other providers include an imputed VCF in their download. This pipeline ingests it as a separate, explicitly-tagged layer (imputed=true). Imputed rows are never silently merged with measured calls, never allowed to vote in conflict resolution, and always filterable in the report. This is the zero-effort option for users who already have the file.
Tier 2 — TOPMed Imputation Server (NIH/NHLBI): A free, federally operated service using a panel of 97,000+ diverse genomes. Technically excellent — but the service requires job submissions of 20,000–25,000 samples minimum. A lone individual submitting their own genome (1 sample) will be rejected. The sample-count range constraint is a floor, not just a ceiling; the FAQ's chunking guidance is for jobs that exceed the upper limit, not for this. Upload also means the genome transits NIH infrastructure (encrypted; passwords are never stored or resendable; results auto-delete after 7 days; accepts VCF 4.2 not 4.3; autosomes and X only). Mostly not applicable for a single individual's own data.
Tier 3 — self-hosted Michigan Imputation Server 2: The open-source backend that powers the Michigan Imputation Server is publicly available and can be run locally via Docker and Nextflow/Cloudgene. Self-hosting keeps the genome fully on the user's machine (consistent with this pipeline's privacy architecture) and removes the sample-minimum constraint. It hosts multiple panels including CAAPA (African American), which is materially better for African-admixed ancestry than the standard 1000-Genomes panel. The cost: Docker setup, ~14 GB+ panel download, 100 GB+ free disk recommended, and an afternoon of admin configuration. Not a one-click install, but the right option for users who care about both privacy and panel quality.
The planned §14 superkit exporter (measured-only, conflict-resolved) is the intended input for any imputation run — it produces the cleanest possible starting point.
Note: several commercial services sell imputation as a paid product. Some use only the dated 1000-Genomes reference panel. The free/open options above are often better.
Never commit an imputation server API key or token to this repository.
- Array resolution ceiling: ~600k–700k measured positions out of ~3 billion. The pipeline annotates what the arrays see, not the full genome.
- Haplogroup subclades: Y-chromosome assignment via yhaplo from array data gives broad-branch haplogroups; fine subclades (e.g. R-L21 → R-DF13 → ...) require Big-Y or whole-genome sequencing. mtDNA from haplogrep3 in chip mode is similarly broad-branch.
- Imputed data is a statistical estimate: tagged and kept separate from measured calls throughout, but present in the report under the
⊙ imputedflag. Filter it out if you need measured-only results. - Archaic introgression — conservative floor: the §9 common-archaic catalog (positions shared by both Altai Neandertal and Denisova) deliberately misses Neandertal-specific loci (e.g. BNC2, STAT2 region, OAS cluster) and Denisova-specific loci (e.g. EPAS1, some HLA alleles). Any count is a floor, not a total. The African-ghost-lineage signal relevant to African-admixed ancestry has no current intersectable public reference and is not computed.
- Annotation ≠ clinical truth: ClinVar entries range from "criteria provided, multiple submitters" to single-submitter VUS with no functional evidence. Review status is surfaced in the report; interpret accordingly.
- Tested input formats: AncestryDNA, Vitagene, Genes for Good. Other consumer formats are structurally similar but untested.
v1.0 — personal/exploratory project. Not monetized.
Modules §3–§10 are built and tested on the three input formats above. §11 (ancient DNA proximity) and §14 (superkit exporter) are documented in BUILD_SPEC.md but not yet implemented.
License: MIT.