Conversation
- add shared seqrepo and seqrepo data proxy providers in services - reuse shared seqrepo provider in deps to remove duplicate config - align vrs sequence proxy behavior with dcd-mapping expectations
- add shared helpers to translate hgvs, normalize alleles, and compute ids - clear cached merkle digests before identification so mutated alleles do not keep stale ga4gh identifiers - document the invariant that all allele identification must route through the helper to prevent digest correctness regressions
Introduce a reusable declarative mixin implementing transaction-time SCD Type 2 versioning for rows that change over time, used by either versioned entities or link/association rows. - A row is live while valid_to is NULL; current/as_of express the half-open [valid_from, valid_to) point-in-time predicate so call sites never hand-roll it - supersede_with (single row) and supersede_live_where (bulk) stamp the retired valid_to and successor valid_from with one timestamp for a gap-free handoff regardless of transaction boundaries - retire/retire_live_where are withdrawal primitives; retire cascades to live child links named in __retire_cascade__ - bulk supersede refuses to run on cascade-bearing classes since it cannot fire the cascade - consumers must add a partial unique index over their natural key WHERE valid_to IS NULL as a backstop against duplicate live rows Cover the mixin with unit tests against purpose-built models, using explicit timestamps far from the transaction clock to prove the gap-free handoff and verify the partial unique index backstop.
Add a lib module with reusable helpers for parsing and rewriting HGVS strings ahead of VRS translation. - extract_accession returns the reference accession (substring before the first colon), tolerant of whitespace and missing separators - split_cis_phased_hgvs expands a bracketed multivariant expression into fully-qualified components carrying the original accession and coordinate prefix, since ga4gh's AlleleTranslator only yields a single Allele per call and each component must translate alone - join_cis_phased_hgvs is the inverse, recombining components into one bracketed block and returning None when they do not share a single accession and coordinate prefix Cover the helpers with unit tests including the split/join round trip and the mixed-accession and mixed-prefix rejection cases.
Add VRS translation support for cis-phased multivariant HGVS, building on the new hgvs split helpers. - translate_hgvs_to_variation translates each component HGVS to an Allele independently, returning a bare Allele for a single component and wrapping two or more in a CisPhasedBlock, mirroring dcd_mapping's vrs_map._construct_vrs_allele; the reverse-translation job emits bracketed genomic forms the AlleleTranslator cannot translate directly - identify_variation generalizes identify_allele to blocks, clearing every member and location digest plus the block's own before identifying so a stale member digest never propagates into the block id; the block digest is order-independent so a set dedups to one row Cover both with unit tests, including the order-independent block digest and the stale-digest clearing.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
Introduce parallel mapping tables for the Better Reverse Translation epic (#746). The existing mapped_variants table is left untouched (frozen serving) while the new schema is built out separately. - add alleles, mapping_records, and mapping_record_alleles tables with their ORM models; alleles are content-addressed by vrs_digest and shared across mapping records via the link table - mapping_record_alleles.is_authoritative distinguishes the assay's actual measurement from translator-derived links, since the same VRS allele can be authoritative for one record and derived for another - put mapping_records and mapping_record_alleles on valid-time versioning (ValidTime mixin): a re-map retires the prior live row by closing valid_to instead of deleting, so history is retained and point-in-time queries are a single predicate; partial unique indexes promote "one live row per key" to the database - derive Allele.transcript and MappingRecord.transcript as hybrid properties from the HGVS columns rather than storing them, so they cannot drift; drop the stored alleles.transcript column - add the cross_level_translation annotation type, written once per variant to record whether filling unmapped levels succeeded, was skipped, or failed - annotate Variant and TargetGeneMapping with mapping_records relationships and typed primary keys
Add type stubs for the VRS and hgvs APIs used by the reverse translation work so callers can be type-checked without casts. - ga4gh.core: type ga4gh_identify and re-export PrevVrsVersion - ga4gh.vrs: stub the data proxy, AlleleTranslator, and normalize; translate_from returns Any so callers annotate the concrete VRS subtype they expect without a redundant cast - hgvs.assemblymapper: stub AssemblyMapper with the g_to_t/c_to_g/c_to_p conversions used for cross-level translation
Wire the variant-annotation package into the server extra as a local editable (PEP 660) dependency so the API can drive the reverse translation pipeline. - declare variant-annotation as a develop path dependency in pyproject.toml and the server extra; lock pulls in its transitive deps (variant-translation, biopython, openpyxl, et-xmlfile) - mount ../variant-annotation into the dev and worker containers and prepend it to PYTHONPATH so the editable install resolves - pass --no-directory to poetry install in the Dockerfile so the build does not try to install the not-yet-copied editable sibling - ignore_missing_imports for variant_annotation.* in mypy: its editable import hook is unfollowable and it ships no py.typed, so treat it as untyped rather than maintaining drift-prone local stubs - add a Makefile with dev/test/lint/format targets
Rework the variant mapping job to persist its results into the new MappingRecord / Allele / MappingRecordAllele schema instead of MappedVariant, building on the closure tables for the Better Reverse Translation epic. - create one MappingRecord per variant (including failed variants, with null VRS data); successfully post-mapped variants additionally get-or-create an authoritative Allele and link it via an is_authoritative MappingRecordAllele - supersede a variant's prior live record through ValidTime supersede_with — retiring it and its allele links and inserting the new record under one timestamp — instead of flipping a current flag, so the old/new handoff is gap-free - require a TargetGeneMapping for every mapped score, raising on a miss rather than tolerating a null link, since dcd-mapping guarantees one - combine cis-phased members when deriving hgvs_assay_level - update the mapping job tests to assert against MappingRecord and the authoritative-allele link, including the gap-free supersession and cascade-retire of the prior link TODO#765: mapping is not idempotent, so each run always creates a new authoritative link.
Add a seqrepo_data_proxy to the worker context in both the standalone context and the on_job_start hook, so jobs performing VRS translation have a SeqRepo-backed data proxy available alongside the cdot HGVS data provider. Mirror it in the mock worker context used by tests.
Add a reverse_translate_variants_for_score_set worker job that builds the cross-level HGVS equivalence class for every mapped variant in a score set, persisting the candidates as non-authoritative alleles. - for each current authoritative MappingRecord, collapse the assay-level HGVS to its protein consequence and expand to all coding/genomic candidates via a single batched construct_equivalent_variants call from the variant-annotation library, run off-loop in a thread - resolve each record's coding (NM_) transcript from its target gene's cdna alignment, falling back to a batched UTA NP_→NM_ lookup for protein-level mappings; records with no coding transcript are skipped - translate each candidate to VRS and write it as a get-or-created Allele linked non-authoritatively to the record, deduping by vrs_digest and never relinking the record's authoritative allele - supersede the prior live derived links with the new set in one gap-free operation via ValidTime.supersede_live_where - record per-variant cross_level_translation annotation status (success / failed / skipped), retaining per-candidate translation errors as metadata; the job fails only when every variant fails - add WorkerCoordinateTranslator and NullTranscriptSource adapters for the library's translation ports, deferring AssemblyMapper init so its network calls do not fire under mocked unit tests - get_or_create_allele helper, job registration, and a pipeline dependency placing reverse translation after mapping and before CAR submission TODO#765: a re-run supersedes the whole derived set wholesale because re-mapping re-mints the records; idempotent records would let unchanged derived links stay live.
…n date Previously the cdna-transcript lookup was keyed by target_gene_id alone, which meant a re-mapped score set could bind a stale NM_ transcript from an earlier run rather than the one the current run emitted. - Key cdna_transcript_by_run on (target_gene_id, mapped_date); within a key the highest-id row wins, so a same-run replacement takes precedence. - Carry mapped_date through the MappingRecord query so each record anchors to its own run's cdna row. - Add _TranscriptResolutionSkipReason to distinguish recoverable skips (protein-coding target, transcript unresolved) from correct skips (non-coding/regulatory target, no protein consequence). Emit skip_category in annotation_metadata. - Add target_gene_id to _TranscriptResolution so skip classification can look up the target's TargetCategory. - Add Mapped[] type annotations to TargetGene.id and .category columns. - New tests: genomic-accession coding target RT, latest-cdna-row-within- run selection, stale-cdna-row isolation, skip classification (coding recoverable vs. regulatory correct), and cdna TGM persistence in the mapping job. Refs mavedb-api#763
Forward translation emits predicted protein consequences in parentheses (e.g. p.(Ala222Val)). The parens denote inference, not a distinct variant form, so normalize to the bare p. form before storing. Strings without prediction parens are returned unchanged. Includes parametrized unit tests.
…st collisions - Re-identify each component via normalize_and_identify after translate_from so the stored vrs_digest reflects current content, not a stale value cached by the reused AlleleTranslator's Merkle tree. Without this, distinct biological variants at the same position (e.g. A>C and A>T) share one digest and are merged by the digest-keyed get_or_create_allele. - Use ga4gh_identify with in_place="always" in identify_allele and identify_variation so an allele that already carries a translator-stamped id has it recomputed, not retained. - Coerce ReferenceLengthExpression -> LiteralSequenceExpression in normalize_and_identify, mirroring dcd_mapping's _rle_to_lse, so RT-built alleles hash identically to the mapper's authoritative alleles and dedup correctly across sources. - Add regression tests: stale-id overwrite, distinct-alt digest isolation, cis-phased ordering canonicalization, and RLE->LSE coercion.
…tighten comments - Emit the protein consequence (result.hgvs_p) as a protein-level member of the equivalence set alongside the coding/genomic candidates. Prediction parens (p.(Ala222Val)) are stripped via strip_protein_prediction_parens before translation and storage. Protein-assay inputs are excluded (the protein is already the authoritative allele). - Trim all inline and docstring comments to essential why; remove redundant prose throughout reverse_translation.py. - Add tests: protein allele persistence, skip-classification for all three skip categories (no_assay_level_hgvs, transcript_unresolved, no_coding_transcript).
…rom failures Previously, variant mapping success was determined solely by the presence of pre/post-mapped alleles. This conflated genuinely failed mappings with benign absences (intronic variants, no-protein-consequence variants) that legitimately produce no allele. - Add `MappingOutcome` enum to `lib/mapping/schema.py` mirroring `dcd_mapping.schemas.MappingOutcome`, with an `is_benign_absence` helper to classify INTRONIC and NO_PROTEIN_CONSEQUENCE outcomes - Rewrite the per-record outcome logic in `map_variants_for_score_set` to branch on the typed outcome rather than allele presence: MAPPED -> SUCCESS, benign absence -> SKIPPED, FAILED -> FAILED - Replace the `successful_mapped_variants` scalar with a typed `Counter[MappingOutcome]` that feeds `mapped_count`, `failed_count`, and `skipped_count` tallies in logs and final state decision - Derive `mapping_state` from genuine failures only; all-benign result sets are treated as complete, not failed - Raise `NonexistentMappingResultsError` when a score annotation has no `outcome` field (malformed/older payload) - Expand test coverage for intronic and no-protein-consequence scenarios, verifying correct status and failure-category assignment
… UTA-backed source WtCodonMode.ALL reads the reference codon via TranscriptSource.codon_at, which requires a real UTA connection. The previous NullTranscriptSource always returned None, silently breaking WT-codon resolution. - Extract `uta_transcript_source()` context manager into `translation_ports.py`; removes the ad-hoc UTA connection setup that was duplicated in the job and the now-deleted NullTranscriptSource - Scope the live UTA client around the full `run_in_executor` call so the connection outlives the synchronous executor block - Remove NullTranscriptSource; callers that only need transcript_for_protein (already resolved by the job) also benefit from the real client without extra cost - Add a TODO noting that non-substitution consequences (del/ins/delins/ fs/ext) are miscounted as FAILED rather than SKIPPED (#767)
…ndle null gracefully - `translate_hgvs_to_variation` now attaches an `Expression` to each allele produced from a cis-phased or single HGVS string, mirroring the dcd_mapping authoritative-allele convention so `post_mapped` is self-describing without a separate round-trip. - `hgvs_from_vrs_allele` returns `None` instead of crashing when `expressions` is null or empty (valid for cis-phased block members), and `get_hgvs_from_post_mapped` propagates that as a `None` result. - `post_mapped` is now serialized with `exclude_none=True`, matching the mapper's output format. - Minor formatting clean-ups in reverse_translation.py (no logic change).
… model - `submit_score_set_mappings_to_car` now operates on Allele rows (authoritative + RT-derived) rather than MappedVariant, deduplicating by allele_id so each VRS allele is registered exactly once regardless of how many variants share it. Adds `force_reregister` param and per-allele outcome counters. - `submit_score_set_mappings_to_ldh` queries MappingRecord + Allele for pre/post-mapped data instead of the deprecated MappedVariant join. - `construct_ldh_submission_entity` signature updated to accept MappingRecord and Allele separately, since those fields now live on different models. - `warm_clingen_cache` switched to the shared `get_alleles_for_score_set` helper to keep allele scope consistent across all three jobs. - Extracts `get_alleles_for_score_set` and `ScoreSetAlleleRow` into `lib/clingen/alleles.py` as the single canonical query for both CAR and cache jobs.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
…-keyed refresh Replace MappedVariant-based gnomAD linkage with valid-time GnomadAlleleLink rows keyed on the deduplicated Allele. Linking covers every current allele of a score set (authoritative and RT-derived), so protein/coding score sets — whose genomic allele is RT-derived — are no longer dropped; per-variant annotation status flows through the _annotate_gnomad choke point against authoritative links only (interim bandaid, the AnnotationEvent migration seam). Refresh / idempotency model: - One live link per allele (unique index on allele_id); a gnomAD version bump supersedes rather than accumulating one live link per version. - Supersede only on change — an unchanged re-run leaves the live link untouched, so the valid-time history records no spurious boundary. - Version-keyed skip avoids re-fetching alleles already current at the version; a force param bypasses the skip (re-ingestion / heal) without churning unchanged links. - Per-variant status is a per-run audit event: created / preexisting / skipped. Normalize CAIDs across the Athena join: the gnomAD Hail dump drops leading zeros (CA025094 -> CA25094), so exact-string matching silently missed zero-padded CAIDs. Extract group_alleles_for_annotation as the shared allele-grouping primitive for allele-subject annotation jobs (adopted by gnomAD; VEP/ClinVar to follow). Refs #742. Partially addresses #722 (CAID-completeness tracking there stays open).
Migrate the VEP functional-consequence job (Step 2 of the annotation infrastructure migration) off MappedVariant onto the deduplicated allele model. - New ValidTime vep_allele_consequences table: a single allele-keyed row collapses record + link (the consequence is a scalar with no shared external entity, unlike gnomAD/ClinVar). One live consequence per allele via a partial unique index. - Job runs over the score set's full allele set (authoritative + RT-derived) via get_alleles_for_score_set + the shared grouping primitive; VAS still fans only to authoritative variants (the bandaid seam). - Version-key the refresh on the Ensembl release (/info/software): skip alleles already current, abort if the release can't be fetched. Supersede is value-keyed, not version-keyed — an unchanged consequence at a new release bumps source_version/access_date in place to avoid churning history. force bypasses the skip (e.g. after editing VEP_CONSEQUENCES). - A no-result is treated as a non-answer, never a negative: held consequences are not retired on an empty/failed VEP fetch. - Annotation links stay one-directional to Allele (no reverse back-ref). Adds lib + job tests covering linkage, RT-derived scope, version skip, in-place bump, supersede-on-change, no-result handling, and release-fetch failure.
Step 3 of the #742 external-annotation migration. ClinVar linkage moves off MappedVariant onto the deduplicated allele model. - Rename clinical_controls -> clinvar_controls (ORM model + table + unique constraint). Internal only: the ClinicalControl* view models, the /clinical-controls serving endpoints, and the frozen mapped_variants_clinical_controls association are unchanged, since the view-model name is both the OpenAPI schema name and the record_type discriminator consumed by the UI. - New clinvar_allele_links ValidTime table + ClinvarAlleleLink model. Multi-live: partial unique index (allele_id, clinvar_control_id) WHERE valid_to IS NULL, so an allele accumulates one live link per release. - Refactor refresh_clinvar_controls onto get_alleles_for_score_set + group_alleles_for_annotation (payload = CAID, full allele scope). Links are get-or-create; a same-version re-resolution to a different control supersedes newest-wins (gap-free retire+insert) rather than leaving two live links. VAS writes funnel through the _annotate_clinvar choke point, fanned only to authoritative_variant_ids and version-scoped. - Additively capture ClinVar's VariationID: nullable clinvar_variation_id column populated forward from the variant_summary TSV (parse degrades to None on archival schemas lacking the column). Unserved; the dedicated clinvar_variants remodel is deferred to the read-cutover. Tests rewritten for the allele model, covering the multi-live link writes, the version-scoped supersede guard, and the authoritative-only VAS fan-out.
Steps 4 & 5 of the #742 migration. Both jobs are redundant under the allele model: Allele.hgvs_g/c/p are populated by the mapping job, and the reverse-translation equivalence space (genomic/coding/protein alleles linked per MappingRecord) replaces the ClinGen PA<->CA translation table. - Remove populate_hgvs_for_score_set and populate_variant_translations_for_score_set from the pipeline DAG (both were leaf nodes — no dependency edges to repair), the worker registry (BACKGROUND_FUNCTIONS + STANDALONE_JOB_DEFINITIONS), and the external_services package exports. - Delete the two job modules and the now-orphaned ClinGen HGVS helpers (extract_hgvs_from_ca_allele_data / extract_hgvs_from_pa_allele_data), used only by the HGVS job. - Keep lib/variant_translations.py and the variant_translations table/model, marked FROZEN (serving-only) — they back old-model serving and are dropped at read-cutover. - Delete the obsolete job tests and their conftest fixtures.
get_allele_translations(db, allele_id, *, as_of=None) returns an allele's full cross-layer equivalence set (genomic/coding/protein) by traversing the MappingRecordAllele link graph: allele -> its live links -> mapping record(s) -> all co-linked alleles. The relation is co-membership in a MappingRecord's allele set, not a shared identifier — ClinGen's CAID spans only the nucleotide layers (the protein allele carries a distinct PA), so the link graph is the only thing tying all layers together. This replaces what the retired variant_translations PA<->CA table provided. Forward-compatible with temporal reads: the same half-open valid-time predicate is applied at both the anchor and fan-out hops, so passing as_of reconstructs the equivalence set as of any instant. Defaults to the currently-live set.
…bulary Introduces the AnnotationEvent log: a single append-only event table whose subjects are Variant and Allele, selected by annotation_type via a polymorphic CHECK. "Current" is derived (DISTINCT ON … id DESC), never stored. Adds the shared Disposition (present/absent/not_applicable/failed) and EventReason vocabulary, the v_current_annotation_events view, and the supporting migrations.
No identifier or behavior changes. Rewords the sibling/cousin terminology in prose (see the companion refactor commit for code identifiers), cuts repetitive design-history narration, and fixes one inverted claim in lib/allele_measurements.py's module docstring: a protein assay's consequence is reachable through the shared protein node precisely when it has *not* yet been reverse-translated, not "even when" it has.
…pping-and-storage
…pping-and-storage
vrs_digest asserts that an allele's identifier is the GA4GH digest of its own post_mapped content, but nothing enforced it: writers adopted the id minted by the producer, which for mapped variants is computed before normalization. Because vrs_digest is a UNIQUE dedup key on an immutable, ValidTime-less table, a wrong value there cannot be corrected in place afterwards. Add canonical_variation_document() to re-derive identity and a single canonical serialization for any variation arriving from outside, and route both the live mapping ingest and the mapped_variants backfill migration through it so identity is recomputed rather than inherited. Add the audit_allele_identifiers script to measure drift before and after the backfill and to repair safely-correctable rows.
The API contribution stamped date=datetime.today(), which is not provenance: it records when a view was serialized, not when anything was contributed. What this contribution asserts is which software produced the annotation, and that is the agent's version. The wall-clock stamp also made the object unique per call, so identical provenance never deduplicated and va.ndjson changed bytes on every run over unchanged data, defeating checksum manifests and incremental sync for consumers of the public dump.
functional_consequence has been the only stored fact about a VEP result: no record of which transcript_consequences entry it came from, or whether it was a transcript-specific match at all versus VEP's cross-transcript most_severe headline, which regularly describes a different overlapping isoform than the allele's own transcript (#772). Add four nullable columns to vep_allele_consequences: consequence_terms (every term from the matched transcript entry, severity-ordered), consequence_source (transcript / most_severe / reference_identical, mirrored from variant_annotation.lib.vep.ConsequenceSource by VepConsequenceSource), matched_transcript (set only when the source is transcript), and resolver_version (variant_annotation.lib.vep.RESOLVER_VERSION at write time). resolver_version pairs with the existing source_version as the second axis the current-release skip keys on, so a resolution-rule fix re-queries every allele instead of looking current forever under an unchanged Ensembl release. All nullable with no backfill: a legacy NULL never matches the current resolver_version, so it is re-queried once and filled in place.
…rovenance lib/vep.py's local VEP_CONSEQUENCES ranking, get_functional_consequence, and run_variant_recoder duplicated variant_annotation.lib.vep's resolution rule under a different, staler ranking. Drop them in favor of resolve_consequences over the shared library, so this job and the offline CLI pipeline resolve a consequence identically by construction, and write the new provenance columns (consequence_terms, consequence_source, matched_transcript, resolver_version) in link_vep_consequences_to_alleles alongside the existing headline-term write. Add VepLinkVerdict.RETAINED_ON_ABSENCE: when VEP returns nothing for an allele that already has a live consequence, the prior value is kept rather than overwritten with a null, since a settled negative from one run should not erase a previously confirmed positive. The worker job now builds a transcript-aware VepInput per allele (only coding and genomic alleles are submitted; a protein allele's consequence is carried by the coding alleles its reverse translation enumerates from, so it is never VEP queried directly), wires in UTA's coding_interval_reference for best-effort reference-identical detection, and gates the current-release skip on resolver_version in addition to source_version so a resolution-rule change re-queries every allele instead of looking current forever. Drops the protein/Recoder end-to-end test now that protein alleles are never submitted.
…anscript-matched populate_vep_for_score_set was disabled (#772) because its consequences came from VEP's top-level most_severe_consequence, the worst call across every transcript overlapping a position rather than the one the variant was mapped to. Now that the worker job resolves against the allele's own transcript via the shared resolution kernel, re-add the job to the pipeline definition. Depend on reverse_translate_variants_for_score_set rather than the old submit_score_set_mappings_to_car: VEP needs the coding/genomic HGVS reverse translation produces, not confirmation that mappings were submitted to CAR.
…skip, not a failure (#767) Split the reverse-translation error loop on the library's typed reason: NOT_TRANSLATABLE maps to (not_applicable, not_translatable) and is counted as skipped; every other error keeps the failed path. Adds the NOT_TRANSLATABLE event reason. This keeps benign "nothing to translate" variants out of the FAILED tally, which the retroactive backfill (#747) depends on for a trustworthy failure count.
…ss retired MappedVariant surfaces
Add a shared deprecation helper and route every retired surface through it, so the
/mapped-variants 301 redirects, the /score-sets/{urn}/mapped-variants 410 tombstone, and the
deprecated CSV query parameters all emit consistent Deprecation / Link / Warning headers and
record each hit under one queryable deprecation_marker. The redirects name their 1:1 successor
as rel=successor-version; the tombstone (no wire-compatible successor) points at variant-details
via Warning. Sunset stays unset until a removal date is ratified. The usage marker makes 'who is
still on a deprecated surface' answerable before that date is chosen.
…e for its deprecation window
mapped/{urn}.mapped-variants.json is the last dump artifact sourced from the frozen MappedVariant
table; vrs/{urn}.vrs.ndjson supersedes it with a re-shaped (correct) representation. Gate its
emission on EXPORT_LEGACY_MAPPED_VARIANTS (default on) so it drops cleanly at the chosen version
boundary -- one operational change, not a code edit -- and removes with the table drop. Record the
deprecation and the re-shape rationale in the dump changelog.
…concile gate backfill_campaign.py adds two read-only commands for the MappedVariant -> allele-substrate backfill. 'coverage' reports each published score set's furthest lifecycle state (none -> partial_reshape -> reshaped -> rt -> enriched) with a roll-up. 'reconcile' is the gate that blocks dropping the frozen mapped_variants tables: it asserts measured-level coverage parity, zero per-variant gnomAD/ClinVar/VEP annotation regression, and no query reader of the legacy tables outside the migration/export allowlist, exiting non-zero unless the drop is safe.
…pping run (#763) Reverse translation resolved a variant's coding transcript from cdna TargetGeneMappings keyed by (target_gene_id, mapped_date). mapped_date is day-granular, so two mapping runs of one score set on the same calendar day were indistinguishable: a later same-day remap that emitted no cdna row would bind the earlier run's stale transcript instead of correctly skipping. Record the producing JobRun on each TargetGeneMapping (nullable FK, ON DELETE SET NULL) and resolve a record's transcript by its own run via (job_run_id, target_gene_id), so a run with no cdna row resolves to nothing and the variant is skipped as transcript-unresolved. Rows with no job_run_id (pre-column, or reshaped from legacy) fall back to the day-granular key. The resolution is contained in a single _cdna_transcript_resolver factory. Adds same-day two-run and mapped_date-fallback regression tests.
The dev dependency allowed ruff ~0.15.0 while the pre-commit hook pins v0.6.8, and the two format some idioms differently (dict-argument call hugging, chained-call breaks). Since the hook is what gates commits, running a newer local ruff rewrote code the hook then reverted, thrashing diffs. Pin the dev dependency to 0.6.8 so poetry run ruff and the hook agree; bump both together in future.
Cis-phased multivariant assay HGVS (`NC_…:g.[a;b;…]`) cannot be forward-translated -- the engine's parser rejects the allele-list opening bracket. These were previously sent to the engine anyway, where they counted as failures that the drop gate reads as a regression. - add `is_cis_phased_hgvs` to key on the allele-list opener while excluding tandem-repeat brackets (`c.101_102[4]`) - screen cis-phased inputs out of the reverse translation job before they reach the engine - add `CIS_PHASED_UNSUPPORTED` event reason and record these as a benign NOT_APPLICABLE skip instead of a failure
…t request.url
A mapped-variant URN is {score_set_urn}#{n}, so request.url truncates at the '#'
when Starlette rebuilds it by re-parsing the decoded path -- dropping the variant
number, the sub-resource, and the query string that followed it. The redirect
target itself was unaffected (it's built from the routed urn path param), but the
query string appended to it and the surface logged to record_deprecated_usage
were both silently wrong for any request combining a fragment-shaped URN with a
real query string.
Renamed the helper to _redirect_to_successor -- _redirect didn't say what it
actually built (an RFC 8594 deprecation response), and the module already uses
"successor" as the term for the replacement resource.
map_variants_for_score_set delegates the whole score set to dcd-mapping in a single opaque blocking call, so progress_updated_at cannot advance for its duration. cleanup_stalled_jobs (PROGRESS_STALL_MINUTES=30) then falsely reaps any set that maps for longer than 30 minutes, and the retry re-runs from scratch and is reaped again -- large sets can never complete. Bracket the map call with a liveness keepalive that refreshes the heartbeat on a cadence under the stall threshold, plus a size-aware asyncio.wait_for budget that bounds a wedged mapper instead of riding the 23.5h wall-clock backstop. The keepalive runs in the worker event loop, so it dies with a crashed worker and genuine crashes still go stale and get reaped. Interim workaround for delegating opaque work; the durable fix is the mapper reporting progress or chunking the call. Principle documented in best_practices.md under "Long external delegations".
…ures plan_enqueue walked run_score_set_pipelines' cohort in a single fixed pass (gene cluster, then URN), and a FAILED score set looked identical to a never-run one. A handful of chronically-failing score sets early in that order would therefore win the same concurrency slots on every re-invocation forever, starving score sets later in the order that had never been attempted, and stalling the whole cohort behind a known error. Untried entries now always fill available slots first, in cohort order; only leftover slots go to entries whose latest pipeline failed, also in cohort order so cache-coherent per-gene fill still holds within each pass. latest_pipeline_by_score_set computes each score set's latest pipeline once and is shared between planning and the report table.
The editable sibling path dependency only resolved when variant-annotation happened to be checked out next to this repo, so it couldn't be installed in CI or a fresh clone. Point it at the bbi-lab/variant-annotation v0.1.0-dev.1 tag instead, which resolves the same way everywhere and drops the need for the Dockerfile's --no-directory workaround.
bencap
force-pushed
the
feature/bencap/allele-centric-mapping-and-storage
branch
from
September 18, 2026 23:58
066ebb6 to
be449bb
Compare
This was
linked to
issues
Sep 19, 2026
Coverage Report for CI Build 36060064271Warning No base build found for commit Coverage: 90.027%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
…s collection
Four test modules broke collection under the "Pytest on Core Dependencies" CI job
(no fastapi/starlette/hgvs/variant-annotation installed):
- tests/lib/annotation/conftest.py hard-imported VariantAnnotationContext, which now
reaches mavedb.lib.cat_vrs -> logging.context -> fastapi. Every test module in the
directory already requires psycopg2 (also a server extra) for its DB fixtures except
test_flatten.py, which needs neither — so falling back to None on ImportError, matching
the try/except already used for conftest_optional.py, costs nothing under core deps and
keeps that one file collectible.
- tests/lib/test_deprecation.py only exercises deprecation_headers (no logging.context
dependency), but the module import drags fastapi in through record_deprecated_usage;
needs its own importorskip("fastapi") since it isn't otherwise psycopg2-gated.
- tests/lib/test_vep.py and tests/lib/test_vrs_utils.py were missing importorskip guards
for variant_annotation/hgvs, the established pattern used elsewhere in the suite.
tests/db/test_mixins.py had no guard at all and errored at fixture setup (not collection),
which the interrupted run above was masking.
models/gnomad_variant.py and models/clinical_control.py mixed a Mapped[] annotation with a legacy Column() call for their Float and nullable-String attributes; the SQLAlchemy mypy plugin only infers the left/right-hand types correctly from mapped_column(), so switched those declarations to it (the other Column() fields in each class weren't affected and are left as-is). lib/gnomad.py and scripts/audit_allele_identifiers.py each read an Optional/wider-union value that a query or the callee's own contract already guarantees is narrower at that point (a WHERE clause excluding nulls; vrs_object_from_mapped_variant only ever producing an Allele or CisPhasedBlock despite its wider declared return type) — assert the invariant mypy can't see rather than loosening the callee's real contract.
bencap
force-pushed
the
feature/bencap/allele-centric-mapping-and-storage
branch
from
September 21, 2026 19:22
94907a4 to
764375f
Compare
bencap
marked this pull request as ready for review
September 21, 2026 19:23
…ect UTA - Bump variant-annotation to v0.1.0-dev.2, which retries transient UTA connection failures in reverse translation - Record TranslationErrorReason.UPSTREAM_UNAVAILABLE as failed/api_error so outages stay distinguishable from genuine translation errors - uta_transcript_source yields the reconnecting UtaClient.from_url client
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allele-centric data model: deduplicated VRS alleles, valid-time versioning, and annotation events
Replaces MappedVariant as the source of truth for mapping/annotation with a deduplicated, content-addressed Allele model, and moves every downstream annotation job (VEP, ClinVar, gnomAD, ClinGen CAR/LDH) onto it.