Skip to content

Soft-responsibility imputation, posterior variance, and sparse dropout simulation (#564) - #571

Merged
aarmey merged 2 commits into
mainfrom
sparse-imputation
Nov 14, 2026
Merged

aarmey merged 2 commits into
mainfrom
sparse-imputation

Conversation

@aarmey

@aarmey aarmey commented Nov 14, 2026

Copy link
Copy Markdown
Member

Closes #564.

Builds on the observed-only marginalization already landed in #568 (_estimate_observed_gaussian_log_prob, observed-only _m_step / _fit_emeans, and removal of SoftImpute pre-imputation).

Summary

  • Soft-Responsibility Imputation & Uncertainty (ddmc/clustering.py):
    • Updated DDMC.impute(mode="soft", return_var=False) to default to posterior expectation imputation $\hat{X}{ij} = \sum{c=1}^k \gamma_{ic} \mu_{cj}$ (mode="soft"), while preserving hard cluster-center assignment via mode="hard".
    • Added return_var=True to return posterior predictive variances (0.0 for observed entries, law of total variance $\sum_{c=1}^k \gamma_{ic} (\sigma_{cj}^2 + \mu_{cj}^2) - \hat{X}{ij}^2$ for mode="soft", and $\sigma{c^*, j}^2$ for mode="hard").
  • Sparse Spatial / Low-Input Dropout Simulation (ddmc/datasets.py):
    • Added simulate_dropout(p_signal, missing_fraction, mechanism="mcar", col_groups=None, random_state=None) supporting "mcar", intensity-dependent "mnar" (limit-of-detection), and "block" (MS run / spatial ROI) dropout.
  • Validation Tests (ddmc/tests/test_sparse_imputation.py):
    • Verified soft vs. hard imputation and exact posterior variance calculations.
    • Verified "mcar", "mnar", and "block" dropout behaviors and input validation.
    • Verified that under 75% missingness, DDMC with observed-only marginalization and a sequence prior (seq_weight > 0) achieves higher cluster recovery (ARI > 0.85) and lower held-out reconstruction MSE than DDMC(seq_weight=0.0) and column-mean imputation.

Verification

  • uv run ruff format --check ddmc: 32 files already formatted
  • uv run ruff check ddmc: All checks passed
  • uv run ty check ddmc: All checks passed
  • uv run pytest -v: 82 passed

@aarmey
aarmey merged commit 3aea428 into main Nov 14, 2026
2 checks passed
@aarmey
aarmey deleted the sparse-imputation branch November 14, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Observed-only marginalization and sequence-prior imputation for ultra-sparse spatial and microscale phosphoproteomics (HTAN / DVP / DIA)

1 participant