Skip to content

Speed up DDMC.fit: faster responsibility matmuls and cached background PWM - #576

Merged
aarmey merged 1 commit into
mainfrom
perf-matmul-bg-cache
Nov 14, 2026
Merged

aarmey merged 1 commit into
mainfrom
perf-matmul-bg-cache

Conversation

@aarmey

@aarmey aarmey commented Nov 14, 2026

Copy link
Copy Markdown
Member

Follows from profiling DDMC.fit (context: #566's scalability target).

1. Responsibility matmuls. resp.T @ X with tall-skinny resp (many peptides, few clusters) is very slow under multithreaded OpenBLAS on a 48-core machine: 73 ms vs 7.4 ms for the equal (X.T @ resp).T on a 6,022 x 1,000 batch. It runs 2-3 times per mini-batch step and in the M step, so a new helper _rtx uses the faster form everywhere. The gain depends on core count and will be smaller on machines with fewer cores.

2. Background PWM cache. Binomial.__init__ rebuilt the background PWM from about 360k PhosphoSitePlus sequences on every fit, but it depends only on the pY/pS/pT counts and include_ptms. It is now memoized in _cached_background_pwm (BackgroundSeqs is split so the counts can be the cache key), and __init__ copies it before mutating.

Timings (48-core box, Binomial, k=50 for the synthetic runs):

before after
CPTAC 1,311 x 215, warm fit (k=30) 0.74 s 0.29 s
Sequence-model setup, warm 0.49 s 0.01 s
Synthetic 60,000 x 1,000, no missing 44 s 27 s
Synthetic 60,000 x 1,000, 50% missing 59 s 34 s

The first fit in a process still pays the cache-fill cost (about 1.1 s).

Equivalence. I compared scores_ from main and this branch for Binomial and PAM250, with and without PTMs, on complete and 50%-missing CPTAC data. The maximum difference was about 1.5e-11 (floating-point reordering). Ruff, ty and pytest (98 passed) are clean.

Not addressed here: the ufunc.at scatter-adds and centroid initialization (5-9 s at scale).

@aarmey
aarmey merged commit a565a92 into main Nov 14, 2026
2 checks passed
@aarmey
aarmey deleted the perf-matmul-bg-cache branch November 14, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant