Repository navigation
PR for 0.5.0 release - #18
Merged
Merged
Conversation
Why: several components sampled per-genotype latents in plates sized
data.batch_size. Every numpyro autoguide -- including the AutoDelta
behind MAP, pre-MAP and tfs-prefit-calibration -- then held one
parameter per batch position instead of per genotype. The full-batch
index is binding-first and reshuffled each step, so bulk genotypes'
latents were scrambled even without mini-batching. The component SVI
guide was unaffected.
What:
- activity/{hierarchical,horseshoe}_geno, dk_geno/hierarchical_geno,
ln_cfu0/{hierarchical,hierarchical_factored}, theta/categorical_geno
and noise/logit_normal sample in a library-sized
{name}_genotype_plate and slice with data.batch_idx; latent priors
and guides are no longer wrapped in scale_vector.
- Theta-noise components take an optional data argument, passed by
generative/model.py. noise/beta's {name}_dist stays batch-shaped (a
documented exception).
- New inference/batch_safety.py detects batch-dependent latents by
tracing at two batch sizes (under jax.eval_shape), and
orchestrator_latent_dimension sizes an AutoContinuous latent vector.
- run_optimization runs the check before fitting with any AutoGuide
and refuses an unsafe model; for AutoMultivariateNormal it reports
the dense-covariance size and warns above 4 GB.
- get_map_posteriors/get_laplace_posteriors refuse MAP checkpoints
with batch-sized genotype latents instead of clipping indices.
- draw_prior no longer returns plate sites as latents.
- CLAUDE.md: component-authoring step 6; CHANGELOG entry.
Effects: MAP/pre-MAP/prefit fits change numerically for bulk
genotypes; checkpoints written before this fix must be refit. An
autoguide fit of a batch-dependent model (e.g. MAP with beta theta
noise) now fails fast instead of silently scrambling.
Verification: test_batch_safety covers all registry axes (plus
epistasis); an ad hoc sweep including thermo PnnC/PddG found only the
beta exception. New tests cover the run_optimization guard and the
latent-dimension count. pytest tests/tfscreen: 4527 passed, 6
skipped; tfmodel/inference + generative + scripts --runslow: 2356
passed.
Co-authored-by: Suryansh Singh Thakur <suryanshsinghthakur001@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: the component guide was the only variational family available to tfs-fit-model. Comparing it against numpyro's autoguides, and eventually trying a dense or low-rank MVN, needs a way to choose the guide end to end. The previous commit made per-genotype latents safe for autoguides under mini-batching, so they can now be exposed. What: - run_inference: AUTOGUIDES/GUIDE_TYPES registry (delta, auto_normal, auto_diagonal_normal, auto_multivariate_normal, auto_low_rank_multivariate_normal), resolve_guide_type (accepts numpyro class names), check_guide_kwargs. setup_svi takes guide_type, guide_kwargs and init_values (init_to_value). - Checkpoints record guide_type/guide_kwargs; restore_svi_from_checkpoint rebuilds that guide (legacy checkpoints -> component). get_posteriors drops autoguide auxiliary sites. - tfs-fit-model: --guide_type, --guide_rank, --guide_init_scale for analysis_method=svi; an autoguide starts at the pre-MAP point (or the guesses); options are validated before fitting; resuming with a different guide is refused. - tfs-sample-posterior routes by recorded guide_type (delta -> Laplace, others -> guide sampling), since AutoNormal parameters share AutoDelta's _auto_loc names. - The batch-safety guard's messages use guide_type names. - test_model_smoke: fit the beta-noise config with the component guide; the batch-safety guard (previous commit) refuses it under AutoDelta. - CLAUDE.md and CHANGELOG updated. Effects: default behavior is unchanged (component guide for SVI, AutoDelta for MAP/pre-MAP). Checkpoints gain two keys; old checkpoints still load. Verification: new unit tests for the registry, kwargs, checkpoint metadata, restore, fit-model and sample-posterior routing; autoguide round-trip tests and configure -> fit -> sample smoke tests. pytest tests/tfscreen: 4560 passed, 10 skipped; tfmodel/inference + generative + scripts + smoke-tests --runslow: 2432 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several tests called run_optimization, _run_nuts and
run_prefit_calibration with their default out_prefix ("tfs",
"tfs_prefit") or a bare "myroot", leaving checkpoint, loss and params
files (plus a checkpoints/ dir) in whatever directory pytest was
launched from.
- test_run_inference.py: autouse fixture chdirs each test into tmp_path
- test_fit_model_helpers.py: mock_ri_for_nuts chdirs into tmp_path;
the explicit "myroot" prefix now lives under tmp_path
- test_prefit_calibration_smoke.py: pass out_prefix under tmpdir to
both run_prefit_calibration calls
Library defaults are unchanged. Verified with a full
tests/tfscreen + tests/smoke-tests --runslow run (4506 passed): a
per-test cwd watcher saw no new files and git status stays clean.
Directory keeps track of future work/scientific questions related to the analysis pipeline and its development. Currently populated with our project to rethink and re-implement how congressional transformation.
Checking each step of the congression plan needs calibration pooled across a simulation grid. This brings tfs-summarize-calibration over from the guide-selection branch (code only, from e925b90; no grids or results). tfs-summarize-calibration (analysis/calibration_grid.py) reads each run's combo.json and tfs-summarize-fit outputs and reports coverage, calibration error/bias, PIT KS, interval width, RMSE and Pearson r per run x quantity x stratum (has-binding x spiked/bulk, theta regime). It averages arms over --replicate_keys and pairs each run with a --baseline run fit to the same simulated data. The docstring no longer calls spiked-origin genotypes congression-free, since wt and the spiked singles are also encoded in the bulk sub-libraries. tfs-summarize-fit picked its config and losses with *_config.yaml and *_losses.txt, which in grid run directories also match the simulate config and the prefit/pre-MAP losses; the right file won only by alphabetical order. It now picks the config by content (data + components sections) and ignores prefit/premap losses. New console script: tfs-summarize-calibration (reinstall to register). CHANGELOG and CLAUDE.md updated. Verified: full tests/tfscreen suite passes (4583 passed, 10 skipped). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The simulator gave every plasmid in a cell that cell's whole abundance, in both _sim_sequencing and _calc_genotype_cfu0, so a cell carrying n plasmids counted as n cells in reads and ln_cfu_0. Bulk genotypes (transformed with transformation_poisson_lambda) were inflated relative to the monoclonal spikes by lambda/(1 - exp(-lambda)) (1.19 at lambda = 0.357), which made a genotype's simulated spike/bulk make-up depend on lambda. The design calculation (expected_library_composition, bulk_fraction) assumes the opposite. A new helper, _plasmid_shares, gives each valid plasmid slot 1/n of its cell; both functions now weight by it, so each cell counts once. Growth (_sim_growth) is unchanged; combining co-resident plasmids' theta and dk is step 2 of future/congression-physics-plan.md. Effects: simulated read counts and ln_cfu_0 change for any config with lambda > 0. Verified: new tests for the shares, for sequencing of mixed and duplicate-genotype cells, and for simulated abundance matching the design pool_fraction at lambda = 0, 0.357 and 1.5 (these fail on the old code for lambda > 0). Full tests/tfscreen suite passes (4590 passed, 10 skipped). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The simulator combined a multi-plasmid cell's whole k*t across its
plasmids (multi_plasmid_combine_fcn: min/max/mean/...). dk_geno varies
more across genotypes than the whole theta signal, so dk_geno, not
theta, decided which plasmid set the cell's growth.
A co-transformed cell now gets cell-level theta and TF activity from
congression_theta_rule ('max': the highest-theta plasmid at each
condition sets both) and dk_geno from congression_dk_rule ('dilution':
share-weighted mean), then k_pre/k_sel/k*t from the same growth model,
theta rescale, growth transition and tube noise as any genotype
(selection_experiment._cell_kt). Rules live in simulate/cell_rules.py.
thermo_to_growth.growth_rates/growth_rate_one_condition are now the
one theta/activity/dk_geno -> k formula for genotypes and cells,
replacing _apply_growth_params. With lambda > 0, _check_cell_components
refuses phenotypes whose k_pre/k_sel the growth block does not rebuild.
Breaking: multi_plasmid_combine_fcn is removed and refused with a
message naming the new keys (both optional). Example configs, docs,
CLAUDE.md and CHANGELOG updated. Step 2 of
future/congression-physics-plan.md.
Verified: new unit tests for the rules, _cell_kt (incl. growth
transition, NaN theta, duplicate-genotype cells) and the checks; full
tests/tfscreen suite passes (4599 passed, 10 skipped). tfs-simulate on
examples/simulate at lambda = 0 gives byte-identical outputs to the
previous commit; at lambda = 1.5 runtime (22 s vs 23 s) and memory
are unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
One flag (congression_mask, with ln_cfu0_spiked_mask as its inverse) set both a genotype's congression purity and its ln_cfu0 prior class. Spiked-origin genotypes are mostly bulk cells (wt and the spiked singles are also encoded in the bulk sub-libraries), so the two need to be separate before the fit can mix clean and congressed cells. GrowthData gains bulk_fraction, a library-sized float per genotype (indexed by batch_idx). ModelOrchestrator._build_bulk_fraction takes it from the library composition table's bulk_fraction; __unknown__ counts as bulk (1.0, as the congression mask treats it), and any other genotype missing from the table, or a value outside [0, 1], raises. The legacy spiked_genotypes list gives 0 for spiked, 1 otherwise. ln_cfu0_spiked_mask still comes from in_spiked_origin, and congression_mask still drives today's correction. Not yet read by the model, so fits are unchanged. future/congression-physics-plan.md records the step 3 fit design and the step 3.0 study that shaped it. On simulator ground truth (real library design, lambda = 0.357, dk_geno spread scaled from the sim prior down to the measured ~0.001/min), the exact reads-level mixture was compared with candidate estimators: - today's fit rule (E[max] theta, own dk) is no better than no correction once dk varies; averaging theta cannot represent a mixture of growth rates - averaging the rate first works only for a narrow dk spread; the second-order cumulant fails on heavy dk tails - Monte Carlo with fixed co-resident sets is unbiased at every spread, with RMS falling as 1/sqrt(K); a pooled background is adequate - at the measured spread the whole effect is modest (0.05 RMS ln_cfu) The design: fixed per-genotype co-resident sets stratified by N, one loss per step, lambda entering through the weights, logit_norm retired. Also records the bench dk_geno measurement (eight genotypes, 0.024 +/- 0.001). Steps 3.0 and 3.1 of future/congression-physics-plan.md. Verified: new TestBulkFraction tests; full tests/tfscreen suite passes (4607 passed, 10 skipped); model and prediction smoke tests pass (25). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
future/ was named for deferred ideas, but it now holds the active congression plan and is starting to hold the numerical work behind its decisions. planning/ covers ideas, active plans and the studies that inform them, kept together so they can link to each other. - git mv future planning; references updated in the plans, CLAUDE.md and two code comments (cell_rules.py, data_class.py). - planning/README.md describes ideas, plans and studies, with study conventions: one studies/<slug>/ directory with a README (question, decision fed, how to run, inputs, commit, results) plus the script; plan and study link both ways; outputs not committed; a study is frozen once a plan cites it; dev/ stays untracked scratch. - The step 3.0 estimator study moves from dev/ to planning/studies/congression-estimator/ (script defaults to the example simulate config) with a README holding its full results. Rerun on the current commit reproduces the recorded numbers exactly. - CLAUDE.md gains a short Planning section. No code behavior changes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The full-batch training index is binding-first and reshuffled every step, so at full batch it is a permutation of the library, not the identity. Three components returned library-ordered values without slicing them through batch_idx: - activity/hierarchical_mut and activity/horseshoe_mut sliced only when batch_size < num_genotype, so at full batch every bulk genotype got another genotype's activity. Fits using them without mini-batching should be rerun. - theta/_simple indexed its per-genotype theta by the batch-relative geno_theta_idx alone. The pre-fit uses it on a binding-only subset whose index is library order and never reshuffled, so pre-fit results are unaffected, but any other use would mis-order. All three now slice through batch_idx (model and guide). The shape-based batch-safety check could not see this class of bug, since every shape is right. New find_batch_order_mismatches / find_orchestrator_batch_order_mismatches (inference/batch_safety.py) trace two orderings of the full batch with the latents held fixed and check that growth_pred, theta_growth_pred, binding_pred and theta_binding_pred permute with the index. test_batch_safety.py runs it across the component registry, epistasis and binding-only variants; only noise/beta, the documented exception, is flagged. CLAUDE.md's component rules now require unconditional slicing through batch_idx; CHANGELOG updated. Verified: the new order tests fail on the old code for exactly the three components (5 failures) and pass with the fix; full tests/tfscreen suite passes (4657 passed, 10 skipped); prefit-calibration and model smoke tests pass (28). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
reports/ and docs/badges/ are regenerated by every run_all_tests.sh run and cluttered diffs (reports/flake.txt alone was 7 MB). Untrack and gitignore them; the script now writes everything, badges included, under reports/ and stops on the first failing step. README.rst gets the live GitHub Actions status badge for tests.yml instead of a committed SVG. Add .flake8 excluding .claude/ (agent worktrees), build and venv dirs: ~85% of flake.txt was lint of .claude/worktrees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZqUx3DNWbnCRdrZK75ifz
Stop committing test reports and badges
The congression mixture (step 3.3 of planning/congression-physics- plan.md) builds each congressed cell from its co-resident plasmids, looked up by library index. It needs every genotype's dk_geno and, because a cell's activity follows the plasmid that sets its theta, every genotype's activity -- not just the batch's. Every dk_geno (fixed, pinned, hierarchical_geno) and activity (fixed, hierarchical_geno, horseshoe_geno, hierarchical_mut, horseshoe_mut) component's define_model and guide take return_population=False. When True they return (tensor, population): every genotype's value in library order, from the same draw (no new sample sites), or None when the latents arrived as a batch-sized substitution (posterior forward pass). The shared helper generative/components/_population.py:: per_genotype runs exactly the old slice-then-compute path when the population is not requested. GrowthData gains external_dk_population / external_activity_population as the prediction fallback, mirroring external_theta_population. Nothing requests the population yet, so fits are unchanged. CLAUDE.md adds the rule for new dk_geno/activity components; CHANGELOG and the plan updated (3.2 done; categorical_geno's batch-ordered population theta noted for 3.3). Verified: new test_component_population.py (37 tests; every registered dk_geno/activity variant, model and guide, full and mini batch: population[batch_idx] equals the batch values, default call unchanged, wt pinned; batch-sized substitution gives None); full tests/tfscreen suite passes (4694 passed, 10 skipped); model and prediction smoke tests pass (25). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
model.py builds the congression background by calling the theta component's run_model with batch_idx = geno_theta_idx = arange(num_genotype). categorical_geno sliced its per-genotype latent to the batch when sampling, so that call returned batch-ordered theta at full batch and indexed past the batch under mini-batching (repeating the last genotype). The empirical correction's CDF is order-free, so full-batch fits were unaffected; fits combining categorical_geno, transformation=empirical and mini-batching used a corrupted background and should be rerun. theta now stays library-sized and run_model indexes it through batch_idx[geno_theta_idx] (the hill_geno pattern). The theta_theta deterministic site is now stored for every genotype. The congression mixture (step 3.3c) looks co-resident genotypes up in the population theta by library index, so it must be library-ordered for every theta component. New test_population_theta.py checks this for each theta component (model and guide; shuffled full batch and mini-batch). test_categorical_geno.py's fixtures used an inconsistent geno_theta_idx and constant values that hid indexing; they now use batch-relative indices and distinct values. Verified: the new test fails on the old code for categorical_geno only (4 cases); full tests/tfscreen suite passes (4726 passed, 10 skipped); model, posterior and prediction smoke tests pass (27). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The congression mixture (step 3.3c) approximates each congressed cell's expected growth by a fixed quadrature over co-resident plasmid sets, drawn once per genotype and never redrawn (see the plan's "Fit design (step 3)" and planning/studies/congression-estimator/). ModelOrchestrator._draw_coresident_sets fills GrowthData.coresident_idx ((num_genotype, K, N_max) library indices, -1 = empty slot) and coresident_n (co-residents per set), stratified by co-resident count. Co-residents are independent draws from _coresident_pool: the bulk share of the pool, pool_fraction * bulk_fraction, over genotypes with growth data (library genotypes with no data, mostly dropouts, are left out; __unknown__ is never drawn). The legacy spiked_genotypes path uses a uniform pool over non-spiked genotypes. New settings congression_sets (default [12, 3, 1]) and congression_seed (default 0) are validated, written to the config's components block and read back, so a refit reproduces the same draws. Not yet used by the model, so fits are unchanged. CHANGELOG, CLAUDE.md and the plan updated. Verified: new TestCoresidentSets tests (shapes and strata, pool equals the bulk share, draw frequencies follow it, __unknown__ never drawn, legacy uniform pool, empty pool, seed reproducibility, validation, library-sized under batching, configure/read_configuration round trip); full tests/tfscreen suite passes; configure, model and prefit-calibration smoke tests pass (33). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
get_posteriors, get_laplace_posteriors and get_map_posteriors run the model over genotype chunks of forward_batch_size and sliced every genotype-indexed latent to the chunk first. Components that index library-sized parameters by library position (hill_geno, thermo.*, and categorical_geno) then read past the sliced arrays for every chunk after the first, and JAX clamped the index instead of raising. For any library larger than one chunk (512 by default), the deterministic sites saved for genotypes past the first chunk -- theta_growth_pred, growth_pred, and everything tfs-predict-theta/-growth derive from them -- were wrong. The forward pass now receives library-sized latents whole, as in training (components slice them with batch_idx), and slices only batch-positional latents (noise/beta) and the saved output (RunInference._batch_positional_latents, from the batch-safety check). Posterior files for libraries larger than forward_batch_size should be regenerated with tfs-sample-posterior. Verified: new test_forward_chunking.py compares posterior and MAP posterior files at forward_batch_size 512 and 2; it fails on the old code (hill_geno, categorical_geno, MAP) and passes with the fix, on HEAD alone (tests/tfscreen/tfmodel/inference, 235 passed incl. slow). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The empirical transformation replaced each bulk genotype's theta by the expected maximum over co-resident plasmids and grew one trajectory with the genotype's own dk_geno. Averaging theta cannot represent a mixture of growth rates; in simulation the correction removed almost none of the congression error once dk_geno varies (planning/studies/congression-estimator/). New transformation 'mixture': each genotype's cells split into a clean class and K congressed classes built from the fixed co-resident sets (step 3.3b). A congressed cell takes theta and activity from its highest-theta plasmid per titrant concentration and the mean dk_geno of its plasmids (dilution), matching the simulator. Classes ride a leading axis through theta_rescale -> calculate_growth -> growth_transition, and mix as ln_cfu = ln_cfu0 + logsumexp(log w + G), w_cong = bulk_fraction (1 - exp(-lambda)) split over sets by the Poisson probability of their co-resident count (log space; lambda gradient finite for spike-only genotypes). Theta noise acts on the genotype's own theta; theta_growth_pred is unchanged. Breaking: - 'empirical' and 'logit_norm' are removed and refused by name with a message; configs must switch to 'mixture' (fits not comparable). _congression.update_thetas stays for Stage 1.5 only. - Transformation interface: define_model/guide sample lambda; cell_classes(focal, population, params, data); NEEDS_POPULATION. - GrowthData.congression_mask removed (purity is bulk_fraction); the categorical_geno/empirical check is gone (fixed in 3.3a). - A mixture model with bulk genotypes and an empty co-resident pool is refused. - predict() passes library-wide references (dk_geno/activity: posterior medians; theta: the theta component at posterior-median parameters on the prediction's concentration grid) and the fit's co-resident sets to the subset model; a raw MAP checkpoint with a mixture model now raises (run tfs-sample-posterior). - Growth tensors grow by 1 + K (K = 16 default); use mini-batching. Verified: new test_mixture.py (priors; weights sum to 1 and match Poisson strata; finite lambda gradient; cell_classes vs an independent loop implementation) and test_mixture_model.py (every growth_transition and condition_growth broadcasts over classes; lambda -> 0 and f = 0 reduce to single; theta_growth_pred unchanged; posteriors independent of chunk size); prediction tests (subset = full rows, grid-following theta, MAP refusal); full tests/tfscreen passes with --runslow (4768); all smoke tests pass (39). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Since step 1 the simulator splits a cell's abundance among the plasmids it carries, so a genotype's abundance by co-resident count n is the zero-truncated P(M = n + 1), not Poisson(n; lambda). The 3.3c mixture still used the Poisson weights (each plasmid getting the whole cell): at lambda = 0.357 it made a bulk genotype 70% clean instead of 83%, and would have recovered lambda ~ 0.18 against a simulated 0.357. - mixture: w_cong = f (1 - P(M = 1)), P(M = 1) = lambda e^-lambda / (1 - e^-lambda); congressed sets weighted by P(M = n + 1). A series branch keeps small lambda (and its gradient) accurate. The fit's lam is now the simulator's transformation_poisson_lambda; the zero class (cells that never grow under selection) plays no part. Docs define lambda and the conversion from a measured mean plasmids/transformant. - Retire Stage 1.5 of the empirical pipeline: --congression_lambda is removed from tfs-fit-genotypes and tfs-build-empirical (plus --library_config and --save_theta_history on tfs-fit-genotypes) with their outputs. It corrected theta alone with the E[max] operator the fit dropped in 3.3c, used Poisson weights and a spiked/bulk split, and ignored dk dilution. genotype_fit/congression.py, simulate/empirical/congression.py and transformation/_congression.py are deleted; a mixture-refit replacement is filed as planning/empirical-mixture-refit.md. - tfs-summarize-calibration: the spiked/bulk "origin" stratum becomes "purity" (spike / mixed / bulk, i.e. bulk_fraction 0, between, 1). Verified: new tests check the weights against simulated cell shares (selection_experiment._plasmid_shares) and the zero-truncated strata, and that the small-lambda series agrees with the direct formula with finite gradients; the retired arguments raise; purity strata cover spike, mixed and bulk. Full tests/tfscreen passes with --runslow (4744); all smoke tests pass (39). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
prediction.copy_orchestrator rebuilt the prediction model from orchestrator.settings, which carries both library_file and the spiked list derived from it; ModelOrchestrator refuses the pair, so tfs-predict-growth (and prior-predictive growth and the genotype trajectory plots) failed for every config written by tfs-configure-model --library_config. The derived list is now dropped when library_file is set, as write_configuration already does. Adds planning/studies/congression-calibration/ for step 3.5: 54 runs (simulated lambda 0 / 0.357 / 1.0 x wide / tight dk_geno x 3 seeds x fit single / mixture with matched lambda prior / mixture with the measured 0.357 prior), each simulate -> configure -> prefit -> SVI -> posterior -> predict -> summarize-fit on SLURM, paired against single in tfs-summarize-calibration. The simulator and fit share the max theta rule, so the grid checks the fit's machinery and is the baseline for step 4. Verified: new test_library_file.py test fails without the fix and passes with it; prediction tests and the prediction smoke test pass (262). One mixture run (lambda 1.0, wide dk) ran end to end locally with shortened fits, and tfs-summarize-calibration read it with purity x has_binding strata and the lam ground truth. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wt:D88A and wt:K84L each grew between their parents in 4CP at intermediate IPTG, close to the tighter binder: not consistent with weaker-binder dominance, and not able to separate max from a partition-function rule. Records the limits (no lambda, pheS only) and that single transformants taking over is partly expected under any rule, with ways to firm it up. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The calibration grid (54 runs, all converged) does not yet answer its question: the k/dk_geno slide is unanchored in this pipeline (no wt pin, no base_growth, prefit k_scale at its ceiling), congression is below the design's noise floor (single's errors do not change with lambda), and lambda is pulled up whatever the truth. Diagnosis (runs 0001/0002, same data): the mixture's forward model is correct (lambda -> 0 reproduces single to 1e-6), but the fit uses its congressed classes, through dk dilution, to fit the read-count detection floor: with pseudocount 1, rows with 0 reads sit +1.26 ln above the truth and 1-2 reads +0.62, so dying genotypes' trajectories bend upward. Real data share this path. Adds grid_masked.yaml (6 runs, rows with < 5 reads dropped via a new optional min_counts in run.srun) to test the diagnosis, the diagnosis scripts, and a study .gitignore for the grid outputs. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-ups from the 3.5 detection-floor diagnosis. The per-sample level offset absorbs noise shared by every genotype in a tube (tube growth noise, plating, PCR), which the model now treats as independent per-row error. The count likelihood fits reads with a negative binomial and a per-sample offset centered on plating: exact low-count information, no binomial approximation or pseudocount, and misassignment (index hopping, calling efficiency) modeled as added reads. Censoring comes first and waits on the masked grid results. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
tfs-setup-sim-grid resolved only the top-level thermo_data and calibration_file keys. Nested file references -- a params file in binding_data.spiked_binding.choose_by or binding_data.library_binding.choose_by, and empirical.phenotype_model -- were copied verbatim into each run's tfs_sim_config.yaml and then resolved against the run directory, so tfs-simulate could not find them unless the run template copied the file in. Setting up run directories is the grid tool's job, not the run script's. _SIM_PATH_KEYS is now a list of key paths, nested ones included. _resolve_paths follows them through dicts on a deep copy and resolves relative values against the base config's directory (base config) or the grid YAML's directory (simulate overrides). The existing relativize step then writes them relative to each run subdir. The choose_by keywords stratified/random are left alone. The congression-calibration study drops the cp of hill_params.csv from run.srun and the hill_params_file template block from grid.yaml and grid_masked.yaml. Its README and simulate_config.yaml comments now describe the new behavior. CLAUDE.md and CHANGELOG.md are updated. Verification: new tests cover nested choose_by and phenotype_model resolution, a keyword choose_by, a missing or non-dict parent, and a binding_data override resolved against the grid YAML. The simulate and util suites pass (878). tfs-setup-sim-grid on the study's grid.yaml wrote a choose_by path that resolves from each run directory, and tfs-simulate run from one run directory read hill_params.csv (the simulated parameters match the file). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Resolve nested file paths in tfs-setup-sim-grid
Dropping growth rows with < 5 reads changed almost nothing (lambda at truth 0 still 0.53 under a 0.357 prior; slide and theta RMSE unchanged), so the detection floor was not what drives the mixture. The joint log density of the lambda-0 fits differs almost entirely in the binding term: the default binding_weight (growth rows / binding rows, ~251) with simulated binding SD 0.001 dominates the objective, and the mixture's congressed classes let bulk genotypes hand their growth away through dk dilution so theta and shared hill_mut effects can follow binding. Adds grid_bw1.yaml (6 runs, binding_weight 1) through a new optional binding_weight in run.srun, sets the SLURM limit to 2 h, adds the diagnosis scripts (compare_masked, diag_basin, diag_sites; diag_lam0 takes GRID), and records the masked results and revised diagnosis in the study README and the plan. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 4. With the true (plain) model on the null simulation, pinning k and m at the truth costs about 745 nats (784,031 against 783,286), so stage 1 prefers the steep m; at the truth theta is far better (singles RMSE 0.020 against 0.056, coverage 0.70 against 0.10). attribute.py (new: growth rows, binding genotypes and prior sites at two MAPs) puts the loss in pheS+4CP (-396), in rows whose true theta is below 0.1 (-366), deep singles at high IPTG, growing with time: slow growers whose counts their co-transformed partners dominate. Plus -161 from the theta population priors, +37 from binding. Fitted lambda 0.335 free, 0.301 pinned, truth 0.357. Run 5 set up: make_sim.py --arm nocong (null without congression), stage1.py --no_mixture and stage1.srun NOMIX=1, free and pinned, to test whether the mixture's approximation causes the bias. Verified: flake8; bash -n; the nocong config and a --no_mixture fit on a subsampled simulation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 5: the null simulation without congression, fitted without the mixture. m is still 14% steep (22-25% with congression) and the truth still costs 383 nats (745 with). At the truth theta is nearly exact (singles RMSE 0.010, coverage 0.91). The attribution puts the remainder in the theta population priors (about -130 of -144 prior nats: the logit low, logit delta and log K scales and offsets) and in growth spread evenly (-235): the MAP over-shrinks the singles' curves and a steeper m restores the growth swing. The other half of run 4's bias is the congression mixture's approximation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hill_geno gains theta_logit_low_hyper_scale_fixed, theta_logit_delta_hyper_scale_fixed and theta_log_hill_K_hyper_scale_fixed next to the existing log(n) hold: > 0 makes that population SD a deterministic site at the held value with no guide parameters, as in hill_relative. One helper (_hyper_scale in the model, a per-SD table in the guide) replaces the log(n) special case; guide parameter names are unchanged, so old checkpoints load. Why: on the joint-fit simulation's stage 1 without congression (run 5), m is 14% steep with the true model. Checking before run 6 corrected run 5's reading: the MAP's population SDs are large, not small (logit theta_low 6.8 against the truth's 2.6), and what compresses the singles' theta range (0.88 to 0.085 against 0.99 to 0.01) is the population location, which sits at the default hyperprior's 2.0; the SDs' tight HalfNormal priors (log K's at 0.1) add their cost at the truth. Run 6 tests held SDs, moved location priors and both on the no-congression simulation; stage1.py --set_priors and stage1.srun PRIORS/TAG pass the items. Verified: new tests (each SD held alone: deterministic site at the held value, no guide parameters, others learned, sample sites matching, the held SD scaling the offsets); the hill_geno and shift tests; a fit with every SD held on a subsampled simulation (no theta SD in the MAP params, the extra items replacing the study's own); the full unit suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 6. On the no-congression simulation, moving the theta population's location priors to the truth's medians changes nothing, and holding its SDs at the truth's spread makes m steeper (30%) and theta worse. Regression on the truth shows what the free fit does: the singles' theta is 0.048 + 0.90 theta_true, m stretched by about 1/0.90, dk_geno right. For singles without binding that is close to a symmetry of the growth model (k absorbs the constant, m the slope), resisted only by wt and the eight spike binding curves, and the logit-scale population prior pushes along it by charging for saturated plateaus; smaller held SDs push harder, the location barely matters at SDs near 7. Remedies listed in the README: weaker pooling, more binding genotypes, or a parameterization whose prior does not prefer mid-range theta. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 7, the two cheapest remedies for run 6's squeeze of the singles' theta: hold the theta population SDs large (10, 10, 2), and measure more singles' binding (2, as in the real data's in-library pair, and 20), each alone and the large SDs with 20. stage1.py --add_binding_singles N appends N random singles with growth data, measured on the lab's 2 uM grid from their true Hill curves with the spikes' noise, to the simulated binding table (OUT_DIR/binding.csv), so the growth data stay run 5's; stage1.srun takes ADDB. analyze.py scores the added singles as their own group. Verified: flake8; bash -n; a subsampled fit with 5 added singles (9 spikes plus 5 singles in the binding table, the config pointing at it, analyze.py grouping them); the launch loop's expansion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 7, on the no-congression simulation. Twenty extra binding singles cut stage 1's m bias from 14% to 6% and the singles' theta RMSE from 0.046 to 0.021 (slope on the truth 0.90 to 0.96); two do about what weak pooling (SDs held at 10, 10, 2) does, 14% to 11%; weak pooling with twenty, 5%. Each binding curve pins one genotype's theta on an absolute scale against run 6's squeeze. Theta coverage is still 0.36-0.38 (0.91 at the true k and m). README and planning/joint-fit-return.md step 5 updated, including the experimental implication: binding on about twenty singles spanning theta. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 8, config only: 20 binding singles on the congression null simulation (with and without weak pooling), 50 on the no-congression one, and the whole recipe on the joint truth (plain with 20 binding singles, then the full model started from it, chained by a dependency). The README records why there is no separate interval study: at the true k and m the singles' intervals cover 0.91, so the undercoverage is the shared parameters' bias. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 8. Twenty binding singles halve the congression null simulation's m bias (21% to 12%; weak pooling adds nothing), leaving about twice the no-congression bias (6%), so the congression mixture's approximation is now the largest piece. Fifty binding singles take the no-congression bias only from 6% to 5%. The full recipe on the joint truth with twenty binding singles gets kanR's m within 1%, pheS's within 7% (15% without), log s covered and tight, and the singles' theta RMSE from 0.070 to 0.048; pheS's graded warp is the least identified piece. README and planning/joint-fit-return.md step 5 updated; next is the mixture's co-resident draws. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The congression mixture's fixed co-resident draws (congression_sets, entry n-1 = number of sets with n co-residents, default 12 3 1) were an orchestrator setting no CLI could change. tfs-configure-model now takes --congression_sets (ints, written to the config's components) and refuses it without transformation_model mixture; unset, the orchestrator keeps its default. Why: in the joint-fit simulation study the mixture's approximation is the largest remaining source of stage 1's m bias once binding singles are added (run 8: 12% with congression against 6% without). Run 9 fits the congression null simulation with 75, 140 and 430 sets (stage1.py --congression_sets, stage1.srun CSETS=). Verified: new configure tests (forwarded as a list of ints, absent when unset, refused with the single transformation); the CLI reference test (docs/source/cli.rst regenerated); a CLI round trip on the smoke data (config components hold [30, 10, 2] and the reloaded model draws 42 sets of 1 to 3 co-residents); stage1.py with 100 30 10 on a subsampled simulation (2.1x the default's wall time); the full unit suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim, run 9, the congression null simulation with twenty binding singles: 75 co-resident sets instead of the default 16 take stage 1's m bias from 13% / 11% to 10% / 9%, and 430 sets only to 9% / 8%, against 6% without congression. The number of draws is about a third of the congression excess; the rest is in the mixture's form. All three fits ran the GPU out of memory in the Laplace Hessian after the MAP, so m comes from each MAP. README and planning/joint-fit-return.md step 5 updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-sim. The README gains a summary of theta for growth-only singles across the fits: RMSE 0.02-0.05 with twenty binding singles, mostly a shared affine squeeze (slope 0.92-0.96), and intervals 2-6 times too narrow (about 1.8 times even at the true k and m with congression). Run 10 maps theta against the number of binding singles: 2, 5, 10 and 20 on the no-congression simulation with three random choices each, and 5 and 10 on the congression one. analyze.py reports each group's median 95% width, SD of z and slope on the truth; stage1.srun takes BSEED. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…les) planning/studies/joint-fit-sim, run 10. On the no-congression simulation, the growth-only singles' theta RMSE is 0.010 + 0.053 / sqrt(n + 2) for n = 0 to 50 binding singles (the floor is the fit at the true k and m), with three random choices per count agreeing within 0.003: the binding curves average down the shared scale error, so most of the gain comes by 20 and halving what is left there would take about 90. Coverage climbs only to 0.51 at 50 (SD of z 2.2); with congression everything is worse (0.037 and 0.31 at 20). binding_scaling.py (new) scores the fits and draws RMSE, coverage and SD of z against n; README updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/joint-fit-real: stage1_real.py runs the recipe from the joint-fit simulation study on the real screen (wt, spikes and singles, the probably swapped pheS+4CP replicate 2, t_sel 82 row dropped, binding at 2 uM through tfs-process-binding, hill_geno, the homodimer mixture with 50 20 5 co-resident sets and lambda 0.38 +- 0.15): the plain model, then the full one (warps, s, binding offsets) started from it, with the Laplace Hessian in chunks of 8, theta predicted at the growth and the binding concentrations. stage1_real.srun defaults to the cluster inputs; report.py tabulates k, m (against the within-tube values), the warps, s, tau and each binding genotype's offset and residuals. The README gives the expectation from the simulations (m 17-21% steep with this few binding anchors, theta intervals provisional). The lab's binding data stay in the gitignored planning/dev-data. Verified: flake8; bash -n; stage1_real.py and report.py end to end on a subsampled simulation (75 sets, Laplace at chunk 8, both theta predictions). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/findings-ledger.md (new, active) collects what the planning files, studies and git log establish, refute and leave open about the growth/binding model, lambda, the k/dk_geno slide and m, with sources. It comes from a review prompted by the joint fit's real stage 1: a week of joint-fit work had re-found July's results (in-library binding anchors pin m; lambda runs away under a model conflict) and early October's (wt's 10x IPTG midpoint offset against 2 uM binding, its in vivo floor), and the 2 uM agreement that motivated rejoining was in a gauge that absorbs any affine difference between binding and growth. planning/studies/joint-fit-real/README.md records run 1 (every binding offset negative, wt's midpoint 10x off, lambda 0.82-2.10, m and the warps not interpretable). planning/joint-fit-return.md gains a review section: steps 6-8 paused until the binding-to-growth link is decided; the simulation results stand under the model's own assumptions. CLAUDE.md's planning section points at the ledger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/studies/x-vs-binding/hill_compare.py (new) fits a Hill curve to each genotype's two-stage growth X (kanR alone and both libraries) and to its binding at 2 and 20 uM. Among genotypes with curves of real amplitude (wt, M42I, M42I/H74A, D88A, V95N), M42I's K is 0.18x wt's in growth against 3.6x and 5.3x in binding, a factor of 20-30 (the 2026-10-01 "no map" had about 20 on compressed X); M42I/H74A keeps the order but sits 10x lower in growth; D88A and V95N agree within 2x. wt's growth midpoint equals its 20 uM binding midpoint. Growth curves stay broader (n 0.7-0.95 against 1.9-3.2 at 20 uM). Both facts behind the "no map" survive the compression fix; planning/findings-ledger.md updated (Open item 2 closed, the repressor-level reading recorded as a hypothesis). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planning/release-0.5.0.md (active): the release is the two-stage relative fit (tfs-fit-two-stage) validated, documented and tagged, so the lab can take a screen from counts to X with reasonable intervals. It records where things stand on 2026-10-19 (the k shift fails as built; deep X coverage 0.90 / 0.76-0.81 without it; the kanR warp is not carried through the command; wt monoculture rates are a check, not an input), the user's decisions (repair the k shift, else ship without it; no warp in 0.5.0; drop pheS+4CP replicate 2, t_sel 82; the measured lambda prior; a stated depth threshold and flag policy; kanR's m from the tube totals, with the within-tube value stated as a limitation), and the code, validation, docs and release items. The paper analysis, thermodynamic models and experimental design are out of scope. relative-m run 7 (row_order.py): the suspect row's IPTG tubes look reversed with the 0 tube in place (labeled 0.1 uM reads as 1 mM, 1 uM as 30 uM; rank 10 of 720 assignments against 188 as labeled), but the 10 uM slot does not fit, the low tubes cannot be ordered, and the lab rules out that flip, so the row is dropped rather than relabeled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 1 of the two-stage validation found the k-shift refits measuring the optimizer, not k: they restarted at Adam's default step size (1e-3), fell about 6.7e5 nats off stage 2's MAP in the first window and re-descended along the dk_geno population against wt, so on the real data X moved +0.32 even with k unchanged. The kshift step of tfs-fit-two-stage now: - starts each refit at two_stage.kshift_step_size (default 1e-5, final 1e-6), overriding any two_stage.fit step size; - holds dk_geno's hyper_loc, hyper_scale and hyper_shift at stage 2's MAP in every refit (two_stage.dk_geno_hypers, pin_dk_geno_hypers: growth.dk_geno.pinned.* rows in the refit's priors CSV). Stage 2's values rather than stage 1's: stage 1's singles give a scale of 0.53 against stage 2's 1.53, so holding them would change the validated stage 2; - runs a third refit with k unchanged (two_stage.kshift_control, on by default), predicts stage 2's X at its MAP (stage2/tfs_pred_theta_map.csv) and writes kshift/drift.csv (each refit's median, q10 and q90 change in X over non-wt rows, two_stage.refit_drift); it warns when the control moves more than a quarter of the median |shift|. Stage 1, the jackknife and stage 2 are unchanged. Existing kshift/ directories are reused as they are. The release checklist (planning/release-0.5.0.md) marks item 6 done, and planning/two-stage-fit.md step 4 records the decision. Run 2 of planning/studies/two-stage-validation/ reruns kshift and predict on seed 1, seed 2 and the real data, with pass criteria. A known risk is recorded there: the data preferred run 1's slid state by about 1.5e4 nats, and the per-genotype dk_geno offsets can still move together at a small prior cost, which the control refit will show. Also: CHANGELOG, CLAUDE.md, examples/tfmodel/two_stage.yaml, docs/source/cli.rst regenerated. Tests: the helpers (test_two_stage.py) and the step (test_fit_two_stage_cli.py: step sizes, pins, the control, drift.csv, the warning, the settings). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0.5.0 was never tagged or shared in the lab; the version bump came before the two-stage work. Item 20 now folds CHANGELOG's [Unreleased] into the existing [0.5.0] section (dated 2026-10-03) and redates it at release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release checklist items 7 and 8 (planning/release-0.5.0.md). tfs-fit-two-stage's read_two_stage_config now fails fast on what the two-stage route does not support: binding_df, a theta growth or binding noise model other than zero (genotype_offset among them), the titrant-shift, binding-offset and warp priors in set_priors, and a growth_priors table whose m_warp column turns the warp on. Most of these failed later inside stage 1 (hill_relative refuses binding and theta noise); the warp did not fail at all: stage 1 fitted it and stage 2, whose pins carry only k and m, dropped it. The genotype_offset module, hill_geno's titrant shift and linear's m_warp docstrings now say they are experimental. The real-data recipe (planning/studies/real-data-fit/fit/run_recipe.srun) runs tfs-correct-tubes before tfs-process-counts, with the corrections file in planning/dev-data (gitignored; CORRECTIONS, default ../../tube_corrections.yaml). That file excludes the pheS+4CP replicate 2, t_sel 82 row (decision 3); checked locally, 118 tubes to 112. Tests: the new refusals and the m_warp check in test_fit_two_stage_cli.py. CHANGELOG updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release checklist item 9 (planning/release-0.5.0.md): users should read
results from one table, not from the step directories.
A new last step, summary, writes {out_dir}/tfs_x.csv with one row per
genotype and concentration: X's quantile ladder with the shared k shift
(predict/), the shift itself (k_shift, from kshift/shift.csv), the
genotype's number of mutations, its reads in total and per library
(counted in chunks from the growth table into
{out_dir}/genotype_reads.csv once), in_training_data, theta_scale, and
the marker-consistency consistency_z and inconsistent (NaN for wt, for
genotypes the check did not cover, and when it did not run). It is
rewritten on every call, so it picks up a consistency step run after
it. Library functions: two_stage.genotype_reads, num_mutations,
x_summary.
On run 1's real outputs: 217,945 genotypes, 1.7 M rows, about 40 s to
count 23 M growth rows. A depth flag waits on decision 5's threshold.
The validation study's submit.sh queues the summary after kshift_predict
and consistency; its README gives the command for the three runs now in
the queue. CLAUDE.md, CHANGELOG, docs/source/cli.rst updated. Tests: the
helpers (test_two_stage.py), the step in a full run, before consistency
and without predict (test_fit_two_stage_cli.py).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI (pandas 3.0.6) failed the same 7 tests on every job, all in test_shared_shift.py with "ValueError: assignment destination is read-only"; locally (pandas 2.3) they passed. shared_shift.row_shifts set wt's shift to zero in the array from Series.to_numpy(), which pandas 3's copy-on-write returns as a read-only view. Any fresh install would have failed tfs-predict-theta and tfs-predict-epistasis with --shared_shift, and so the two-stage fit's predict step. It now takes a copy (to_numpy(copy=True)). tests/conftest.py turns pandas' copy-on-write on when pandas is older than 3, so the local suite behaves as CI's does. The full suite passes with it on (the two parallel-vs-serial tests too). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release checklist items 15-18 (planning/release-0.5.0.md): the lab needs one documented route from a real screen's counts to curves with intervals. docs/source/screen-analysis.rst covers what the analysis needs (counts, tube table, OD600 and calibration, library YAML, corrections, the measured lambda, loose growth priors), checking and correcting the tube table (tfs-check-tubes, tfs-correct-tubes, tfs-process-counts), the two-stage YAML with the validated settings and why, what each step does, how to run it on one machine or as a SLURM chain, its cost, the columns of tfs_x.csv, which genotypes to use, what X is and how it differs from binding, and what the intervals carry and leave out (stage 1's m bias, the pheS gap, kanR's absolute slope, congression's per-genotype error). Two numbers are marked [pending validation round 2]: the read-depth threshold and the deep genotypes' coverage. The index, quickstart and README point to the page. fitting.rst makes the two-stage fit the route for a screen, explains why the single relative MAP is not, and marks the joint growth/binding model experimental. examples/tfmodel/ gains growth_priors_loose.csv (two_stage.yaml named it, but it was not there) and a SLURM chain, two_stage_submit.sh with two_stage_step.srun (absolute paths; the summary step; account and partition to edit). CLAUDE.md's examples table and CHANGELOG updated. The docs build with -W and no warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release checklist item 19 (planning/release-0.5.0.md). docs/source/cli.rst was already current. CLAUDE.md: the relative-X section said which inference to use was open; it now names the two-stage fit as the route for a screen (the single full-library MAP compresses X about 1.45-fold) and keeps the single-fit comparison as history. The planning section leads with the release plan and its settled decisions, says the joint-fit plan is paused and relative-m an idea, and notes that the single-MAP recipe applies the dev-data tube corrections and has been replaced by the two-stage fit for X. docs/source/pipeline.rst points to the screen-analysis page, and replaces "tube cleanup belongs in the lab's own script" with tfs-check-tubes and tfs-correct-tubes. The docs build with -W. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Validation run 2 tested the k-shift repair on seed 1, seed 2 and the real data. The refits no longer slide (the real control moved X by -0.008, against +0.32 in run 1), but on the simulations the shift is a third (seed 1) and a tenth (seed 2) of the true one, plus and minus are lopsided, and the control still moves more than a quarter of the shift. Holding the dk_geno population blocks the slide and also part of X's real response to k. Under release decision 1 the fallback applies (user, 2026-10-20: 0.76-0.90 coverage is fine for now). tfs-fit-two-stage gains two_stage.kshift (default false): 'all' skips the jackknife (which only feeds the shift) and kshift, predict runs without --shared_shift, and the summary ignores any stale shift table, leaving k_shift blank. --step jackknife and --step kshift still run; kshift: true restores the shift. The fit now takes about half a day. Seed 1's coverage by depth without the shift (validate.py, on the cluster): 0.98 / 0.90 / 0.88 / 0.90 / 0.76 at <=1e2, 1e2-1e3, 1e3-1e4, 1e4-1e5 and >1e5 reads; slope 0.04 / 0.39 / 0.88 / 1.02 / 1.02. The usable-genotype threshold is 1,000 reads (decision 5). - docs/source/screen-analysis.rst: the steps without the shift, the cost, the k_shift column, the 1,000-read threshold with the depth table, and k's uncertainty listed among what the intervals leave out; no placeholders left. - examples/tfmodel/two_stage.yaml (kshift: false, documented) and the SLURM chain (stage1 -> stage2 -> predict and consistency -> summary; the jackknife array and kshift only with kshift: true). - The validation README records run 2's results and the decision; the release checklist (decisions 1 and 5, items 12, 15, 16), planning/two-stage-fit.md step 4, the findings ledger (open item 5), CLAUDE.md, CHANGELOG and docs/source/cli.rst follow. - Tests: the default run without jackknife, kshift or shift, and the stale-shift case; the shift tests now set kshift: true. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release checklist item 11 (planning/release-0.5.0.md). The pandas 3 failure reached CI and not the maintainer's laptop because nothing said which versions tfscreen works with, and a new install on a GPU cluster would have got CPU-only jax. - pyproject.toml: a lower bound on every dependency, at the oldest versions the suite has passed with (the maintainer's environment: pandas 2.3, numpy 2.3, jax/jaxlib 0.11.2, numpyro 0.22, flax 0.12.2, scipy 1.16, numba 0.63, ...). No upper bounds: CI keeps testing the newest releases. A dry-run install in that environment changes nothing but tfscreen. - A cuda12 extra (jax[cuda12]): pip's plain jax is CPU-only on Linux. - constraints/tfscreen-0.5.0.txt: the pip freeze of a fresh venv (macOS, Python 3.13.11, pip install -e ".[test]") whose full suite passed on the newest releases (6,127 passed; pandas 3.0.6, numpy 2.5.4, jax 0.12.0). A dry-run resolve of it with jax[cuda12] for Linux x86-64 gives jax-cuda12-plugin/pjrt 0.12.0 on Python 3.12; on 3.11 it fails (contourpy 1.4 dropped 3.11), so it is documented as Python 3.12 or newer. - planning/studies/fresh-install/: a SLURM job that clones from GitHub, installs with the extra and the constraints in a new venv, checks jax sees the GPU, runs the unit tests and the quickstart; README with the local results. - Install instructions in README.rst, quickstart.rst and screen-analysis.rst (which also now says about half a day of GPU time); CLAUDE.md, CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's Python 3.11 jobs failed at install ("No matching distribution found
for jax>=0.11.2"): jax 0.11 and newer, the new floor, ship no 3.11
wheels. requires-python is now >=3.12, and the test matrix is 3.12,
3.13 and 3.14 on Linux and macOS (the coverage job stays on 3.12). A
dry-run resolve of the fresh install's versions found wheels for every
package on Python 3.12 and 3.14, Linux x86-64 and macOS arm64.
No constraints file: the previous commit's docs pointed at
constraints/tfscreen-0.5.0.txt, but a top-level directory for it beside
src, docs and tests is clutter (user), and the file itself was never
committed. The README, quickstart, "Analyzing a screen" page,
pyproject.toml comment, CLAUDE.md, CHANGELOG and the fresh-install job
no longer mention it; the screen page asks for the environment's pip
freeze to be saved with the analysis, and release checklist item 23
records it for the manuscript.
The user's manual fresh install on the cluster (wipe, new environment)
worked (planning/studies/fresh-install/). Also: Python version
classifiers, CONTRIBUTING, the fresh-install job's version check,
checklist item 11 done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Natalie's dry run of "Analyzing a screen" (release checklist item 13): - "What you need" listed a corrections file as if it already existed, and the page then went straight into making one. The list now holds what the experiment and the fit need, and says the next section makes the growth table and, if a tube is mislabeled, the corrections file. - "Correct the tube table" is now "Make the growth table": what the growth table is (growth.csv, the YAML's growth_df), a skip note for readers who already made and checked it (as in process-raw), and four numbered steps: make a first table and check the tubes; if nothing is flagged, rename it to growth.csv; if tubes are flagged, write the corrections file; apply it and rerun tfs-process-counts on the corrected tube table. This replaces "Then apply it and make the growth table the fit reads", which did not say that the counts are reprocessed or what comes out. - process-raw.rst points to the screen page. Release checklist: items 10 (the screen page sets predict_epistasis), 13 (the dry run, with these fixes) and 14 done. planning/two-stage-fit.md: steps 4 and 8 done; the plan stays active for steps 5 and 7, the students' design-sweep tools, after the release. The docs build with -W. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release checklist item 20 (planning/release-0.5.0.md). 0.5.0 was never tagged, so the entries made since 2026-10-03 under [Unreleased] join it rather than a new version: [0.5.0] is dated 2026-10-24 and opens with a summary of the release (one command, tfs-fit-two-stage, from counts to X curves with intervals; coverage on simulations; Python 3.12+), then those entries, then "Changes up to 2026-10-03" with the earlier summary and entries unchanged. [Unreleased] is empty; the comparison links at the end already point at v0.5.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release PR (harmslab#18) failed CI's coverage job: every test job passed, but diff-cover --fail-under=100 found 114 of the 7,771 lines changed since harmslab main untested, across 22 files (mostly error branches and skip-if-done paths). Tests now cover all of them; the gate stays at 100%. Tests added (assertions on behavior, not just execution): - two-stage fit: refusals (non-linear growth, missing inputs per step, a bad consistency library, jackknife subsamples of different sizes), resuming a half-finished stage, k and m from the MAP when extraction skips pinned sites, skipped jackknife and kshift; two_stage helpers; predict_theta's warning without value columns; configuration_io's non-standard dataclasses. - prediction: binding-plate sites taken by binding label in a subset, plus two defensive checks (a binding genotype without a value; a posterior site the prediction model lacks) that real subsetting cannot reach, so their tests patch internals; summarize-fit's keyed truth and ddG_sigma failure paths; genotype_offset without ordinals. - thermodynamic loader and simulator: column_map and thermo_options edge cases, predict_unmeasured for C4 PddG U0 and PK U1 (untested until now), truth-table and theta-at-concentration edge cases, the run_simulation subsample branch, simulate_cli's offset and scale outputs. - process_raw: tube-correction and tube-QC edge cases and refusals, binding refusals, the CLI modules run as scripts; library_subsample's missing-origin error. One bug, found while covering thermo/io.py: load_ddG_prior_csv failed ("The truth value of a Series is ambiguous") whenever one file column supplied two structures, a column_map such as {HD: H} for a model or, with no model, a column kept as its own structure and copied to another. The numeric check now looks at each source column once. Two tests; CHANGELOG entry under 0.5.0 Fixed. Also: release checklist item 21 done (the release test run). Verified locally as CI runs it (fresh venv, newest releases): pytest -n 6 --runslow --cov tests/tfscreen, 6,216 passed; total coverage 96% (gate 90%); diff-cover against harmslab main, 7,773 lines, 0 missing, 100%. Fatal lint clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release for the first manuscript on the 2026 screen: someone in the lab
can take a screen from per-tube read counts to per-genotype X curves with
95% intervals through one command,
tfs-fit-two-stage, documented in"Analyzing a screen" (
docs/source/screen-analysis.rst). On simulations ofthat screen the deep genotypes' intervals cover the truth 0.88-0.91 of the
time up to 1e5 reads and 0.76-0.81 above. tfscreen now needs Python 3.12 or
newer.
See CHANGELOG for details