drop --spiked

CLAUDE.md

@@ -43,10 +43,10 @@ flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127
 tfs-process-fastq          # FASTQ → read counts
 tfs-process-counts         # counts → ln_cfu DataFrames
 tfs-process-presplit       # Process pre-split count files
-tfs-configure-model        # Generate YAML config template
+tfs-configure-model        # Generate YAML config template (+ library composition snapshot)
 tfs-prefit-calibration     # Pre-fit linking function via MAP
 tfs-fit-model              # Main hierarchical Bayesian inference
-tfs-fit-genotypes          # Per-genotype MLE fits of the growth model (+ optional congression de-attenuation)
+tfs-fit-genotypes          # Per-genotype MLE fits of the growth model (+ optional congression de-attenuation; spiked genotypes come from --library_config)
 tfs-sample-posterior       # Draw posterior samples from fitted model
 tfs-sample-prior           # Draw prior predictive samples
 tfs-extract-params         # Extract parameters from checkpoint
@@ -166,7 +166,21 @@ Net flow: `configure` (scalar) → `prefit` (per-condition `k_loc` indexed rows
 
 ### Configuration (YAML)
 
-See **YAML Standards** below for the full conventions. The tfmodel config (`tfs_configure_config.yaml`) is generated by `tfs-configure-model` and drives `tfs-fit-model`. It contains `data`, `components`, `priors_file`, and `guesses_file` sections. Do not hand-edit it.
+See **YAML Standards** below for the full conventions. The tfmodel config (`tfs_configure_config.yaml`) is generated by `tfs-configure-model` and drives `tfs-fit-model`. It contains `data`, `components`, `priors_file`, `guesses_file` and — for a growth model — `library_file` plus a `library` provenance block. Do not hand-edit it.
+
+### Library description (`--library_config`)
+
+`tfs-configure-model` takes `--library_config`, the library YAML that describes the screened library — **the same file handed to `tfs-process-fastq`**. It is required whenever `growth_df` is given (a binding-only model has no congression correction and no ln_cfu0 latents, so it refuses one). It replaced the old `--spiked` list, which could not express the central fact that the spiked controls are *also* encoded in the bulk sub-libraries: wt and each spiked single mutant arrive both as a monoclonal spike and as bulk library members, and they cannot be told apart by sequence.
+
+Keys read (via `LibraryManager`): `reading_frame`, `first_amplicon_residue`, `wt_seq`, `degen_sites`, `tiles`, `tile_combos`, `spiked_seqs`, plus `library_mixture`. Everything else (a full simulate config, say) is ignored.
+
+Flow: `genetics.library_composition_table(library_config)` (`genetics/library_design.py`) →  one row per genotype with `is_wt`, `in_spiked_origin`, `pool_fraction` and `bulk_fraction` → written to `{out_prefix}_library.csv`. The **snapshot CSV is what `tfs-fit-model` reads**, never the YAML; this mirrors the priors/guesses CSV pattern and makes the table the place to override the design's numbers. `ModelOrchestrator(library_file=...)` derives today's `spiked_genotypes` from `in_spiked_origin` and exposes the full table as `orchestrator.library_df`. `spiked_genotypes` is still accepted directly (and is what `tfs-prefit-calibration` uses when it clears both keys for its calibration-subset model), but the two are mutually exclusive, and a config carrying `spiked_genotypes` is the legacy path.
+
+**Two checks, two different guarantees.** The *genetics* keys are attestable: they determine the genotype names, so `tfs-configure-model` fails fast if any genotype in `growth_df`/`presplit_df`/`base_growth_df` is absent from the library (`check_genotypes_in_library`; `__unknown__` excepted). A residue-numbering shift mismatches essentially every mutant and is caught immediately. `library_mixture` is **declared-only**: no upstream artifact ever saw it, so nothing can cross-check it. It is recorded verbatim in the config's `library` block, and that is the whole defense — the values should be the best available estimate of the realized pool, not just the intended design.
+
+A library-derived spiked genotype that has no growth data is **reported, not fatal** (spikes can drop out of a real dataset), unlike the hand-supplied `spiked_genotypes` list, which still raises.
+
+`pool_fraction`/`bulk_fraction` are computed and snapshotted but **not yet consumed by the model** — `bulk_fraction` is the `f_g` the observable-level congression mixture will use (see `future/congression-physics-plan.md`), and the current backend still treats spiked as a binary via `congression_mask`/`ln_cfu0_spiked_mask`.
 
 ### Pre-fit model accounting (`tfmodel/model_stats.py`)
 
@@ -457,13 +471,13 @@ Both grid CLIs import from `tfscreen.util.grid_utils` for run-name generation, J
 | `simulate-and-analyze/run.sh` | Jinja2 shell template for simulate-and-analyze runs |
 | `tfmodel/grid.yaml` | Example tfmodel grid for `tfs-setup-grid` |
 | `tfmodel/run.srun` | Jinja2 SLURM template rendered into each tfmodel grid run subdir |
-| `process_raw/library_config.yaml` | Minimal library genetics config for `tfs-process-fastq` |
+| `process_raw/library_config.yaml` | Minimal library config for `tfs-process-fastq` and `tfs-configure-model --library_config` (genetics keys + `library_mixture`) |
 
 ### `process_raw` YAML
 
 `tfs-process-fastq` accepts a `run_config` argument that is passed to `LibraryManager`. It reads only the library genetics keys: `reading_frame`, `wt_seq`, `degen_sites`, `tiles`, `expected_5p`, `expected_3p`, `tile_combos`, `spiked_seqs`. All other keys are ignored.
 
-**Recommended practice**: maintain one `run_config.yaml` per experiment and pass it to both `tfs-simulate` and `tfs-process-fastq` — no need for a separate file. Use `examples/process_raw/library_config.yaml` only when you need a minimal standalone library config (e.g. for processing real data without a matching simulation).
+**Required practice**: maintain **one** `run_config.yaml` per experiment and pass that same file to `tfs-simulate`, `tfs-process-fastq` and `tfs-configure-model --library_config`. Nothing in the pipeline checks that the three saw the same file: the genotype-set check in `tfs-configure-model` catches a genetics mismatch, but `library_mixture` (which only `tfs-configure-model` and `tfs-simulate` read) can silently drift. Use `examples/process_raw/library_config.yaml` only when you need a minimal standalone library config (e.g. for processing real data without a matching simulation); note it carries `library_mixture` as well as the genetics keys.
 
 ### Terminology
 
@@ -502,6 +516,10 @@ Positional argument order (use only what the script needs):
 3. `data_file` — path to a long-form observable CSV (e.g. `tfs-cat-response`
    takes `data_file x_obs y_obs`; `tfs-extract-epistasis` takes `data_file y_obs`)
 
+### `--library_config`
+
+`tfs-configure-model`, `tfs-fit-genotypes` and `tfs-build-empirical` take the library YAML as `--library_config`. It is an exception to the positional-if-required rule: `tfs-configure-model` requires it only when `growth_df` is given, and `growth_df` is itself a flag, so a positional would be wrong for the binding-only call. Validate it in the function body instead.
+
 ### Output flag
 
 Always `--out_prefix` (never `--out_root`, `--out`, or `--output_file`). The function parameter must also be named `out_prefix`.

docs/source/analysis.rst

@@ -19,11 +19,13 @@ Step 1: Configure Model (``tfs-configure-model``)
 --------------------------------------------------
 
 Validates the input data, maps categorical labels to numerical indices, selects
-model components, and writes three configuration files:
+model components, and writes the configuration files:
 
 * ``{out_prefix}_config.yaml`` — main configuration read by all downstream steps
 * ``{out_prefix}_priors.csv`` — prior distribution settings for all parameters
 * ``{out_prefix}_guesses.csv`` — initial-value guesses for array parameters
+* ``{out_prefix}_library.csv`` — per-genotype library composition resolved from
+  ``--library_config`` (growth models only)
 
 ``binding_df`` (direct binding measurements) is the only required argument.
 ``growth_df`` is optional; omitting it configures a binding-only model. See
@@ -35,8 +37,20 @@ data is useful, and the exact file format for ``binding_df``, ``growth_df``,
 
     tfs-configure-model binding.csv \
         --growth_df growth.csv \
+        --library_config run_config.yaml \
         --out_prefix tfs_configure
 
+``--library_config`` is the library YAML describing the screened library — the
+**same file** passed to ``tfs-process-fastq``. It is required whenever
+``--growth_df`` is given. It tells the model which genotypes came in as
+monoclonal spikes (and so are congression-free), replacing the older
+hand-written ``--spiked`` list; because a spiked control is usually also
+encoded in the bulk sub-libraries, the resolved table records each genotype's
+expected ``pool_fraction`` and ``bulk_fraction`` rather than a yes/no flag.
+``tfs-configure-model`` fails if any genotype in the data is absent from the
+library, which catches a residue-numbering or wt-sequence mismatch between the
+library description and the reads it was called against.
+
 Key component flags (see :ref:`model-components` for the full list):
 
 * ``--condition_growth_model`` — how growth rate depends on TF occupancy (default: ``linear``)

docs/source/grid.rst

@@ -77,6 +77,7 @@ Grid YAML format
         variants:
           - binding_df: ../data/binding.csv
             growth_df:  ../data/growth.csv
+            library_config: ../data/run_config.yaml
 
       # 'auto' enumerates every registered component for an axis.
       # Incompatible combinations are skipped automatically.
@@ -120,8 +121,8 @@ Key rules:
   component axes; e.g. ``condition_growth`` rather than
   ``--condition_growth_model``).
 * Relative paths in ``configure_model`` blocks (``binding_df``, ``growth_df``,
-  ``thermo_data``) are resolved relative to the grid YAML and re-expressed
-  relative to each subdirectory in the written config.
+  ``library_config``, ``thermo_data``) are resolved relative to the grid YAML
+  and re-expressed relative to each subdirectory in the written config.
 * The ``auto`` form enumerates every registered component for the given axis.
   Incompatible combinations (e.g. ``power`` growth + ``logit`` theta_rescale)
   are caught by ``tfs-configure-model``, skipped, and logged in

docs/source/process-raw.rst

@@ -113,8 +113,14 @@ genotype across all samples, ready for hierarchical modelling.
     to match this row to a counts CSV file.
   * ``library`` — name of the physical library this sample belongs to. Genotypes
     are filtered and frequency-normalised within each library.
-  * ``sample_cfu`` — total colony-forming units (CFU) measured for this tube.
-  * ``sample_cfu_std`` — standard deviation of ``sample_cfu``.
+  * ``sample_ln_cfu`` — natural log of the total colony-forming units (CFU)
+    measured for this tube.
+  * ``sample_ln_cfu_std`` — standard deviation of ``sample_ln_cfu``.
+
+  Instead of ``sample_ln_cfu``/``sample_ln_cfu_std``, you may supply
+  ``sample_ln_cfu_var``, or the linear-space ``sample_cfu`` with
+  ``sample_cfu_std`` (or ``sample_cfu_var``); the log-space columns are then
+  inferred. Log-space columns are used when both are present.
 
   The following columns are not used by ``tfs-process-counts`` itself but are
   carried through into the ``ln_cfu`` output and are **required by the growth
@@ -179,8 +185,12 @@ can be matched to the correct growth conditions.
   * ``replicate`` — replicate index; matched to the growth data.
   * ``condition_pre`` — pre-selection condition name; matched to the growth
     data.
-  * ``sample_cfu`` — total CFU for this tube.
-  * ``sample_cfu_std`` — standard deviation of ``sample_cfu``.
+  * ``sample_ln_cfu`` — natural log of the total CFU for this tube.
+  * ``sample_ln_cfu_std`` — standard deviation of ``sample_ln_cfu``.
+
+  As for ``tfs-process-counts``, these may instead be inferred from
+  ``sample_ln_cfu_var`` or from ``sample_cfu`` with ``sample_cfu_std`` (or
+  ``sample_cfu_var``).
 
 * ``counts_csv_path``: Directory containing the per-sample count CSV files.
 * ``output_file``: Path for the output presplit CSV.

examples/process_raw/library_config.yaml

@@ -42,3 +42,23 @@ tile_combos: ["single-1", "single-2", "double-1-2"]
 spiked_seqs:
 - gccagccacgtttctgcgaaaacgcgggaaaaagtggaagcggcgatggcggagctgaattacattcccaaccgcgtggcacaacaactggcgggcaaacagtcgttgctgattggcgttgccacctccagtctggccctggcggcgccgtcgcaaattgtcgcggcgattctgtctcgcgccgatcaactgggtgccagcgtggtggtgtcgatg  # k84l
 - gccagccacgtttctgcgaaaacgcgggaaaaagtggaagcggcgattgcggagctgaattacattcccaaccgcgtggcacaacaactggcgggcaaacagtcgttgctgattggcgttgccacctccagtctggccctgcacgcgccgtcgcaaattgtcgcggcgattctgtctcgcgccgatcaactgggtgccagcgtggtggtgtcgatg  # m42i/k84l
+
+# ---------------------------------------------------------------------------
+# Library composition (for tfs-configure-model)
+#
+# Relative amount of each transformed sub-library in the pooled library.  Only
+# ratios matter.  Keys must be exactly the tile_combos above, plus 'spiked'
+# when spiked_seqs is defined.
+#
+# tfs-configure-model uses these -- with the genetics keys above -- to work out
+# each genotype's expected share of the pool and what fraction of its cells
+# come from the bulk (congression-affected) sub-libraries rather than from the
+# monoclonal spike.  Nothing cross-checks these numbers against the data, so
+# they should be the best available estimate of what actually went into the
+# pool, not just the intended design.
+# ---------------------------------------------------------------------------
+library_mixture:
+  single-1: 100
+  single-2: 100
+  double-1-2: 1_000
+  spiked: 1

examples/simulate-and-analyze/run.sh

@@ -42,6 +42,11 @@ fi
 echo ">>> Simulate library"
 tfs-simulate "${config_file}" "${run_dir}" --seed "${seed}"
 
+# The simulate config also describes the library (genetics keys +
+# library_mixture), so tfs-configure-model reads the same file.  Resolve it to
+# an absolute path before changing directory.
+library_config="$(cd "$(dirname "${config_file}")" && pwd)/$(basename "${config_file}")"
+
 cd "${run_dir}"
 
 # ---------------------------------------------------------------------------
@@ -66,7 +71,7 @@ tfs-configure-model \
     --theta_growth_noise_model logit_normal \
     --theta_binding_noise_model zero \
     --growth_noise_model normal_kt \
-    --spiked wt M42I H74A K84L M42I/H74A M42I/K84L H74A/K84L D88A \
+    --library_config "${library_config}" \
     --growth_shares_replicates \
     --epistasis
 

examples/simulate-empirical/simulate_config.yaml

@@ -11,7 +11,8 @@
 # Workflow:
 #   # 1. real data -> empirical generating distribution (writes emp_phenotype_model.json)
 #   tfs-build-empirical growth.csv 42 \
-#       --binding_file binding.csv --spiked_file spiked.txt --out_prefix emp
+#       --binding_file binding.csv --library_config library_config.yaml \
+#       --out_prefix emp
 #   # 2. simulate from it
 #   tfs-simulate simulate_config.yaml out_dir/
 #

examples/tfmodel/grid.yaml

@@ -35,6 +35,9 @@ configure_model:
     variants:
       - binding_df: ../data/binding.csv
         growth_df:  ../data/growth.csv
+        # The library description -- the same YAML used for tfs-process-fastq.
+        # Required whenever growth_df is given.
+        library_config: ../data/run_config.yaml
 
   - name: fixed_components
     variants: