diff --git a/.gitignore b/.gitignore
index 76f27d4..b12d296 100644
--- a/.gitignore
+++ b/.gitignore
@@ -199,8 +199,24 @@ reorganization.md
slurm-*.out
slurm-*.err
+# Generated Slurm job directory (.sub files, MANIFEST, backups)
+jobs/
+
+# Generated figures (regenerate from tutorials/ notebooks)
+plots/
+
# HPC scripts (local infrastructure)
scripts/
+# Manual/editor backup copies
+*.bak
+*.bak2
+
+# OpenEye run artifacts (Omega logs, Spruce/HYBRID parameter + status dumps)
+oeomega_*.txt
+*_settings.param
+*_status.txt
+*.oedu
+
# Legacy code outputs (not needed in repository)
src/TACTICS/thompson_sampling/legacy/src/
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..a897375
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,99 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.2.0] - 2026-07-18
+
+Parallel evaluation with slow evaluators (Fred docking, ROCS, ML models) was
+non-functional in 1.1.0. Three defects, reported by Donald van Pinxteren on
+2026-06-23, combined to make `processes > 1` unusable; all three are fixed.
+
+### Fixed
+
+- **Evaluation parallelism could not be enabled at all.**
+ `ThompsonSampler.from_config()` hardcoded `processes=1`, so setting
+ `processes` anywhere in a user script had no effect and there was no
+ API-level way to turn on parallel evaluation. A 1000-iteration Fred docking
+ screen ran single-threaded on a 128-core allocation with no error or
+ warning. `ThompsonSamplingConfig` now exposes `processes` (and
+ `min_cpds_per_core`), and `from_config()` passes them through.
+
+- **`processes > 1` always crashed with OpenEye evaluators.**
+ `ParallelEvaluator` called `pool.map(sampler.evaluate, ...)`, which pickles
+ the bound method and therefore the sampler, the evaluator, and any
+ SWIG-wrapped C++ object it holds. `OEDock` (Fred) and the ROCS shape engine
+ raise `TypeError: cannot pickle 'SwigPyObject' object`, so the run died on
+ the first batch before any docking happened.
+
+ Workers now build their own evaluator. The pool is created with an
+ `initializer` that constructs the evaluator inside each worker from its
+ picklable Pydantic config, once per worker rather than once per molecule.
+ The OpenEye object is never pickled and never crosses the pipe. This works
+ under both `fork` and `spawn`, so it is correct on Linux, macOS and Windows
+ rather than only where `fork` is the default.
+
+ Verified end to end with a real `OEDock` under `spawn`: the evaluator is
+ confirmed unpicklable, and all products still dock correctly across workers.
+
+- **Segfault on `import TACTICS` when OpenEye and prompt_toolkit are both installed.**
+ `evaluators.py` imported the OpenEye toolkits at module level, which
+ initialises the global C-level `libexpat` parser. `tqdm` pulls in
+ `prompt_toolkit`, whose progress-bar formatter calls
+ `xml.dom.minidom.parseString()` at module level and re-enters `libexpat`
+ through `pyexpat`, conflicting with OpenEye's initialisation and killing the
+ interpreter (exit 139) before user code ran. OpenEye is now imported lazily,
+ on first construction of an OpenEye-backed evaluator.
+
+- **`MLClassifierEvaluator` failed when OpenEye was absent.** `joblib` was
+ imported inside the OpenEye `try/except`, leaving it undefined in
+ environments without OpenEye despite being unrelated to it.
+
+- **`exp()` overflow in the CATS Boltzmann softmax.** On heavy-tailed,
+ zero-inflated score landscapes a single outlier combined with a small CATS
+ temperature overflowed `exp()` to `inf`, producing NaN probabilities and
+ aborting the search with "probabilities contain NaN". The softmax now
+ subtracts `max(z)` before exponentiating, which is shift-invariant and
+ leaves the resulting probabilities unchanged.
+
+### Added
+
+- `ThompsonSamplingConfig.processes` and `.min_cpds_per_core` — evaluation
+ parallelism, defaulting to `1` (unchanged behaviour).
+- `ThompsonSampler.set_evaluator(evaluator, evaluator_config=...)` — supplies
+ the picklable recipe workers use to rebuild the evaluator. `from_config()`
+ provides it automatically; a warning is emitted if `processes > 1` without it.
+- `LookupEvaluatorConfig.default_score` — score for product codes absent from
+ the lookup table. Defaults to `None` (existing NaN behaviour); set to `0.0`
+ for sparse libraries such as DEL read counts, where an unmeasured
+ combination is a true non-binder rather than missing data.
+- Layer-1 and layer-2 search diagnostics in `library_analysis.diagnostic_plots`
+ (`plot_gmic_directed_exploration`, `plot_adaptive_intensity`,
+ `plot_reagent_usage_action_panel`, `plot_gmic_vs_oracle`).
+
+### Changed
+
+- **TT-TS GMIC min-observation gate removed.** `TopTwoSelection._calculate_gmic`
+ returned `0.0` whenever the least-observed active reagent fell below
+ `min_observations`; `RouletteWheelSelection` never had this gate. On large
+ components a single under-observed reagent pinned the whole component's GMIC
+ to zero, distorting the rotation. Removing it raises adenine TT-TS top-100
+ recovery from 87.1 to 93.2 and roughly halves replicate variance
+ (sd 18.4 → 10.6). `min_observations` is still accepted for backward
+ compatibility but is now inert in `TopTwoSelection`.
+
+### Notes for users
+
+Set `processes` to the number of cores you have allocated when using a slow
+evaluator; leave it at `1` for `LookupEvaluator`/`DBEvaluator`, where process
+overhead exceeds lookup cost. On macOS and Windows the default start method is
+`spawn`, so entry-point scripts must be guarded with
+`if __name__ == "__main__":`. Under `spawn`, and on clusters generally, the
+design unit (`.oedu`) must be readable from every node — automatic on shared
+filesystems (NFS, GPFS).
+
+## [1.1.0] - earlier
+
+See git history for releases prior to this changelog.
diff --git a/README.md b/README.md
index 3f08bd9..dec2e3a 100644
--- a/README.md
+++ b/README.md
@@ -1,408 +1,227 @@
-# TACTICS: Thompson Sampling-Assisted Chemical Targeting and Iterative Compound Selection for Drug Discovery
+# TACTICS
-A comprehensive library for Thompson Sampling-based optimization of chemical combinatorial libraries, featuring a unified architecture with flexible strategy selection, modern Pydantic configuration, and preset configurations for out-of-the-box usage.
+**Find the best compounds in a combinatorial library without enumerating all of them.**
-## Quick Start with Interactive Tutorials
+TACTICS (Thompson Sampling-Assisted Chemical Targeting and Iterative Compound
+Selection) uses Thompson Sampling to search ultra-large combinatorial libraries,
+scoring only a small fraction of the products while still recovering most of the
+top hits. A typical run evaluates **1–2% of the library** and recovers **~85–95%
+of the true top-100**.
-TACTICS includes interactive [marimo](https://marimo.io) notebooks for learning and exploration.
-For full documentation, see the [TACTICS Documentation](https://aakankschit.github.io/TACTICS/).
+📖 [Documentation](https://aakankschit.github.io/TACTICS/) · 🧪 [Tutorials](#learn-more) · 📋 [Changelog](CHANGELOG.md)
-### Installation
+---
-```bash
-pip install chem-tactics[tutorials] # Includes marimo
-```
+## Is this for you?
-### Running Tutorials
+TACTICS fits if you have:
-**As an interactive app** (recommended for exploration):
-```bash
-marimo run tutorials/thompson_sampling_tutorial.py
-```
+- A **combinatorial library** defined by a reaction plus lists of reagents
+ (2 or 3 components), and
+- A **scoring function** that is too slow to run on every product — docking,
+ shape similarity, or an ML model — or a big table of precomputed scores.
+
+If your library is small enough to score exhaustively, you don't need TACTICS.
+
+## Install
-**In edit mode** (for learning/modification):
```bash
-marimo edit tutorials/thompson_sampling_tutorial.py
+pip install chem-tactics
```
-### Available Tutorials
-
-| Tutorial | Description |
-|----------|-------------|
-| `library_enumeration_tutorial.py` | SynthesisPipeline and enumeration |
-| `thompson_sampling_tutorial.py` | Selection strategies comparison |
-| `reaction_config_builder.py` | ReactionConfig builder |
-| `library_component_comparison.py` | Library component analysis |
-| `legacy_vs_current_comparison.py` | Legacy vs current benchmark |
-
-> **Note**: Tutorials default to the bundled Thrombin dataset. Select "Local Data" mode to use your own files.
-
-## Key Features
-
-- **Unified Thompson Sampling Framework**: Single `ThompsonSampler` with pluggable selection strategies
-- **Multiple Selection Strategies**:
- - Greedy (pure exploitation)
- - Roulette Wheel (adaptive thermal cycling)
- - UCB (Upper Confidence Bound)
- - Epsilon-Greedy (balanced exploration/exploitation)
- - Bayes-UCB (Bayesian upper confidence bound)
- - Boltzmann (temperature-based selection)
-- **Warmup Strategies**: Balanced (recommended), Standard, Enhanced
-- **Preset Configurations**: 5 ready-to-use presets for common use cases
-- **Modern Pydantic Configuration**: Type-safe configuration with full validation
-- **Parallel Processing**: Batch mode with multiprocessing for expensive evaluators
-- **Multiple Evaluators**: Lookup, Database, ROCS, Fred, ML classifiers, and more
-- **Synthesis Pipeline**: `SynthesisPipeline` architecture for single-step, alternative SMARTS, and multi-step reactions
-- **SMARTS Toolkit**: `ReactionDef` with built-in validation, visualization, and protecting group support
-- **Library Enumeration**: Efficient generation of combinatorial reaction products with `write_enumerated_library()`
-- **Library Analysis**: Comprehensive analysis and visualization tools
-- **Polars DataFrames**: Fast, efficient data handling throughout
-
-## Package Structure
+Requires Python 3.11+.
-```
-TACTICS/
-├── thompson_sampling/
-│ ├── config.py # ThompsonSamplingConfig (Pydantic v2)
-│ ├── presets.py # Preset configurations
-│ ├── factories.py # Factory functions for component creation
-│ ├── core/ # Core unified sampler
-│ │ ├── sampler.py # ThompsonSampler (unified)
-│ │ ├── evaluators.py # All evaluator classes
-│ │ └── evaluator_config.py # Evaluator Pydantic configs
-│ ├── strategies/ # Selection strategies
-│ │ ├── greedy.py
-│ │ ├── roulette_wheel.py
-│ │ ├── ucb.py
-│ │ ├── epsilon_greedy.py
-│ │ ├── bayes_ucb.py
-│ │ └── config.py # Strategy Pydantic configs
-│ ├── warmup/ # Warmup strategies
-│ │ └── config.py # Warmup Pydantic configs (Balanced, Standard, Enhanced)
-│ └── baseline.py # Random baseline sampling
-├── library_enumeration/ # Library generation tools
-│ ├── synthesis_pipeline.py # SynthesisPipeline - main entry point
-│ ├── enumeration_utils.py # EnumerationResult, EnumerationError
-│ ├── file_writer.py # write_enumerated_library()
-│ ├── generate_products.py # Product generation utilities
-│ └── smarts_toolkit/ # SMARTS validation and configuration
-│ ├── config.py # ReactionDef, ReactionConfig, StepInput, DeprotectionSpec
-│ ├── _validator.py # ValidationResult, internal validation
-│ └── constants.py # Protecting groups, salt fragments
-└── library_analysis/ # Analysis and visualization
-```
+
+Optional extras and development install
-## Repository Structure
+```bash
+pip install chem-tactics[tutorials] # interactive marimo notebooks
+pip install chem-tactics[test] # test dependencies
+# development install
+git clone https://github.com/aakankschit/TACTICS.git
+cd TACTICS
+pip install -e ".[test]"
```
-TACTICS/
-├── src/TACTICS/ # Core package (pip installable)
-│ ├── thompson_sampling/ # Thompson Sampling algorithms
-│ ├── library_enumeration/ # Library generation tools
-│ ├── library_analysis/ # Analysis and visualization
-│ └── data/ # Bundled tutorial datasets
-│ └── thrombin/ # Thrombin inhibitor dataset
-│
-├── tutorials/ # Interactive marimo tutorials
-├── tests/ # Unit and integration tests
-└── docs/ # Sphinx documentation
-```
-
-## Quick Start
+
-### Simple Out-of-the-Box Usage with Presets (Recommended)
+## Quickstart
-The easiest way to get started is using presets with `SynthesisPipeline`:
+Screen a two-component amide library against a table of precomputed scores:
```python
from TACTICS.library_enumeration import SynthesisPipeline, ReactionConfig, ReactionDef
from TACTICS.thompson_sampling import ThompsonSampler, get_preset
from TACTICS.thompson_sampling.core.evaluator_config import LookupEvaluatorConfig
-# 1. Create synthesis pipeline (single source of truth for reactions)
-rxn_config = ReactionConfig(
+# 1. Describe the library: one reaction + one reagent file per component
+pipeline = SynthesisPipeline(ReactionConfig(
reactions=[ReactionDef(
reaction_smarts="[C:1](=O)[OH].[NH2:2]>>[C:1](=O)[NH:2]",
step_index=0,
- description="Amide coupling"
)],
- reagent_file_list=["acids.smi", "amines.smi"]
-)
-pipeline = SynthesisPipeline(rxn_config)
+ reagent_file_list=["acids.smi", "amines.smi"],
+))
-# 2. Create evaluator config
+# 2. Describe how to score a product
evaluator = LookupEvaluatorConfig(ref_filename="scores.csv")
-# 3. Get a preset configuration
-config = get_preset(
- "fast_exploration", # Quick screening with epsilon-greedy
- synthesis_pipeline=pipeline,
- evaluator_config=evaluator,
- mode="minimize", # Use "minimize" for docking scores
- num_iterations=1000
-)
-
-# 4. Create sampler from config and run
+# 3. Take a tuned preset and run
+config = get_preset(synthesis_pipeline=pipeline, evaluator_config=evaluator)
sampler = ThompsonSampler.from_config(config)
-warmup_df = sampler.warm_up(num_warmup_trials=config.num_warmup_trials)
-results_df = sampler.search(num_cycles=config.num_ts_iterations)
+
+sampler.warm_up(num_warmup_trials=config.num_warmup_trials)
+results = sampler.search(num_cycles=config.num_ts_iterations)
sampler.close()
-# 5. Analyze top results
-print(results_df.sort("score").head(10))
+print(results.sort("score", descending=True).head(10))
```
-**Available Presets:**
-- `"fast_exploration"` - Epsilon-greedy strategy, quick screening
-- `"parallel_batch"` - Batch processing with multiprocessing (for slow evaluators)
-- `"conservative_exploit"` - Greedy strategy, focus on best reagents
-- `"balanced_sampling"` - UCB strategy with theoretical guarantees
-- `"diverse_coverage"` - Maximum diversity exploration
+`results` is a Polars DataFrame of every product that was evaluated, with
+columns `score`, `SMILES` and `Name`.
+
+> With `LookupEvaluator` and `DBEvaluator`, TACTICS skips molecule generation
+> entirely and scores by product code, so `SMILES` reads `FAIL` and products
+> are identified by `Name` (the underscore-joined reagent names). Evaluators
+> that need a molecule — docking, ROCS, fingerprints — populate `SMILES`.
-### Parallel Batch Processing (for Expensive Evaluators)
+## Slow scoring functions: use more cores
-For slow evaluators (docking, ML models), use batch mode with multiprocessing:
+Docking and ROCS take seconds per molecule, so evaluation dominates runtime.
+Set `processes` to the number of cores you have — each worker builds its own
+docking session, so OpenEye evaluators parallelize correctly:
```python
-from TACTICS.library_enumeration import SynthesisPipeline, ReactionConfig, ReactionDef
-from TACTICS.thompson_sampling import ThompsonSampler, get_preset
from TACTICS.thompson_sampling.core.evaluator_config import FredEvaluatorConfig
-# Create synthesis pipeline
-rxn_config = ReactionConfig(
- reactions=[ReactionDef(
- reaction_smarts="[C:1](=O)[OH].[NH2:2]>>[C:1](=O)[NH:2]",
- step_index=0
- )],
- reagent_file_list=["acids.smi", "amines.smi"]
-)
-pipeline = SynthesisPipeline(rxn_config)
-
-# Configure slow evaluator (molecular docking)
-evaluator = FredEvaluatorConfig(design_unit_file="receptor.oedu")
-
-# Get parallel batch preset
config = get_preset(
- "parallel_batch",
synthesis_pipeline=pipeline,
- evaluator_config=evaluator,
- mode="minimize", # Docking scores (lower is better)
- batch_size=100, # Sample 100 compounds per cycle
+ evaluator_config=FredEvaluatorConfig(design_unit_file="receptor.oedu"),
+ mode="minimize", # docking scores: lower is better
+ batch_size=100,
)
+config.processes = 32 # or os.cpu_count()
-# Create sampler and run
sampler = ThompsonSampler.from_config(config)
-warmup_df = sampler.warm_up(num_warmup_trials=config.num_warmup_trials)
-results_df = sampler.search(num_cycles=config.num_ts_iterations)
-sampler.close()
```
-### Custom Configuration (Advanced)
+> **Leave `processes=1` for `LookupEvaluator` and `DBEvaluator`.** Their lookups
+> are faster than the cost of shipping work to a worker, so parallelism makes
+> those runs *slower*.
-For full control, create custom configurations:
+> **On macOS/Windows**, guard your entry point with
+> `if __name__ == "__main__":` — the default `spawn` start method re-imports
+> your script in every worker. On a cluster, make sure the `.oedu` receptor is
+> readable from every node (automatic on NFS/GPFS).
-```python
-from TACTICS.library_enumeration import SynthesisPipeline, ReactionConfig, ReactionDef
-from TACTICS.thompson_sampling import ThompsonSampler, ThompsonSamplingConfig
-from TACTICS.thompson_sampling.strategies.config import RouletteWheelConfig
-from TACTICS.thompson_sampling.warmup.config import BalancedWarmupConfig
-from TACTICS.thompson_sampling.core.evaluator_config import LookupEvaluatorConfig
+## What can I plug in?
-# Create synthesis pipeline
-rxn_config = ReactionConfig(
- reactions=[ReactionDef(
- reaction_smarts="[C:1](=O)[OH].[NH2:2]>>[C:1](=O)[NH:2]",
- step_index=0
- )],
- reagent_file_list=["acids.smi", "amines.smi"]
-)
-pipeline = SynthesisPipeline(rxn_config)
+**Scoring functions** (`evaluator_config`):
-# Create fully customized configuration
-config = ThompsonSamplingConfig(
- synthesis_pipeline=pipeline,
- num_ts_iterations=5000,
- num_warmup_trials=5,
- strategy_config=RouletteWheelConfig(
- mode="maximize",
- alpha=0.1, # Initial heating temperature
- beta=0.1, # Initial cooling temperature
- ),
- warmup_config=BalancedWarmupConfig(
- observations_per_reagent=5,
- use_per_reagent_variance=True,
- ),
- evaluator_config=LookupEvaluatorConfig(
- ref_filename="scores.csv",
- score_col="binding_affinity"
- ),
- batch_size=10,
- log_filename="optimization.log"
-)
+| Evaluator | Use it for | Speed |
+|---|---|---|
+| `LookupEvaluatorConfig` | A CSV/Parquet table of precomputed scores | instant |
+| `DBEvaluatorConfig` | Scores in a SQLite database | instant |
+| `FPEvaluatorConfig` | Fingerprint similarity to a reference ligand | fast |
+| `ROCSEvaluatorConfig` | 3D shape/colour overlay (OpenEye) | slow |
+| `FredEvaluatorConfig` | Docking into a receptor (OpenEye) | slow |
+| `MLClassifierEvaluatorConfig` | A trained scikit-learn model | varies |
-# Create sampler and run
-sampler = ThompsonSampler.from_config(config)
-warmup_df = sampler.warm_up(num_warmup_trials=config.num_warmup_trials)
-results_df = sampler.search(num_cycles=config.num_ts_iterations)
-sampler.close()
+**Presets** (`get_preset(name, ...)`):
-# Save results
-results_df.write_csv("my_results.csv")
-```
-
-### Random Baseline Sampling
-
-```python
-from TACTICS.library_enumeration import SynthesisPipeline, ReactionConfig, ReactionDef
-from TACTICS.thompson_sampling import RandomBaselineConfig, run_random_baseline
-from TACTICS.thompson_sampling.core.evaluator_config import LookupEvaluatorConfig
-
-# Create synthesis pipeline
-rxn_config = ReactionConfig(
- reactions=[ReactionDef(
- reaction_smarts="[C:1](=O)[OH].[NH2:2]>>[C:1](=O)[NH:2]",
- step_index=0
- )],
- reagent_file_list=["acids.smi", "amines.smi"]
-)
-pipeline = SynthesisPipeline(rxn_config)
+| Preset | What it does |
+|---|---|
+| `recommended` *(default)* | Top-Two Thompson Sampling — best overall |
+| `recommended_rws` | Roulette wheel with criticality-aware thermal cycling |
+| `baseline` | Balanced warmup + greedy, for measuring what the search adds |
+| `legacy_rws` | Reproduces Zhao et al. 2025 |
-config = RandomBaselineConfig(
- synthesis_pipeline=pipeline,
- evaluator_config=LookupEvaluatorConfig(ref_filename="scores.csv"),
- num_trials=1000,
- num_to_save=100,
- ascending_output=False,
- outfile_name="random_results.csv"
-)
-
-results_df = run_random_baseline(config)
-```
-
-## Configuration
+Start with `recommended`. If you are benchmarking, run `recommended` and
+`recommended_rws` and take the better result — they favour different library
+structures.
-### Pydantic Configuration Models
+
+Building a config by hand instead of using a preset
-The package uses Pydantic v2 for robust configuration validation:
+Presets are just `ThompsonSamplingConfig` objects, so you can construct one
+directly to control the strategy, warmup and search budget:
```python
-from TACTICS.library_enumeration import SynthesisPipeline, ReactionConfig, ReactionDef
-from TACTICS.thompson_sampling import ThompsonSamplingConfig
-from TACTICS.thompson_sampling.strategies.config import EpsilonGreedyConfig
-from TACTICS.thompson_sampling.warmup.config import BalancedWarmupConfig
-from TACTICS.thompson_sampling.core.evaluator_config import LookupEvaluatorConfig
-
-# Create synthesis pipeline
-rxn_config = ReactionConfig(
- reactions=[ReactionDef(
- reaction_smarts="[C:1](=O)[OH].[NH2:2]>>[C:1](=O)[NH:2]",
- step_index=0
- )],
- reagent_file_list=["acids.smi", "amines.smi"]
-)
-pipeline = SynthesisPipeline(rxn_config)
+from TACTICS.thompson_sampling import ThompsonSamplingConfig, ThompsonSampler
+from TACTICS.thompson_sampling.strategies.config import TopTwoConfig
+from TACTICS.thompson_sampling.warmup.config import EnhancedWarmupConfig
-# Automatic validation and type checking
config = ThompsonSamplingConfig(
- synthesis_pipeline=pipeline, # Required: single source of truth
+ synthesis_pipeline=pipeline,
+ evaluator_config=evaluator,
+ strategy_config=TopTwoConfig(mode="maximize", beta=0.5),
+ warmup_config=EnhancedWarmupConfig(),
+ num_warmup_trials=3,
num_ts_iterations=1000,
- strategy_config=EpsilonGreedyConfig(mode="maximize", epsilon=0.2),
- warmup_config=BalancedWarmupConfig(),
- evaluator_config=LookupEvaluatorConfig(ref_filename="scores.csv"),
+ batch_size=100,
+ processes=1,
+ seed=42,
)
-```
-
-### Configuration Validation
-```python
-from pydantic import ValidationError
-
-# Invalid configuration raises ValidationError
-try:
- rxn = ReactionDef(
- reaction_smarts="invalid-smarts", # ValidationError: Invalid SMARTS
- step_index=0
- )
-except ValidationError as e:
- print(f"Configuration error: {e}")
+sampler = ThompsonSampler.from_config(config)
```
-## Testing
-
-The package includes comprehensive tests for configuration validation:
-
-```bash
-# Run all tests
-pytest tests/
-
-# Run configuration tests
-pytest tests/test_config_validation.py -v
+Configs are Pydantic models, so invalid values raise `ValidationError` at
+construction rather than failing mid-run. Other selection strategies —
+`GreedyConfig`, `RouletteWheelConfig`, `UCBConfig`, `EpsilonGreedyConfig`,
+`BayesUCBConfig` — are available for baselines and comparisons.
-# Run with coverage
-pytest tests/ --cov=TACTICS --cov-report=html
-```
-
-## Documentation
+See the [configuration docs](https://aakankschit.github.io/TACTICS/) for the
+full reference.
+
-- **Full Documentation**: [TACTICS Documentation](https://aakankschit.github.io/TACTICS/)
-- **Interactive Tutorials**: See `tutorials/` for marimo notebooks
-- **API Reference**: Build locally with `cd docs && make html`
+## Learn more
-## Installation
+Interactive [marimo](https://marimo.io) notebooks in `tutorials/` (they default
+to a bundled thrombin dataset, so they run with no setup):
```bash
-# Clone repository and install package in development mode
-git clone https://github.com/aakankschit/TACTICS.git
-cd TACTICS
-pip install -e .
-
-# With interactive tutorials (marimo):
-pip install -e ".[tutorials]"
-
-# With test dependencies:
-pip install -e ".[test]"
+marimo run tutorials/thompson_sampling_tutorial.py # compare strategies
+marimo edit tutorials/library_enumeration_tutorial.py # build a library
```
-## Requirements
+| Tutorial | What it covers |
+|---|---|
+| `thompson_sampling_tutorial.py` | Comparing selection strategies |
+| `library_enumeration_tutorial.py` | `SynthesisPipeline` and enumeration |
+| `reaction_config_builder.py` | Building and validating reaction SMARTS |
+| `custom_evaluator_tester.py` | Writing your own evaluator |
-- Python 3.11+
-- Multiprocessing support
+- **API reference and guides**: [TACTICS Documentation](https://aakankschit.github.io/TACTICS/)
+- **Runnable scripts**: see `examples/`
+- **Build docs locally**: `cd docs && make html`
## Contributing
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Add tests for new functionality
-5. Ensure all tests pass
-6. Submit a pull request
-
-## License
-
-This project is licensed under the MIT License - see the LICENSE file for details.
+Fork, branch, add tests for new behaviour, make sure `pytest tests/` passes,
+then open a pull request.
## Citation
-If you use TACTICS in your research, please cite:
-
```bibtex
@software{tactics,
- title={TACTICS: Thompson Sampling-Assisted Chemical Targeting and Iterative Compound Selection for Drug Discovery},
+ title={TACTICS: Thompson Sampling-Assisted Chemical Targeting and
+ Iterative Compound Selection for Drug Discovery},
author={Aakankschit Nandkeolyar},
year={2025},
- url={https://github.com/your-org/TACTICS}
+ url={https://github.com/aakankschit/TACTICS}
}
```
## Support
-For questions and support:
-- Open an issue on GitHub
-- Contact: anandkeo@uci.edu
+Open an issue on GitHub, or contact anandkeo@uci.edu.
+
+Licensed under the MIT License — see [LICENSE](LICENSE).
---
diff --git a/docs/handoffs/2026-04-09_ema_cats_batch50_benchmark.md b/docs/handoffs/2026-04-09_ema_cats_batch50_benchmark.md
new file mode 100644
index 0000000..1e93bdc
--- /dev/null
+++ b/docs/handoffs/2026-04-09_ema_cats_batch50_benchmark.md
@@ -0,0 +1,129 @@
+# EMA CATS + Batch Size Benchmark Results (2026-04-07 to 2026-04-09)
+
+## Summary
+
+Three full benchmarks were run to evaluate the rotation-evaluation ordering fix, EMA-smoothed CATS multiplier, and batch_size=50 hypothesis. The EMA fix improves TT-TS reliability on hard docking libraries (variance reduction) without affecting aggregate recovery. Batch_size=50 is a negative result — GMIC is batch-size invariant.
+
+## Benchmarks Run
+
+| Benchmark | Config | Output |
+|-----------|--------|--------|
+| `full_benchmark` (baseline) | batch=100, no EMA | `outputs/full_benchmark/` |
+| `full_benchmark_ema_cats` | batch=100, EMA decay=0.1, rotation fix | `outputs/full_benchmark_ema_cats/` |
+| `full_benchmark_batch50` | batch=50, EMA decay=0.1, rotation fix | `outputs/full_benchmark_batch50/` |
+
+All benchmarks: 218,400 trials, 24 libraries, 5 methods, 20 replicates, 10 top-N thresholds.
+
+## Aggregate Recovery (Top-100)
+
+| Method | batch=100 (original) | batch=100 + EMA | batch=50 + EMA |
+|--------|---------------------|-----------------|----------------|
+| TACTICS Enhanced-TT-TS (GMIC) | 86.5 +/- 15.6 | 86.4 +/- 15.7 | 86.3 +/- 15.8 |
+| TACTICS Enhanced-RWS (GMIC) | 85.8 +/- 15.7 | 85.9 +/- 15.5 | 85.7 +/- 15.9 |
+| Legacy Enhanced-RWS | 82.9 +/- 15.6 | 82.9 +/- 15.6 | 82.8 +/- 15.7 |
+| TACTICS Balanced-Greedy | 82.0 +/- 19.0 | 82.0 +/- 19.0 | 82.1 +/- 18.9 |
+| Legacy Standard-Greedy | 81.2 +/- 19.6 | 81.2 +/- 19.7 | 81.1 +/- 19.7 |
+
+**Conclusion**: No meaningful change in aggregate recovery across any configuration.
+
+## Docking Library Results (where EMA fix had visible effect)
+
+### Adenine (hardest SAR, 3-comp, 17% docking failures)
+
+| Method | Original | + EMA | + batch=50 |
+|--------|----------|-------|------------|
+| TT-TS | 93.9% +/- 13.9% | **96.0% +/- 4.0%** | **96.8% +/- 1.1%** |
+| RWS | 71.8% +/- 27.6% | 78.5% +/- 24.0% | 72.8% +/- 27.0% |
+| Legacy RWS | 40.1% +/- 23.8% | 45.1% +/- 27.1% | 45.9% +/- 27.0% |
+| Balanced-Greedy | 31.6% +/- 0.8% | 31.5% +/- 0.9% | 31.6% +/- 1.1% |
+| Legacy Greedy | 31.8% +/- 1.1% | 31.4% +/- 1.5% | 31.0% +/- 2.4% |
+
+TT-TS std progression: **13.9% -> 4.0% -> 1.1%** across the three benchmarks.
+
+### Thrombin (imbalanced 130x3844, 3-comp)
+
+| Method | Original | + EMA | + batch=50 |
+|--------|----------|-------|------------|
+| TT-TS | 81.0% +/- 2.2% | **81.8% +/- 1.4%** | 81.8% +/- 1.8% |
+| RWS | 80.5% +/- 2.4% | 80.2% +/- 2.9% | **81.2% +/- 2.2%** |
+
+### Amide and Quinazoline
+
+Flat across all configurations — near ceiling (amide ~97%) or greedy-reachable (quinazoline ~85%).
+
+## ROCS Libraries
+
+No change. All 20 ROCS libraries show deltas within +/-0.6% recovery and +/-0.5% std across all three benchmarks. The EMA fix and batch size change are neutral on ROCS.
+
+## GMIC Stability Analysis
+
+Diagnostic data confirms GMIC is stable from cycle 0:
+
+| Library | Stabilization cycle | Mean cycle-to-cycle delta |
+|---------|-------------------|--------------------------|
+| adenine | 0-1 | 0.001-0.003 |
+| mannich | 0-1 | 0.002-0.021 |
+| orru | 0-13 | 0.003-0.029 |
+| rxn206 | 0-6 | 0.001-0.018 |
+| quinazoline | 0-10 | 0.0002-0.002 |
+
+GMIC is computed from cumulative posteriors, not the current batch. Batch size does not affect signal quality.
+
+## TT-TS Disagreement Analysis
+
+Most components spend majority of time in saturated region (EMA > 0.8):
+
+| Library | Component | % Saturated (>0.8) | % Neutral | % Resolved (<0.3) |
+|---------|-----------|--------------------|-----------|--------------------|
+| adenine | comp 0 | 22% | 56% | 22% |
+| adenine | comp 1 | 95% | 5% | 0% |
+| quinazoline | comp 0 | 94% | 6% | 1% |
+| orru | comp 2 | 0-6% | 40-66% | 34-56% |
+
+Only orru comp 2 consistently enters the resolved region, triggering heated_scale inflation (3.5-4.0x). The adaptive mechanism is primarily unidirectional (decaying heated_scale from saturation).
+
+## EMA Relative GMIC (RWS Diagnostic)
+
+With `cats_ema_decay=0.1`, the EMA relative GMIC provides stable directional signal for CATS multiplier:
+
+| Library | Component | Raw GMIC | EMA Relative GMIC | CATS Multiplier |
+|---------|-----------|----------|-------------------|-----------------|
+| adenine | comp 0 (critical) | 2.00 | 1.376 | 0.860 (cooled) |
+| adenine | comp 1 (flexible) | 0.90 | 0.618 | 1.164 (heated) |
+| adenine | comp 2 (neutral) | 1.46 | 1.005 | 0.997 (neutral) |
+| quinazoline | all | 2.0-2.4 | 0.91-1.06 | ~1.0 (flat) |
+
+The EMA smooths out per-cycle noise, giving the CATS multiplier a consistent directional signal. Previously (without EMA), the multiplier oscillated randomly.
+
+## Statistical Significance
+
+TACTICS methods significantly outperform Legacy on 16/24 libraries (all pairwise comparisons p < 0.05). On the remaining 8, differences are negligible effect size — these are libraries where greedy methods already perform well (easy SAR). Full Tukey HSD results in `outputs/full_benchmark_batch50/analysis/tukey_hsd_results.parquet`.
+
+## Dead End: batch_size=50
+
+**Hypothesis**: Smaller batches -> more frequent GMIC rotations -> better adaptation -> improved recovery.
+
+**Result**: No improvement. GMIC is batch-size invariant because it's computed from cumulative posteriors. Smaller batches double the number of rotation decisions but provide the same information per decision.
+
+**What this rules out**: Any batch_size reduction as a tuning lever for GMIC-based methods. The cumulative posterior computation makes GMIC insensitive to batch granularity. Halving, quartering, or any other reduction will not change the rotation signal quality. The only effect is increased per-cycle overhead.
+
+## Infrastructure Changes
+
+- Moved 21 `.sub` files from repo root to `jobs/`, created `jobs/MANIFEST.md`
+- Fixed merge job time limit: 1h -> 8h (was timing out on large libraries)
+- Fixed `pl.concat()` schema mismatch in merge: added `how="diagonal"` for diagnostics tier (TT-TS has 16 cols, CATS has 27)
+- Fixed posthoc job: GPU partition -> standard (CPU-only work)
+- Fixed diagnostic benchmark: added missing `cats_ema_decay=0.1` in `RouletteWheelConfig`
+- Updated diagnostic plots notebook to save PDFs to `plots/` directory
+
+## Recommendation
+
+- **Keep batch_size=100** as default for TACTICS methods
+- **Keep `cats_ema_decay=0.1`** as default for RWS — provides smoother CATS multiplier
+- **Keep rotation-evaluation ordering fix** — improves reliability on hard libraries
+- **Revert `_benchmark_config.py`** back to batch_size=100 before next benchmark
+
+## Heptabase Cards Needed
+
+- [DEAD END] batch_size=50 does not improve recovery (marked PENDING in STATUS.md)
+- [RESULT] EMA CATS + rotation fix reliability improvement (optional — finding is in STATUS.md)
diff --git a/docs/handoffs/2026-05-30_docking-benchmark-expansion.md b/docs/handoffs/2026-05-30_docking-benchmark-expansion.md
new file mode 100644
index 0000000..8cd6b17
--- /dev/null
+++ b/docs/handoffs/2026-05-30_docking-benchmark-expansion.md
@@ -0,0 +1,212 @@
+# Handoff — Docking-Benchmark Expansion (sEH + KinDEL + JNK3-ROCS)
+
+**Date:** 2026-05-30
+**Purpose:** Self-contained execution plan so this work can resume after a context clear.
+**Scope decision (user, 2026-05-30):** Expand the Paper-1 docking benchmark beyond the
+current 4 base chemistries (adenine, amide, quinazoline, thrombin) by adding new
+combinatorial docking datasets. Headline new case study = **sEH (soluble epoxide
+hydrolase)**, our own lab's DEL data, with an experimental-binder overlay. Move fast.
+
+---
+
+## 1. The three new dataset streams
+
+| stream | chemistry / comp | size target | target/score | status |
+|---|---|---|---|---|
+| **sEH** (centerpiece) | 3-comp amide-coupling DEL | **~60M** | dock vs sEH receptor (PDB **4OD0**, TBD-confirm) + DEL read_count overlay | SMARTS recovered; needs reagent reconciliation + enumeration + docking |
+| **KinDEL MAPK14 + DDR1** | 3-comp trisynthon | subset to 1–10M (81M full) | insitro docked poses / Kd + (optional) our docking | usable under license; FETCH not vendor |
+| **ROCS-libs-on-JNK3** | reuse enumerable ROCS libs (1×2-comp + 1×3-comp) | native | dock vs JNK3 2ZDT (already prepped) | lowest priority; SMARTS already in reactions.csv |
+
+User framing for JNK3 reuse (important for manuscript): the paper's claim is **"find top
+hits without screening the whole library,"** treating reference scores as ground truth *by
+definition of the benchmark* — NOT claiming docking is physical truth (docking ranks; FEP
+would be truth). So generating docking data on JNK3 for ROCS libraries to get varied
+reagent-scoring landscapes is legitimate; we are not asserting JNK3 biological relevance.
+
+---
+
+## 2. STANDARDIZED FORMAT (all new datasets MUST match this exactly)
+
+Confirmed by inspecting `data/reagents/adenine/` + `data/scores/adenine.parquet` on 2026-05-30.
+
+### 2a. Reagent files
+- Path: `data/reagents//_reagent_.smi` for component i = 0,1,2,…
+- Format: **whitespace-delimited, NO header**, one reagent per line: ` `
+ - e.g. `C1=CC(=C(N=C1)N)CO amidine_001`
+- `` must be a **stable canonical ID** (the product code is built from these).
+
+### 2b. Score / lookup parquet
+- Path: `data/scores/.parquet`
+- Columns: **`Product_Code` (str)`, `Scores` (f64)`** — exactly these two names.
+- `Product_Code` = `"_".join(reagent_name for each component in order)`
+ - e.g. `amidine_003_isocyanide_db_472_aldehyde_335`
+- This is the LookupEvaluator key. **CRITICAL:** the sampler keys on reagent *names*
+ joined by `_`; it never re-derives structure during search (verified in
+ `core/sampler.py:322` + `core/evaluators.py:159`). So the parquet `Product_Code` MUST
+ equal `_`.join of the reagent names in the `.smi` files, same component order. Any
+ mismatch silently yields all-NaN lookups.
+- Optimization mode: docking = **minimize** (FRED/HYBRID Chemgauss4, lower = better);
+ DEL read_count = **maximize**.
+
+### 2c. reactions.csv
+- Path: `data/reactions.csv`, header `rxn_id,SMARTS`.
+- Add one row keyed by the SMARTS lib key (see Local Norm #11 below). The SMARTS must
+ **parse under `AllChem.ReactionFromSmarts`** but — for a pure lookup benchmark — is never
+ applied during search (it only needs to be syntactically valid). Use a real SMARTS if the
+ library is also enumerated in-pipeline.
+
+### 2d. Triple-config registration (Local Norm #12 — edit ALL THREE or recovery silently breaks)
+For every new **docking (minimize)** library, add the lib_id to:
+1. `examples/_benchmark_config.py` → `MINIMIZE_LIBRARIES` (set, line ~25)
+2. `examples/run_posthoc_analysis.py` → `MINIMIZE_LIBRARIES` (set, line ~52)
+3. `examples/_benchmark_config.py` → `SCORE_TO_REAGENT_FOLDER` (line ~31) **if** folder name ≠ lib_id
+4. `scripts/stage_benchmark_data.sh` → `REAGENT_FOLDER` map (line ~42) **if** folder name ≠ lib_id
+
+Symptom of a miss: posthoc recovery ≈ 0 (lib treated as MAXIMIZE → recovery vs worst
+compounds), or a Slurm stage failure for missing reagent files.
+
+### 2e. Name-key resolution (Local Norm #11)
+`run_benchmark.py`: `folder_name = SCORE_TO_REAGENT_FOLDER.get(lib_id, lib_id)`;
+`name_key = folder_name if lib_id.endswith("_hybrid") else lib_id`. Reagent files at
+`data/reagents/{folder_name}/{name_key}_reagent_*.smi`; SMARTS key in reactions.csv =
+`name_key`. Do NOT add ad-hoc fallbacks; preserve the `endswith("_hybrid")` discriminator.
+
+---
+
+## 3. sEH — the centerpiece (full recipe)
+
+### 3a. Source & provenance (no license friction — our own lab's data)
+- Paper: Zhang et al., *J. Chem. Inf. Model.* 2023, 63(16):5120, DOI 10.1021/acs.jcim.3c00588
+ ("Building Block-Based Binding Predictions for DNA-Encoded Libraries"). Mobley co-author.
+ Open access (PMC10466377). Code/data: github.com/MobleyLab/DEL_analysis.
+- Local files (already on disk): `data/reagents/seh/DEL_seH/`
+ - `bb1_list.csv`, `bb2_list.csv`, `bb3_list.csv` (columns `SMILES,iso_SMILES`; 617/338/4529)
+ - `del_binders.txt` (columns `bb1,bb2,bb3,structure,read_count`; **116,666 binders**, read_count 81–4712)
+ - `del_nonbinders.csv` (1.6 GB, local only), `total_compounds.csv` (962 MB, 4.43M rows, local only)
+
+### 3b. Binder definition (the experimental ground truth)
+Binary binder vs non-binder: NGS read counts statistically ≠ 0 at 95% (Poisson). Min
+read_count for a binder = 81. We will overlay these as "experimental hits" to show TACTICS
+recovers them. **Binder labels live in `del_binders.txt`.**
+
+### 3c. Chemistry / SMARTS (RECOVERED — see `research/blueprints/seh_caix_smarts.md`)
+3-cycle amide-coupling DEL: central di-acid scaffold (bb2) + amine bb1 + amine/acid bb3,
+leftover acids methylamine-capped. The recovered multi-step `ReactionConfig` reproduces
+**97.4% of ground-truth products exactly**. The ~270 non-amide bb3 (sulfonyl-Cl /
+isocyanate / aryl-halide) are out of scope for the amide scheme (a deliberate, accepted
+drop). Blueprint has copy-pasteable `ReactionConfig` + per-position coverage + the
+reactant-template-order gotcha. **NOTE:** for a pure docking-lookup benchmark we may not
+even need to enumerate via TACTICS — we can enumerate products directly (RDKit) for docking,
+then score by Product_Code. SMARTS only needed if enumerating in-pipeline.
+
+### 3d. Binder-retention vs library-size (computed 2026-05-30 from del_binders.txt)
+Amide-compatible binders = **109,663 (94.0%** of 116,666; the 5.9% lost all have a
+non-amide bb3 — uniform across potency, only 51 of top-1000 lost).
+Reagent union of amide-compatible binders (canonical): **bb1=606, bb2=388, bb3=1111** →
+full Cartesian to contain ALL = ~261M (too big).
+
+**DECISION: 60M.** Recipe = keep all binder bb1 × bb2, keep **top-K bb3 by #binders**:
+
+| lib size | K bb3 | binders kept | % amide-compat | % all 116,666 |
+|---|---|---|---|---|
+| 40M | 170 | 99,753 | 91% | 85.5% |
+| **60M** | **~255** | **104,207** | **95%** | **89%** |
+| 80M | 340 | 106,160 | 97% | 91% |
+| 261M | 1111 | 109,663 | 100% | 94% |
+
+606×388×255 = **59.96M ≈ 60M, retains 95% of amide-compatible binders.** Can co-optimize
+(trim bb1/bb2 too) to squeeze a few % more at fixed size if desired.
+
+### 3e. ⚠️ OPEN RISK — name reconciliation (do this FIRST, blocks everything)
+The binder file's BB columns do NOT match the master reagent lists 1:1:
+- binders use **665 unique bb1** strings but `bb1_list.csv` has 617; **bb2: 666 vs 338**;
+ canonical: binder bb2 union = 388 but master = 338.
+This is canonicalization/deprotection-form drift (master lists are post-deprotection; binder
+columns may be a different form). **Implication:** some binder reagents are NOT in the master
+lists, so "include the binders" requires building the reagent set from the binder requirement
++ fill, NOT just subsetting the master lists. Step 0 = reconcile to one canonical ID scheme,
+then define library reagent sets = (amide-compatible binder-used reagents) ∪ (top master
+reagents to reach 60M). This guarantees binder inclusion and a consistent Product_Code space.
+
+### 3f. Receptor (CONFIRM before docking)
+No single dominant sEH docking structure. Recommended for OpenEye FRED/HYBRID (needs a holo
+structure with a drug-like inhibitor to define the design unit — apo 1S8O won't do):
+- **4OD0** (lead pick) — human sEH + urea inhibitor, canonical sEH pharmacophore (catalytic
+ Tyr/Tyr/Asp triad), clean single bound ligand. RCSB 4OD0.
+- alts: **3PDC** (used for SBVS of combinatorial libs vs sEH), **3I28** (lead-opt costructure),
+ **3WKE** (t-AUCB tool inhibitor).
+User has no in-house sEH receptor → use a PDB. **Pending: confirm 4OD0 preps cleanly** (pull
+ligand/pocket, build .oedu design unit). Mobley lab may have a preferred structure — ask.
+
+### 3g. Objective choice (design flag, doesn't change construction)
+- Docking score objective → TACTICS recovers top *docking* hits; binder overlay then measures
+ docking's enrichment, not TACTICS skill.
+- **DEL read_count objective → the sharpest "TACTICS finds experimental binders in few
+ iterations" claim, no docking needed.**
+- Recommend doing BOTH: dock for the docking-recovery benchmark; read_count variant for the
+ binder claim. (`maximize` mode for read_count.)
+
+---
+
+## 4. KinDEL (MAPK14 + DDR1) — license VERIFIED, FETCH not vendor
+
+- Verified by reading `raw.githubusercontent.com/insitro/kindel/main/LICENSE.md` (2026-05-30):
+ custom Insitro license **incorporating CC BY-NC-SA 4.0**. **"benchmarking" and "performance
+ evaluation" are explicitly permitted, for-profit or not.** Prohibits commercial use.
+ Grant is **non-transferable + ShareAlike + no patent license**.
+- **Plan (user-approved):** academic paper, open-source code, data **NOT in package
+ distribution**. Datasets go in the separate manuscript-reproduction repo, fetched via a
+ download script pointing at the public S3 `s3://kin-del-2024/data` + citation (same pattern
+ as `download_scores.py`). **Do NOT re-host their data** (non-transferable/SA) — fetch from
+ origin. No commercial use.
+- Data ships per-cycle reacted synthons (378/1128/191 → 81M) + prepared receptor PDBs
+ (MAPK14: 3KQ7/3S3I/5WJJ/5XYY/6SFI; DDR1: 6FEX) + docked poses + off-DNA Kd held-out sets.
+ Enumerable via fragment-stitch on `[Fr]` markers or sample the provided enumerated parquet;
+ subset 81M → 1–10M.
+- Manuscript angle (user): "deliver benchmarking docking datasets for developing active-learning
+ methods."
+
+---
+
+## 5. Recovered sub-agent artifacts (don't redo)
+- `research/blueprints/seh_caix_smarts.md` — full sEH (97.4%) + CAIX SMARTS recipes, coverage,
+ ReactionConfig snippets, gotchas. (The SMARTS sub-agent hit the rate limit and returned an
+ empty final message, but its report was written to disk — recovered.)
+- `outputs/debug/seh_smarts/` — scratch JSONs/SMI (gitignored). `FINAL_SUMMARY.txt` has the
+ headline numbers.
+- **CAIX is too small** (216K products, below the 1M floor) — not a benchmark candidate; skip.
+- Sulfonamide / carbonic-anhydrase candidate **DROPPED** (catalytic-Zn docking too noisy).
+
+---
+
+## 6. ORDERED NEXT STEPS (resume here)
+
+0. **(sEH) Reconcile names** — canonicalize bb1/bb2/bb3 across `del_binders.txt` ↔ master
+ `bb*_list.csv`; pick a stable reagent-ID scheme; resolve the 665-vs-617 / 388-vs-338 drift.
+ Output: a mapping {canonical reagent → stable ID} per component.
+1. **(sEH) Select the 60M reagent set** — all binder bb1×bb2 (amide-compatible) + top-~255 bb3
+ by binder usage, ∪ any binder-required reagents missing from master. Verify final
+ |bb1|×|bb2|×|bb3| ≈ 60M and that 95% of amide-compatible binders are inside.
+2. **(sEH) Write standardized reagent files** `data/reagents/seh/seh_reagent_{0,1,2}.smi`
+ (SMILES + stable ID, no header). Emit a `binder_map` (Product_Code → read_count) for the
+ recovery overlay.
+3. **(sEH) Confirm receptor 4OD0** (or lab structure) → build `.oedu` design unit.
+4. **(sEH) Enumerate + dock on HPC3** (omega + FRED/HYBRID) → `data/scores/seh.parquet`
+ (`Product_Code`, `Scores`), Product_Code matching the reagent IDs exactly.
+5. **(sEH) Register** in reactions.csv + triple-config (§2c–2d). Add to benchmark runner.
+6. **(sEH) Run benchmark + overlay** — report docking recovery AND experimental-binder hit
+ rate (and/or the read_count-objective variant).
+7. **(KinDEL)** stand up download script + enumerable subset (1–10M) in the separate repo.
+8. **(JNK3-ROCS, lowest priority)** dock 1×2-comp + 1×3-comp ROCS libs vs JNK3 2ZDT.
+
+**Do not** kick off enumeration/docking/HPC jobs without explicit user go-ahead.
+
+---
+
+## 7. This-session figure work (already done, for completeness)
+`tutorials/manuscript_plots_docking.py` §6 budget-sensitivity panel: bottom row converted to a
+per-library **stacked decomposition** (per-chemistry segments sum to the bin mean), bar colors
+moved to the **unused Tableau-10 members** (adenine green #59A14F, quinazoline pink #FF9DA7,
+amide brown #9C755F, thrombin gold #EDC949) so they don't collide with the method-line colors;
+legend title dropped (was the white-gap cause). Standalone per-library-exemplars figure removed
+per user. Compiles; pixel-verified.
diff --git a/docs/handoffs/session-archive-2026-03.md b/docs/handoffs/session-archive-2026-03.md
new file mode 100644
index 0000000..6669dcf
--- /dev/null
+++ b/docs/handoffs/session-archive-2026-03.md
@@ -0,0 +1,22 @@
+# Session Log Archive — March 2026
+
+Archived from `research/STATUS.md` on 2026-04-16 to keep the session log under 20 entries.
+
+---
+
+- **2026-03-11**: _Template created. No experiments run yet._
+- **2026-03-14**: Implemented TT-TS and Bayesian additive decomposition. Ran full benchmark (65,430 trials, 21 libraries). TT-TS significantly outperforms RWS on 6/21 libraries (+0.6 pts overall). Additive decomposition adds zero value — removed from codebase. Ran manuscript benchmark (87,240 trials, 4 methods, 20 libraries). Key finding: GMIC criticality-weighted rotation is the primary mechanism (+4.3 pts on 2-comp, +0.9 on 3-comp over Legacy RWS), breaking the 2-component plateau. TT-TS adds modest gains especially on 3-comp. Next: manuscript preparation.
+- **2026-03-15**: Ran top-N sensitivity benchmark (697,600 rows, 8 thresholds 25–250). TACTICS advantage grows monotonically at wider top-N on all 20/20 libraries — criticality is more valuable for broad hit landscape than needle-in-haystack. Key discovery: Legacy RWS crosses below Legacy TS on 2-comp at ~top-150 (8/10 libraries), showing unguided thermal cycling is counterproductive at wider windows. Created marimo notebook for manuscript plots. Next: manuscript figure generation.
+- **2026-03-16**: Investigated TT-TS underperformance on thrombin (29.6:1 imbalance). Parameter sweep showed heated_scale is the sole cause: hs=1.0 recovers 78.6 (p=0.0005 vs Legacy TS), hs=1.5 = 73.5 (ns). Implemented bidirectional per-component adaptive heated_scale via EMA disagreement tracking in TopTwoSelection. Validated: adaptive TT-TS = 77.5 (p=0.0002 vs Legacy TS), matching RWS = 77.3. Fixed Legacy RWS budget bug on thrombin (negative nsearch from hardcoded 2%). Added TACTICS Balanced-Greedy as 5th benchmark method to isolate framework gains. Adopted naming convention: "Framework Warmup-Selection (modifiers)". Manuscript benchmark ready for cluster re-run with 5 methods.
+- **2026-03-16 (session 2)**: Analyzed completed 5-method manuscript benchmark (114,450 trials, 21 libraries). Adaptive TT-TS confirmed no regression: mean delta -0.02 vs fixed hs=1.5 across 20 ETS libraries, thrombin +2.1. Balanced warmup isolated: +1.5 on 2-comp (sig 10/11), +0.4 on 3-comp (sig 1/10) via greedy alone. Key insight: Legacy RWS is *worst* method on thrombin (68.8, below greedy 73.2) — thermal cycling is counterproductive on imbalanced libraries. Both TACTICS TT-TS (77.5) and RWS (78.2) still use Enhanced warmup, inheriting a 30:1 posterior observation imbalance. New hypothesis: Balanced warmup + TT-TS/RWS should further improve thrombin by eliminating this artifact.
+- **2026-03-16 (session 3)**: Ran thrombin warmup comparison (7 methods × 20 replicates, 29,220 budget). **Hypothesis rejected**: Balanced warmup significantly hurts — Balanced-TT-TS=72.5 vs Enhanced-TT-TS=78.0 (-5.5, p<0.0001), Balanced-RWS=73.5 vs Enhanced-RWS=78.5 (-5.0, p<0.0001). Also ran Enhanced-Greedy=73.2 (≈Balanced-Greedy=74.4, ns). Key mechanistic insight: Enhanced warmup's over-sampling of the small component pre-solves acid ranking; GMIC detects the differential posterior quality and redirects search to amines. Without GMIC the asymmetry is inert; with Legacy RWS's round-robin cycling it's harmful. The three-way interaction (warmup × information asymmetry × adaptive rotation) is the core finding. Manuscript methods confirmed as optimal. New open question: direct uncertainty-driven diversity sampling as alternative to thermal cycling (future work, not this manuscript).
+- **2026-03-17**: Spun out `library-diagnostics` worktree to investigate library diagnostic framework for follow-up manuscript. Candidate axes: Free-Wilson additivity, gap hardness, effective reagent rank, NK ruggedness. Goal: prescriptive diagnostics that help users design libraries amenable to TACTICS screening.
+- **2026-03-19**: Created full-results benchmark infrastructure (`_full_results_worker.py`, `run_full_benchmark.py`) that saves raw search data for post-hoc analysis. Ran 218,100 trials (21 libraries, 5 methods, 20 replicates) on HPC3 with 12 shards. All trials completed, zero errors, ~32 GB output. Merge job OOM-killed at 64G due to unnecessary dedup — fixed merge to skip dedup (shards disjoint by construction), bumped to 96G+16 CPUs. Recorded future manuscript pipeline (Papers 2a-4) in STATUS.md. Spun out `reagent-expansion-DrugEx` worktree for DrugEx integration / vendor mapping work (collaboration with Leiden University).
+- **2026-03-21**: Added aggregated Tukey HSD pairwise CI plot to manuscript_plots.py notebook. Discovered pooling ROCS and docking in one Tukey HSD inflates variance (adenine bimodality sd=27.6 dominates) — split into separate ROCS (20 libs, 2,180 query-level pairs) and docking (2 libs, 40 replicate-level obs) analyses. Traced TT-TS adenine rep 15 failure to amidine_003 under-exploration (0.6% search budget vs 4–15% in successful reps, caused by uninformative warmup posterior). Planned docking benchmark expansion: amide 10M + quinazoline 20M from Walters repo, plus ideally 1 additional target protein. Quinazoline enables cross-scoring-function comparison with ROCS niementowski. Next: prepare docking datasets, then manuscript writing.
+- **2026-03-21 (session 2)**: Built docking benchmark infrastructure for expanded libraries targeting JNK3 (PDB 2ZDT, same receptor as Klarich 2024). Selected diverse reagent subsets using BitBIRCH clustering + seed-anchored MaxMin diversity picking (`experiments/reagent_selection/select_reagents.py`). Amide library: 1,000 acids × 10,000 amines = 10M (reaction SMARTS from Walters repo, not rxn101). Quinazoline library: 376 aminobenzoic × 300 acids × 177 amines = 20M (niementowski 100-per-component reagents as seeds, expanded via BitBIRCH). Both libraries enumerated on HPC3 (10M amide 100% success, 20M quinazoline 100% success). Test docking of batch_00000 successful for both: amide 9,980 docked in 8 min, quinazoline 11,155 in 5 min (FRED Chemgauss4, 48 MPI cores, 200 conformers). Bumped to 400 conformers for production runs. Docking target is JNK3 (kinase), giving three protein families (thrombin=protease, adenine=GPCR A2B, JNK3=kinase). Scripts: `data/docking/{enumerate_library.py, dock_batch.sub, submit_docking.sh}`. Next: submit full docking arrays, then process scores into TACTICS benchmark format.
+- **2026-03-19 (session 2)**: Added adenine docking library (29×688×294, 4.9M scored products, FRED Chemgauss4, 17% dock failures). Standardized all data formats: thrombin and adenine converted to `{lib_id}.parquet` with `[Product_Code, Scores]`, reagent files renamed to `{lib_id}_reagent_{i}.smi`, reactions.csv updated with thrombin + adenine SMARTS. Refactored `run_full_benchmark.py` to auto-discover libraries from data directory (eliminated hardcoded library sets and thrombin special-casing). Ran adenine benchmark (100 trials, 20 reps, 5 methods). Created `run_posthoc_analysis.py` with rigorous statistical framework (ANOVA + Tukey HSD + Friedman + Nemenyi + Cohen's d) following Ash et al. 2025. Debugged two post-hoc issues: (1) merged + shard files both loaded causing double-counting — fixed by preferring merged file; (2) NaN scores from docking failures sorted to top of results by Polars — fixed by filtering NaN before ranking. Key finding: TT-TS is exceptionally robust to docking failures (93.9 on adenine, 19/20 reps at 97), while RWS shows bimodal behavior and greedy plateaus at ~32. The uniform distribution of failures across reagents preserves SAR signal — concentrated failures would create blind spots.
+- **2026-03-24 to 2026-03-26**: Deep analysis of Riley (2024) posterior separation methodology and connection to TACTICS results on both docking and ROCS. Major manuscript_plots.py overhaul (pushed to GitHub): (1) data path moved to `outputs/manuscript_analysis/post_hoc`, (2) posterior separation plots: continuous colormap by hit count, z-scored axes, data-driven colorbar, (3) multi-top-N Riley-style grid plots (top-25–250) for all 4 docking + 4 ROCS libraries, (4) gain decomposition split into ROCS (baseline=Legacy RWS) and Docking (baseline=Legacy Greedy) with scoring-type-appropriate baselines, (5) per-library grouped difference bars for TT-TS and RWS(GMIC) vs appropriate legacy baseline, (6) docking top-N sensitivity fixed to use Legacy Greedy baseline. Key findings: SAR additivity governs greedy reachability (docking quinazoline z≈-2.0 vs adenine z≈-0.75); ROCS SAR erodes at wider top-N as first series exhausts; Legacy RWS single-component exploration bottleneck explains 2-comp crossover. Detailed mechanism walkthrough of GMIC (3 layers: weighted rotation, divergence gate, CATS multiplier) and TT-TS (posterior widening/contraction, two-sample disagreement, adaptive EMA). Established correct terminology: TT-TS uses "posterior widening" and "contraction" not "heating/cooling" — no Boltzmann temperature involved. Created 4 Heptabase cards: (1) docking SAR additivity, (2) ROCS SAR erosion + RWS bottleneck, (3) RWS GMIC mechanism walkthrough, (4) TT-TS GMIC mechanism walkthrough. Next: run instrumented searches on HPC3 with track_diagnostics=True to generate per-iteration GMIC/disagreement data for mechanism visualization plots.
+- **2026-03-26 (session 2)**: Fixed TT-TS diagnostics schema mismatch — `disagreement_ema`, `heated_scale`, `effective_scale` were silently dropped when TT-TS records were coerced into GMIC schema. Added `_TTTS_DIAGNOSTICS_SCHEMA`, detection logic, `criticality` alias, 2 analysis functions (`compute_disagreement_convergence`, `compute_scale_adaptation`), 3 plot functions, 13 new tests (43 total). Built diagnostic benchmark infrastructure: `run_diagnostic_benchmark.py` (110 trials: adenine, quinazoline, rxn206, mannich, orru × RWS/TT-TS × 5 reps), ROCS query selection at p10/p50/p90 of advantage distribution, and Slurm job file using `stage_benchmark_data.sh` convention. Selected libraries span the difficulty spectrum: rxn206 (+7.6 advantage), mannich (+3.9), orru (-0.4, negative control). New active hypothesis: mechanism activation scales with query difficulty. Next: push to HPC3, submit `sbatch examples/run_diagnostic_benchmark.sub`, then build visualization notebook from results.
+- **2026-03-27**: Analyzed diagnostic benchmark results (110 trials, all 5 libraries, TT-TS diagnostics fully captured). Original hypothesis partially rejected: mechanism does NOT show sustained activation on hard queries — it fires early and quiets down. Discovered adaptive heated_scale runaway on balanced 3-comp libraries: one component converges fast → EMA drops below 0.3 → uncapped growth inflates scale to 2–5× → trapped in neutral zone. Consistent across all replicates (structural, not stochastic). Considered 3 fixes: (1) raise low_threshold — unsafe, suppresses beneficial inflation on adenine, (2) max_growth_per_step=0.1 — safe, caps runaway without preventing beneficial growth, (3) GMIC convergence gate — unsafe, can't distinguish additive from non-additive SAR. Implemented fix 2 as new default. Added `--methods` filter to `run_full_benchmark.py`. Prepared TT-TS-only validation benchmark (24 libs, 20 reps, 480 trials). Next: submit `sbatch examples/run_ttts_validation.sub`, compare against existing baseline.
+- **2026-03-27 (session 2)**: Built unified benchmark infrastructure. Replaced 8+ copy-pasted benchmark scripts with single `examples/run_benchmark.py` (run/estimate/merge subcommands) + `_benchmark_config.py` + `_benchmark_slurm.py`. Created `/benchmark` skill. Discovered TT-TS validation job (50251973) has insufficient wall time.
+- **2026-03-22**: Completed brute-force docking for amide (10M) and quinazoline (20M) on HPC3. Converted scores to parquet. Ran 5-method benchmark (200 trials, 20 reps). Post-hoc analysis complete (24 libraries, 2.18M recovery rows).
diff --git a/docs/handoffs/session-archive-2026-04.md b/docs/handoffs/session-archive-2026-04.md
new file mode 100644
index 0000000..0655e6e
--- /dev/null
+++ b/docs/handoffs/session-archive-2026-04.md
@@ -0,0 +1,18 @@
+# Session Archive — April 2026
+
+Archived from `research/STATUS.md` on 2026-05-22 when the active session log
+exceeded 20 entries. Restore-style: append these back to STATUS Session Log
+if specific narrative context is needed for an investigation.
+
+- **2026-04-02**: Investigated multi-component heating hypothesis. Diagnostic data showed 2+ components below GMIC 1.0 simultaneously is rare on 3-comp (3% of query-phases). More importantly, 2-comp libraries where BOTH components have GMIC < 1.0 still benefit strongly from single-component heating (+7.6 on rxn206) — the explore/exploit contrast itself drives gains. Deferred continuous temperature gradient to follow-up study. Built manuscript-quality diagnostic plot functions (`plot_rws_diagnostic`, `plot_ttts_diagnostic`, `plot_min_gmic_vs_advantage`) in `diagnostic_plots.py` with 95% CI bands across replicates, rolling-averaged temperature, divergence gate background shading, and scalable colour palette. Created marimo notebook `tutorials/diagnostic_benchmark_plots.py` rendering all 21 diagnostic figures. Cleared active hypothesis — manuscript ready for writing.
+- **2026-04-07 to 2026-04-09**: Ran three full benchmarks: (1) EMA CATS + rotation fix (batch=100), (2) batch=50. EMA fix confirmed: aggregate recovery unchanged, TT-TS reliability improved on hard docking (adenine std 13.9→4.0%, thrombin 2.2→1.4%). batch=50 is a negative result — GMIC stabilizes by cycle 0-13 regardless of batch size, so more frequent rotations add nothing. TT-TS disagreement EMA stays saturated (>0.8) on most components — thresholds may need recalibration. Fixed benchmark infrastructure: merge time limit (1h→8h), diagnostic concat schema mismatch (`how="diagonal"`), posthoc GPU→CPU, moved 21 .sub files to jobs/, created MANIFEST.md. Diagnostic benchmark re-run with correct `cats_ema_decay=0.1`. batch=100 remains the default. Next: revert batch_size to 100, run diagnostic plots with batch=50 data for comparison, then manuscript writing.
+- **2026-04-04/05**: Two-session deep dive into GMIC mechanism on 3-comp ROCS libraries. (1) Decomposed GMIC signal/noise: noise denominator inflated 2-12× by unexplored reagents (recorded as Paper 3 open question + Heptabase card). (2) CATS multiplier is effectively noise on balanced 3-comp: median=1.000, contributes ~10% of temperature differential, 13× smaller Boltzmann shift than rotation alone. (3) Identified rotation-evaluation ordering bug: rotation for N+1 uses N-1 posteriors (100 evals stale). (4) Batch granularity suppresses CATS: 195 update opportunities vs 19,500 at batch=1, divergence gate fires 33% in early search. (5) Mechanism decomposition: GMIC rotation produces +2.2 net over legacy cycling on mannich (positive 79% of queries), -0.8 on orru (uniformly easy SAR). TACTICS advantage is net-positive across the query distribution: gains on hard queries, neutral on easy. New active hypothesis: fixing rotation ordering + EMA-smoothed CATS should give modest-to-medium improvement.
+- **2026-04-04**: Decomposed GMIC signal/noise dynamics on all 10 three-component ROCS libraries using full benchmark diagnostics (1.27M rows per library, 109 queries × 20 replicates × 195 cycles). Key finding: GMIC noise denominator is inflated 2-12× above expected (uniform observation) levels because ~85-90% of reagents remain at warmup-level variance while directed search concentrates observations on ~10-15%. This is a limitation of GMIC: it underestimates SNR for the exploited subpopulation. Per-query trajectory classification: signal crash magnitude predicts GMIC direction (r=-0.595, mannich). DOWN trajectories = false positive SAR from warmup (92% signal crash, self-correcting). UP trajectories = real SAR (35-47% signal retained). Mannich (Δ=+2.5): 171 UP, 45 DOWN, 111 FLAT → differential GMIC drives advantage. Orru (Δ=-0.5): 268 UP, 26 DOWN → uniformly easy SAR, no differential. Created Heptabase card with proposed observation-weighted GMIC variant and reagent usage fraction diagnostic plot. Recorded as Paper 3 open question; Paper 1 discussion point (TACTICS outperforms legacy even with conservative estimates).
+- **2026-04-01**: Deep dive into GMIC-RWS mechanism using diagnostic benchmark data (110 trials, 5 libraries, 11 queries). **Key findings**: (1) GMIC direction is query-specific — different query molecules on the same library produce opposite GMIC directions (rxn206 query_008 vs query_090). The "near-uniform" rotation probabilities are an artifact of averaging across queries with opposing directions. (2) 5/9 ROCS queries show GMIC redirection mid-search, tracking the transition from first SAR series to second-tier reagents. (3) The CATS multiplier provides meaningful modulation even on balanced libraries (43-56% of iterations active, range [0.025, 0.20] effective temp). The alpha/beta ratio defines both the base asymmetry AND the CATS dynamic range — they are one integrated mechanism. (4) min_GMIC (weakest component) predicts TACTICS advantage: <1.0 → large gains, >1.5 → negligible. (5) The smaller 3-comp advantage is due to easier SAR (higher min_GMIC), not mechanism malfunction or the heated_scale "runaway." Retired the max_growth_per_step hypothesis, disabled the growth cap (default=None). Created 2 Heptabase cards: mechanism walkthrough and diagnostics analysis. New hypothesis: multi-component heating to address the structural single-component constraint on 3+ comp libraries.
+- **2026-04-14 to 2026-04-16**: Presentation preparation — deep conceptual analysis of TT-TS mechanism for talk "TACTICS: A well-informed and efficient exploration of large chemical spaces." Reviewed diagnostic trajectories for adenine and quinazoline (GMIC, disagreement EMA, heated_scale per component per replicate). Key framing insights: (1) TT-TS is uncertainty-driven exploration vs RWS score-driven vs Legacy RWS diversity-driven; (2) cooled components are near-greedy, creating informative combinations that test uncertain reagents with strong partners; (3) adenine isocyanide GMIC=0.000 during search vs 0.24 from brute force — observation density bottleneck at 688 reagents; (4) key isocyanide (40% of top-100 hits) found indirectly through co-occurrence with amidine_003, not direct targeting; (5) recovery decrease at wider top-N is combination coverage bottleneck, not mechanism failure; (6) batch size invariance implies larger batches for HPC parallelization with no quality loss. No new experiments or code changes.
+- **2026-04-15 to 2026-04-16**: Manuscript notebook overhaul and statistical validation. (1) Fixed `manuscript_plots.py` data path to use `full_benchmark_ema_cats`. Added method multiselect, baseline selector for bar plots, SAR library dropdown. Replaced hardcoded per-library bar plots with generic cell. (2) Created `tutorials/interactive_sar_explorer.py` (marimo + molplotly) for hover-to-structure SAR visualization. (3) Validated statistical framework against Zhao et al. 2025 (PatWalters/TS_2025 repo): Zhao used paired t-tests on query means (n=109); our ANOVA + Tukey HSD is more rigorous with FWER control. TT-TS significantly outperforms Legacy Greedy on 20/20 ROCS libraries, Legacy RWS on 14/20. Never worse on any library. (4) Quinazoline -0.75pp shift confirmed as replicate noise (p=0.208) via diagnostic comparison — identical GMIC/rotation between pre/post fix. Manuscript uses pre-fix quinazoline data for cleaner presentation. (5) Adenine docking failure analysis: 16.3% failures distributed uniformly (80-92% per active reagent), 34 reagents at 0% from enumeration failures. Added histogram to manuscript notebook. (6) Confirmed Legacy RWS uses batch=100 internally (hardcoded `num_per_cycle=100` in `rws_sampling.py`), despite `batch_size=1` in benchmark config. Next: continue manuscript writing.
+- **2026-04-24**: Visualization tooling only — no research findings. Added `plot_reagent_usage_action_panel` to `src/TACTICS/library_analysis/diagnostic_plots.py`: per-batch grouped bar chart showing fraction of each component's reagents sampled, split into revisit (solid) + new (hatched) segments, with heated-component encoding via opacity (1.0 vs 0.30) and auto-trim of trailing exploitation-only batches annotated "algorithm converged after X batches". Added `_parse_name` (greedy prefix matching against reagent-ID sets to handle IDs containing underscores like `amidine_034_isocyanide_db_511_aldehyde_305`), `_infer_component_name` (returns "Amidine"/"Isocyanide"/"Aldehyde" from common-prefix detection on adenine), and per-library chemistry labels in `tutorials/diagnostic_benchmark_plots.py` derived from `data/reactions.csv` SMARTS (mannich: Amine/Aldehyde/Ketone; orru: Aldehyde/Cyanoester/Amine; rxn206: Alcohol/Sulfonamide; quinazoline: Aminobenzoic acid/Amine/Carboxylic acid). Supports `normalize` modes `library`/`batch`/`share` for imbalanced libraries. Replaces the prior confusing dual-axis GMIC+temperature+disagreement plots for communicating the mechanism in the manuscript. Next: user to review rendered figures in the marimo notebook and select which go into the manuscript.
+- **2026-04-27**: Paper 1 manuscript framing session — mechanism analysis only, no code or experiments. (1) Computed empirical CATS multiplier alignment frequency from diagnostic benchmark (98,730 component-rows): 52.7% fires, 44.0% no-op due to misalignment, 3.2% diversity gate closed. Per-library rotation success ranges 42–62%, late-search drift on balanced libraries (orru 77%→46%, quinazoline 62%→35%). Confirmed previous user explanation of "well-separated means absorb high T" was incorrect — Boltzmann uses standardized z-scores, so absolute mean separation is irrelevant. The actual mechanism preventing the diversity trap is asymmetric: GMIC-biased rotation, asymmetric CATS amplification when aligned, and absence of Legacy's adaptive α inflation. Appended empirical alignment section to existing card 311c0bc1. (2) Decomposed the architectural floor of TACTICS RWS into four layers (Enhanced warmup, asymmetric α=0.10/β=0.05 base, fixed α with no adaptive inflation, stochastic GMIC-weighted rotation) and clarified that CATS was designed as a directional replacement for Legacy's duplicate-feedback adaptive α — both modulate temperature, but on different signals (GMIC asymmetry vs duplicate rate) and with different gating (one-way ratchet for Legacy, two-way bidirectional for CATS). On libraries with no GMIC asymmetry (Regime III), CATS correctly remains silent and α stays fixed at 0.10. (3) Built three-regime library taxonomy for Paper 1 Discussion section organized around mechanism layer dominance: Regime I (imbalanced, docking only — sub-Ia non-additive adenine, sub-Ib mean-separable thrombin/amide), Regime II (balanced multi-cluster ROCS, rotation dominant), Regime III (balanced mean-separable quinazoline + 2-comp ROCS at narrow top-N, architectural floor dominant). Operational definition: imbalanced libraries have max:min reagent count ratio ≥ 10. Initial taxonomy incorrectly classified amide (9.7:1) as balanced and conflated ROCS with docking in Regime III; corrected by user feedback to use 10:1 threshold and segregate score types. Decision: taxonomy belongs in Discussion (emergent from data), Methods describes only metrics computed. (4) Extended analysis to TT-TS: regime taxonomy applies but TT-TS dominates RWS specifically and only in Regime Ia (adenine +17.55 pts; ≤±1 pt on every other library). The mechanism difference is the reagent-level selection primitive — RWS uses μ-based Boltzmann, TT-TS uses σ-based two-sample disagreement — and TT-TS's σ-based primitive is robust to non-additive SAR specifically because it does not rely on means being predictive. (5) Tightened the Riley framework linkage: the additivity axis in the regime taxonomy is *measured* via per-component hit-reagent posterior z-scores (μ separation) and per-reagent σ asymmetry, not asserted from Riley's blog post. Riley motivated the measurement; the measurement does the regime classification work. (6) Identified Paper 1 Discussion argument that ROCS late-top-N behavior is the structural analog of docking non-additivity: ROCS SAR is additive within the first chemotype cluster but non-additive across clusters; Legacy RWS's symmetric round-robin cannot sustain coordinated multi-component exploration into the second cluster (single-component bottleneck), while TACTICS GMIC-directed rotation tracks the SAR shift. Wrote four Heptabase cards total: (a) "TACTICS RWS — Architectural Floor and CATS-vs-Adaptive-α Design", (b) "TACTICS RWS — Three-Regime Library Taxonomy for Paper 1", (c) "Posterior Separation Analysis — Operationalizing Riley's TS-on-Docking Argument" (methods card), (d) "ROCS Multi-Cluster SAR and the Hidden Non-Additivity at Wider Top-N" (discussion card). Benchmark coverage gaps flagged: no imbalanced ROCS libraries, single non-additive docking exemplar (adenine), only one balanced docking library (quinazoline). (7) Late-session refinements appended to two cards: (b) Three-Regime Taxonomy gained a "Sample-Efficiency vs Mechanism Distinction" section making explicit that Regime Ib admits eventual greedy convergence (sample-efficiency story) while Regime Ia does not (mechanism story; mean-based selection has no path to amidine_003 at any budget); decision was that this first-principles argument is sufficient for Paper 1 — recovery-vs-budget curves are not required because evaluation budget is the practical limiting resource in real combinatorial chemistry. (c) Posterior Separation methods card gained an "Observation Density Differential" mechanistic rationale for the 10:1 imbalance threshold: ratio = 10 produces ~3.2× per-reagent posterior std differential, which sustains GMIC asymmetry through typical search budgets and matches the empirical phase change in Legacy RWS damage. Resume next session: draft actual Methods and Discussion paragraphs using these four cards as source.
+- **2026-04-16 (session 2)**: Deep analytical session on TT-TS mechanism across top-N thresholds on adenine, and correction of the "92% GMIC gain" figure. (1) Decomposed TT-TS recovery on adenine into two compartments: amidine_003 (99.4% → 79.1% at top-10000) and other 26 amidines (~88% plateau, matching Greedy). Permanent +12pp advantage at top-10000 comes entirely from amidine_003; TT-TS converges to greedy-like behavior on mean-separable reagents. Saved as Heptabase card. (2) Traced iso_db_472 discovery on adenine: pair-level non-additive SAR (44/46 top hits paired with a003), GMIC for isocyanide stays at 0.000 throughout, discovery proceeds via Bayesian posterior-level updating in cooled-draw sampling — below GMIC's component-level resolution. (3) Derived GMIC observation-density threshold: fails when reagent count > ~5-10% of budget due to noise denominator inflation (2-12×). (4) Corrected "92%" claim: applies to 2-comp ROCS only. Current 24-library decomposition: 89.7% / 70.7% / 96.5% (RWS broken) / 68.5% for 2c-ROCS / 3c-ROCS / 2c-docking / 3c-docking. On 3-comp docking TT-TS contributes +8.22 pts beyond GMIC. (5) Validated Legacy method preference: RWS wins 10/10 on 3-comp ROCS (multi-cluster SAR); Greedy wins 2/2 on 2-comp docking (mean-separable SAR). (6) Cleared stale Active Hypothesis section; identified three candidates for future hypotheses (EMA threshold recalibration, reagent-level diversity sampling, observation-weighted GMIC). 3 Heptabase cards created. No code changes; purely analytical reinterpretation of benchmark data.
+- **2026-04-30**: Two adenine docking-data extensions were brought into the worktree from prior unrecorded work. (1) **FRED amidine_035 merge — confirmed clean.** Verified `data/scores/adenine.parquet` post-merge state: 5,083,598 rows (+174,001 amidine_035), 28 unique amidines (was 27), score range -20.89/21.78 mean -13.10 no nulls. The two amidines absent from the data (`amidine_022`, `amidine_029`) are confirmed not covered by the adenine reaction SMARTS (carried over from the original 2-amidine SMARTS gap). amidine_035 yield ~86%, matching the established 14–17% omega2/FRED failure rate. New Established Result entry added with linked Heptabase card `2ae0cadc-524e-4f11-8846-e886765da0df`. The benchmark rerun job (`jobs/adenine_rerun_amidine_035.sub`) is staged but not yet submitted; existing recovery numbers in lines 30/47/51 reflect pre-merge data. (2) **HYBRID (a2a-lenselink) campaign — 543/580 batches docked.** Second adenine docking campaign targets the Lenselink A2A receptor (`data/docking/adenine/a2ar-lenselink.oedu`), output to `data/docking/adenine/docked_hybrid/` and intended to merge to `data/scores/adenine_hybrid.parquet`. Score files contain real Chemgauss4 docking scores (verified by content inspection of batches 0, 100, 300, 500). 37 batches failed with omega2/OpenMPI orterun crashes — script's `omega_status` check caught the non-zero exit, so HYBRID never ran for those batches and they left 20-byte empty conformer-file stubs in `conformers_hybrid/`. Failures are batch-level (node/MPI instability), distinct from the per-product 17% omega2-flipper failure rate that will appear after parquet merge. `resubmit_full_hybrid.sh` only completed one round (job 51565080); the script supports up to 10 rounds and is not currently running. Next: re-run resubmit script, then `merge_full_hybrid.py` once 580/580 complete.
+- **2026-04-30 (session 2)**: Adenine post-merge benchmark + manuscript framing. (1) **Adenine post-merge recovery computed.** 100 trials, 5 methods × 20 reps × 1 query, 0 errors, ~5 min runtime (`outputs/adenine_rerun_amidine_035/`). Top-100 results: TACTICS RWS = 94.5 ± 2.1 (now best, reliable, no bimodal failures); TT-TS = 87.3 ± 16.3 (3/20 bimodal failures with amidine_003 under-exploration of 549 evals vs 11,500 in successful reps); Legacy RWS = 75.8 ± 19.8 (up from pre-merge 40.1, no longer broken); Greedy methods ~46. **Role flip vs pre-merge**: RWS now beats TT-TS on adenine because amidine_035 contributes 36/100 hits with informative posterior mean (-13.10), creating a μ-based rescue path that GMIC rotation reaches reliably. amidine_003 still 49/100 but its value is reduced share; TT-TS warmup-bimodality (STATUS line 51 mechanism) now fires more often (3/20) because 28 amidines thin warmup observations. Mechanism unchanged; library shifted. (2) **Manuscript framing — 4 Heptabase cards drafted** following close reading of Klarich 2024 (full 14-page paper) and Zhao 2025 (full 10-page paper). Cards: `[RESEARCH GAP] Klarich 2024 — Open Questions` (`402f94ee-cd57-41ef-977d-9e7d8c0fa27b`), `[RESEARCH GAP] Zhao 2025 — Open Questions and Extension Opportunities` (`3cc58e43-6710-4902-9b85-792202da9615`), `[RESEARCH GAP] Systematic Gaps in the Combinatorial-TS Literature` (`e8723de0-72c5-4607-89c3-cff89d1fdb29`), `[OBJECTIVE] TACTICS Manuscript — Aims and Contribution Statement` (`9471bb1b-b04c-4f2c-bcc8-42f53c294e81`). User-decided framing: TT-TS positioned as TACTICS-conceived enhancement (uncertainty-driven σ-primitive) parallel in spirit to Zhao's RWS-as-enhancement-to-TS; manuscript uses post-merge adenine canonically with Discussion-section sensitivity analysis (remove amidine_035 → falls into pure Regime Ia where TT-TS is structurally required); per-library best-Legacy baseline (Greedy or RWS, whichever performs better per library) for the most conservative TACTICS-vs-prior-art comparison; regime taxonomy + posterior separation analysis framed as post-hoc mechanistic characterization extending Riley 2024, **not** as prescriptive library design tools (require ground truth to compute); stopping-criteria flagged as planned study objective resolvable from existing benchmark data. Klarich-explicit gaps captured: theoretical-framework mismatch (TS regret vs tail-finding) directly motivates TT-TS; random-warmup-with-product-duplication motivated Enhanced Warmup; batch-mode TS open question already resolved (STATUS line 36, 110); single dominant building block dilemma (Klarich Table 2: 92/100 in one R1) tied to TT-TS's σ-primitive contribution. Zhao-internal gaps captured within ROCS scope: 3-comp mechanism opacity, signal-blind round-robin, opaque α/β adaptation, library diversity effect (Fig. 5). Reframed as extension opportunities, not flaws. Resume next session: integrate post-merge adenine into full benchmark (`outputs/full_benchmark_ema_cats/`), re-run post-hoc analysis with per-library best-Legacy baseline, then begin drafting manuscript Methods + Discussion paragraphs using the four cards as source.
diff --git a/docs/handoffs/session-archive-2026-05.md b/docs/handoffs/session-archive-2026-05.md
new file mode 100644
index 0000000..4b2b7e8
--- /dev/null
+++ b/docs/handoffs/session-archive-2026-05.md
@@ -0,0 +1,8 @@
+# Session Archive — May 2026 (early)
+
+Archived from `research/STATUS.md` on 2026-05-22 when the active session log
+exceeded 20 entries. Restore-style: append these back to STATUS Session Log
+if specific narrative context is needed for an investigation.
+
+- **2026-05-01**: HYBRID benchmark setup + framework upgrade from regimes to three-axis SAR characterization. (1) **HYBRID parquet built and benchmark scaffolded.** Original `merge_full_hybrid.py` had a 2-column schema bug (HYBRID score files have 4 columns); ran corrected merge via polars on the mount → `data/scores/adenine_hybrid.parquet` (5,203,413 unique products, 28 valid amidines, score range -19.46 to 24.52, mean -8.16). Top hits dominated by amidine_009/006 and **aldehyde_137 (70/100)** + aldehyde_430 (20/100) — different SAR landscape from FRED (where amidine_003/035 dominate). Registered `adenine_hybrid` as discoverable library: added to `MINIMIZE_LIBRARIES`, `SCORE_TO_REAGENT_FOLDER` (→ adenine), `_CPU_SEC_PER_LIBRARY` in `examples/_benchmark_config.py`. Created `jobs/adenine_hybrid_benchmark.sub`. First submission (51579841) ran 0 trials because `scripts/stage_benchmark_data.sh` had a separate hardcoded folder-name override list missing `adenine_hybrid`; refactored stage script to use a single `REAGENT_FOLDER` map mirroring the Python config. Discovery now correctly stages 25 libraries. Two SARS-uncovered amidines (022, 029) are pure-failure batches (not chemically valid products — SMARTS template misapplies); `submit_missing_hybrid.sh` written but the 37 missing batches are these unrecoverable batches, so the campaign is effectively complete. (2) **Hidden-hub posterior-separation analysis on HYBRID.** aldehyde_137 has z(mean) = -1.04 (moderately separable) but z(std) = +6.33 (intrinsically very wide — much wider than FRED's amidine_003 at z(std) = +3.16). aldehyde_430 at z(mean) = -3.61 is a strongly mean-separable rescue (~20% of top-100). HYBRID classifies as Mixed Ia + Ib like FRED post-merge but with the critical component shifted from amidines to aldehydes; predicted method ranking: TT-TS or tied with TACTICS RWS (no bimodal-failure risk because ald137's wider std mechanically protects against warmup-narrow-posterior). (3) **Three-regime taxonomy → three-axis SAR-landscape framework rewrite.** User-driven: refactored `[METHOD]` card `a19b3878-be4e-4e57-b02e-605ff68eaea5` from regime taxonomy to a three-axis framework. **Axes**: (1) hidden-hub size = % of top-N in mean-uninformative reagents (z(mean) > -1.5); (2) μ-reachable share = % of top-N in mean-separable reagents (z(mean) < -1.5); (3) hidden-hub σ-robustness = z(std) of dominant hidden hub. Each axis maps to a specific TACTICS algorithmic mechanism activation. Three improvements over the regime taxonomy: continuous (not categorical), explicitly handles mixed Ia+Ib via independent axes 1 and 2, surfaces the σ-robustness axis (genuinely new) which explains FRED pre/post-merge role flip and HYBRID cross-receptor robustness. Library balance is upstream of the three axes: it influences axis 3 specifically because per-reagent warmup observations scale inversely with component size (small components → narrow empirical posteriors → low σ-robustness → higher TT-TS bimodal-failure risk). Card title auto-updated to `[METHOD] TACTICS RWS — Three-Axis SAR-Landscape Framework for Paper 1`. (4) **Pre-merge composition analysis confirms role-flip mechanism.** Reconstructed FRED pre-merge top-100 by filtering amidine_035 from post-merge parquet: 68 a003 + 5 strongly-mean-separable hits (a015 + a033) + 19 moderately separable + 8 hidden across other amidines. The role-flip threshold is not "any mean-separable rescue exists" but "substantial μ-reachable share emerges": 5% rescue terrain pre-merge was insufficient; 36% from amidine_035 alone post-merge is a 7× jump and tips the comparison. The bimodal-failure mechanism (warmup-narrow-posterior on hidden hub) is unchanged between pre and post merge — what changed is RWS reliability (a035 stabilizes the amidine GMIC signal, eliminating RWS pre-merge bimodal failures) and the size of TT-TS's exclusive σ-niche (68 → 49). Resume next session: integrate post-merge adenine into full benchmark, run per-library best-Legacy baseline post-hoc, draft manuscript Methods + Discussion using the (now-five) framing cards.
+- **2026-05-01 (session 2)**: HYBRID benchmark execution + manuscript framing decisions + several runtime bugs found and fixed. (1) **HYBRID benchmark complete.** 100 trials, 5 methods × 20 reps × 1 query, 0 errors, 6.6 min runtime (`outputs/adenine_hybrid_benchmark/`). Top-100: TT-TS = 94.5 ± 5.1 (winner; **no bimodal failures**; 11/20 reps recover exactly 90 = 70 ald137 + 20 ald430 + 0 other; 9/20 reps recover exactly 100 = 70 + 20 + 10 tail-aldehyde — deterministic recovery of dominant hubs in every replicate); TACTICS RWS = 86.4 ± 14.3 (1 catastrophic [rep 15: recovery 26, only 1 ald137 hit, 902 ald137 evals vs 18,853 in successful reps] + 3 partial bimodal failures from aldehyde_137 GMIC mis-rotation); Legacy RWS = 84.0 ± 5.6; Greedy methods ~30. Best-TACTICS − best-Legacy = +10.5 pp. Cross-docking-pipeline framework validation confirmed: same chemistry, FRED axes (49%, 36%, +3.16) → RWS dominant; HYBRID axes (70%, 20%, +6.33) → TT-TS dominant. **Bimodal-failure mode migrated** from TT-TS-on-FRED (amidine_003 narrow-σ) to RWS-on-HYBRID (aldehyde_137 GMIC under-rotation) — confirms failure is library-and-receptor-specific, not method-specific. (2) **Manuscript framing refinements (user-driven).** Primary claim is **best-TACTICS beats best-Legacy across the board** (+0.4 to +53.8 pp range, only quinazoline ns); three-axis framework is **post-hoc mechanistic explanation, not prescriptive method-selection** (requires ground truth to compute axes); practical guidance is "use both TACTICS methods on any library, take the max — both individually robust, the better one substantially beats prior art on every benchmark library tested"; the *pair* of methods is robust because the bimodal-failure modes are anti-correlated. Adenine FRED + HYBRID positioned as **cross-docking-pipeline controlled experiment** (chemistry held constant, scoring varies — methodologically more defensible than "cross-receptor"); pre-merge adenine stays in supplementary at most. (3) **Bug fixes during HYBRID benchmark setup.** Discovered that `run_benchmark.py` resolved per-library `.smi` filenames and reactions.csv keys inconsistently from the reagent folder path. First-pass fix (this session) changed all 4 sites to use `folder_name`; that fix was wrong for hyphenated ROCS libraries and was superseded by the explicit `name_key` rule codified as CLAUDE.md Local Norm #11. (4) **macOS AppleDouble `._` files break SSHFS-mounted output_dir.** When `run_benchmark.py` writes temp parquets to `outputs/.../tmp_XXX/` on a macOS-SSHFS-mounted dir, macOS creates `._foo.parquet` xattr-companion files that downstream cleanup chokes on (`FileNotFoundError`). Workaround: use local-Mac output_dir (`~/tactics_local_outputs/`), then copy back via `cp -r` post-run. Alternative: `COPYFILE_DISABLE=1` env var disables AppleDouble creation for the process tree; works for keeping output_dir on the mount. Resume next session: integrate adenine post-merge + adenine_hybrid into `outputs/full_benchmark_ema_cats/`, recompute per-library best-Legacy baseline across the 25-library aggregate, draft manuscript Methods/Discussion paragraphs threading the across-the-board-win primary claim with three-axis framework as supporting mechanistic analysis.
diff --git a/pyproject.toml b/pyproject.toml
index 134525c..86e28cb 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "chem-tactics"
-version = "1.1.0"
+version = "1.2.0"
description = "Thompson Sampling-Assisted Chemical Targeting and Iterative Compound Selection for Drug Discovery"
readme = { file = "README.md", content-type = "text/markdown" }
license = {text = "MIT"}
diff --git a/research/STATUS.md b/research/STATUS.md
index 4fe9379..03ad2f0 100644
--- a/research/STATUS.md
+++ b/research/STATUS.md
@@ -1,8 +1,9 @@
# TACTICS — Research Status
> **Worktree branch**: `main`
-> **Last updated**: 2026-04-02
-> **Active hypothesis**: None — manuscript preparation. Mechanism analysis complete, benchmark results validated.
+> **Last updated**: 2026-06-08
+> **Heptabase project tag**: `Project-TACTICS-Primary-Engine`
+> **Active thread (2026-06-08): docking manuscript §3.2 drafted (8 cards) + global `lab-notebook` workflow skill built; 4% re-run complete.** The docking Results §3.2 now exists as 8 reviewed Heptabase cards (order: budget opener → consistency heatmap → case studies coverage/allocation/imbalance/control → discussion). The gate-fixed budget sweep is effectively done — the 4% re-run landed complete (218,800 trials, 0 short cells, recovery_summary current; the budget plots already point at it). Docking headline reframed: TACTICS is *consistent* (81–98% top-100, cross-lib σ 7.2), the cross-library variance lives in Legacy (Greedy 30–97%, σ 25.1) — "library-dependent" describes the SAR structure, not TACTICS performance. FRED-vs-HYBRID narrative dropped (no claims). See Session Log 2026-06-08. _Prior 2026-06-04 framing follows._ **Active hypothesis** (updated 2026-06-04): **(gate-fixed budget sweep SUBMITTED to HPC3 — awaiting results)** Splice+posthoc jobs for 1/3/5% and a full re-run of the OOM-incomplete 4% (mem matched to 5%, exit-guards added) are submitted on HPC3 (CPU — GPU declined: work is CPU-bound, and the 248–370G posthocs exceed GPU-node RAM). When they land, re-run the docking + diagnostic notebook data-load cells → all five budget points become gate-fixed. Diagnostic notebook (`diagnostic_benchmark_plots.py`) repointed to `nogate`; SAR landscape-probe JSONs refreshed (adenine winner flips RWS→TT-TS). New manuscript thread: greedy-vs-RWS **complementary** failure modes + the correction that single-component exploration is a SHARED limitation, not a TACTICS advantage (Established Results amended; new Open Question). _Prior 2026-06-03 framing follows._ **(gate fix RESOLVED — canonical 2% updated)** The TT-TS GMIC min-observation gate (`TopTwoSelection._calculate_gmic`, absent from RWS) was confirmed an artifact and removed (RWS-consistent; 31/31 TT-TS tests pass). The full gate-fixed 2% benchmark **`full_benchmark_nogate`** (218,800 trials, job 52894249) validated it: adenine TT-TS top-100 **87.3±16.2 → 95.5±3.3** (bimodal failures eliminated; now beats TACTICS RWS 94.5), 28-lib aggregate essentially unchanged (TT-TS 87.0→87.3) — surgical to adenine. **Canonical `full_benchmark_ema_cats` updated in place**: gate-fixed TT-TS spliced into the search tier (`examples/splice_gatefixed_ttts.py`; non-TT-TS byte-identical, 0/112 cells changed) + posthoc regenerated; ema_cats stays canonical (no notebook repoint). The cross-benchmark nogate-vs-ema_cats comparison is **confounded by run-to-run RNG** on bimodal hidden-hub libs (clean gate attribution = the same-seed counterfactual; adenine_hybrid RWS swung 86.4→77.5 between runs on failure-count alone). **Counterfactual retired** (removed from `manuscript_sar_plots.py` + `manuscript_plots_docking.py` §4; artifacts deleted; reagent-score-landscape kept; the amidine_035-counterfactual card superseded; new cards `a5457730` gate-fix + `19f566ab` RWS-bimodality). **Next: TT-TS-only gate-fixed sweep at 1/3/4/5% (`jobs/ttts_sweep_*.sub`, prepared, not submitted) → splice + posthoc per fraction; repoint diagnostics notebook to nogate; sweep manuscript prose for stale 'RWS wins adenine'/'TT-TS bimodal' annotations.** Also this session: built the **layer-2** within-component diagnostic `plot_adaptive_intensity`; moved the layer-1/2 diagnostics off the 5% sweep onto the **2% canonical** (regenerated `outputs/hybrid_diag_2pct` for the 4 docking hybrids, which `ema_cats` lacked diagnostics for); identified **stale pre-merge adenine diagnostics/posteriors** in `ema_cats`. See Session Log 2026-06-02. _Prior framing follows._ **Docking mechanism narrative consolidated.** Per-library method winners are explained by a three-mechanism *descriptive* landscape — coverage failure (`max(0, 1.5 + z_mean)`), allocation failure (`z_std × (1 − top1)`), reagent imbalance (`max/min count`) for "does TACTICS beat Legacy" (any one sufficient), plus hidden-hub σ-character for "TT-TS vs RWS"; GMIC routes WHICH component (layer 1), CATS multiplier / disagreement EMA set within-component intensity (layer 2). Formulas are constructed/conventional and fit to the 8 docking libraries → descriptive, NOT a prospective classifier; "run both TACTICS methods, take max" stands. Manuscript figures reworked in `tutorials/manuscript_sar_plots.py` (3-mechanism landscape, counterfactual recolored to −z(mean), hub plots removed, containment curve removed); new GMIC signal→decision→result diagnostic `plot_gmic_directed_exploration` (thrombin) in the diagnostics notebook. 5 new Heptabase cards (3 manuscript + 2 dead-end). Next: layer-2 within-component diagnostic, prototyped on a balanced library first. **(2026-06-01 cont.)** Landscape kept as the three greedy-failure-mode severities (de-clamping x to raw mean-quality tried + reverted; the `max` and the `1.5` are one mechanism, z*=−1.5 = reachability reference); renamed "SAR landscape" → "reagent score landscape"; +1 dead-end card (φ_int interaction-fraction does NOT predict greedy difficulty — no additivity axis explains quinazoline; extends 56dc09a3). **Docking-benchmark expansion (sEH + KinDEL + JNK3-ROCS) DROPPED — not pursued for Paper 1.** sEH dead-ended (DEL read-count is mean-separable, greedy beats TACTICS; see Ruled Out, card 24bd9c07); KinDEL (MAPK14/DDR1) and JNK3-ROCS judged low marginal value (redundant kinase family, single-pipeline, mean-separability risk). The 8 existing docking libraries already span the structural axes, and the FRED/HYBRID cross-pipeline contrast (amide greedy-collapse 95→47, TT-TS +8.8 pp) carries the docking story without it. **Current focus: docking narrative + figures for the manuscript.** Paper-1 scope: a split into an application note (chem-tactics) + a ROCS paper + a docking paper was considered and judged scientifically valid (each regime is independently novel — Zhao 2025 covered only ROCS, so a ROCS-only paper plus a separate docking paper would each stand alone), but **decision is to keep everything in a single comprehensive paper**; the deliberately large "mega-paper" scope is accepted. Prior context follows. Manuscript preparation continues. **Budget-sensitivity sweep fully confirmed at 5 budgets (2026-05-27)** — TACTICS dominance preserved at 1/2/3/4/5% on every (scoring × component) cell, with largest advantage at constrained budgets. Manuscript figure infrastructure complete in `tutorials/manuscript_plots.py` Section 4 (ROCS + DOCK sensitivity figures with 2×2 line+bar panels). DOCK shown to be library-dependent (wide error bars), ROCS uniform (tight error bars) — itself a manuscript finding. Aggregation methodology corrected: per-library best-T then average ≠ aggregate-then-max (DOCK 3-comp at 2% was reported as +5.88, actually +7.32, with adenine driving the difference). Scope reductions this session: top-N range capped at 25-250 throughout (wider top-N dropped from manuscript); sub-1% benchmarks (0.1%, 0.5%) generated but not run (floor binds on 26/28 libs); hidden-hub SAR framework retained for docking only (does not generalize to ROCS). **ROCS manuscript card scaffold complete (2026-05-28)** — 10 new Heptabase cards covering ROCS Results (R1=sensitivity, R2=mean recovery at 2%, R3=per-library Δ, R4=chained decomposition, R5a/R5b=diversity-recovery vector + Δ-slope split, R6=Tukey HSD completeness) + Discussion (D1=diversity-tradeoff centerpiece, D2=practical-vs-statistical commentary) + supporting [METHOD] Statistical Significance vs Practical Significance reviewer-defense card (`ee819848`). All tagged Project-TACTICS-Primary-Engine + Type-Manuscript + (Section-Results / Section-Discussion / Section-Methods). Chained-decomposition analysis added as new presentation device: at top-100 pooled across 20 ROCS libs, Zhao 2025's RWS contribution to recovery vs Klarich-style standard TS is +1.73 pp (33% of total +5.25 pp lift), TACTICS algorithmic additions contribute +3.53 pp (67%); credit redistributes monotonically with top-N to 3%/97% by top-250. On 2-comp ROCS, Zhao's contribution crosses negative at top-100 (−0.26 pp) and reaches −2.28 pp at top-250 — Legacy RWS underperforms Legacy Greedy at wider top-N, and TACTICS recovers this loss while adding positive lift. New `tutorials/manuscript_plots_ROCS.py` separates ROCS plots from docking (1357 lines of ROCS cells migrated); `manuscript_plots.py` retains docking sections + budget-sensitivity DOCK panel. Per-library Δ-bar plot converted from "best of TT-TS / T-RWS" to TT-TS-only after empirical verification that TACTICS RWS beats TT-TS on only 21/200 (ROCS library × top-N) cells with maximum gap 0.79 pp (poparov top-25), well below the manuscript's ~2 pp practical-significance threshold. Journal-fit decision: Digital Discovery / JCIM / JCTC / J Cheminform are the realistic submission targets; Nature family stretch requires presubmission inquiry first, not worth the structural-rewrite cost. Resume next session: write manuscript Results §Budget Sensitivity prose from the corrected figure infrastructure; draft Discussion paragraph on DOCK library-dependent vs ROCS uniform finding; begin parallel docking-section card scaffold in Heptabase. **Statistical framework finalized (2026-05-17)**: scoring-type rule — docking libraries (single query × 20 reps) use planned Welch's t-test + Bonferroni-4 on the four pre-specified TACTICS-vs-Legacy pairs; ROCS libraries (109 queries × 20 reps) use repeated-measures ANOVA + Tukey HSD per Ash et al. 2025. Levene's test dropped from inferential pipeline (visual evidence + per-method σ table is sufficient; pre-test bias avoided). Cohen's d_s dropped from manuscript text (kept in parquet outputs only) because all measurements share the % recovery scale — raw mean diff + 95% CI is the appropriate effect-size statement. CLT-based defense of parametric tests at n=20 with bimodal failure distributions documented in Methods. **Manuscript Methods §Statistical analysis written** in the new `~/Desktop/TACTICS-manuscript/manuscript.qmd` (TACTICS-manuscript repo moved off SSHFS mount mid-session). Discussion §Sensitivity analysis (amidine_035-omitted pre-merge benchmark: TT-TS = 96.0 ± 4.0 wins; TACTICS RWS = 78.5 ± 24.0 bimodal-failing) + §Single-pose docking and ABFE/FEP integration future-work paragraph also written. **Four-paper pipeline formalized in Heptabase (2026-05-18)**: Paper 1 (current manuscript), Paper 2 (TACTICS + DrugEx, vendor-mapping + generative reagent expansion), Paper 3 (TACTICS as cofolding-ensemble structure-quality validator for FEP triage — novel inversion, lit-scan-confirmed), Paper 4 (TACTICS + pathway-aware multi-target for cell-line selection — evaluation-side dual to Zaballa & Hui 2024). Tag hierarchy: `TACTICS-Papers` umbrella + per-paper sub-tags; 6 new OBJECTIVE/RESEARCH-GAP cards (2 per paper) saved to Heptabase with LaTeX math. Resume next session: continue drafting Methods + Results + Discussion paragraphs in `manuscript.qmd` from the now-44-card Heptabase set (`TACTICS-Manuscript` 38 cards + `TACTICS-Papers` 6 new cards spanning Papers 2-4).
---
@@ -19,7 +20,6 @@
- **GMIC is better criticality score estimator**: GMIC and KL divergence gating show similar performance in recovery rate compared to IPR with SNR. Since GMIC is simpler to implement and matches performance it should be used as the default method.
- **Component Criticality based heating Scheme reverse more top hits that legacy RWS**: When using enhanced warm on libraries with equal number of reagents in reaction components. And using similarity based scoring functions Component Criticality Thompson Sampling (CATS) shows better recovery of top 100 molecules compared to legacy RWS.
- **TT-TS outperforms RWS (GMIC) on combinatorial library recovery**: Top-Two TS with asymmetric thermal cycling (heated_scale=1.5, cooled_scale=0.75) significantly outperforms RWS+GMIC on 6/21 libraries, never worse. Overall recovery 86.1 vs 85.5 (65,430 trials, 21 libraries).
-- **Manuscript benchmark — criticality-weighted rotation is the primary mechanism**: Full 4-method comparison (87,240 trials, 20 libraries). GMIC criticality-weighted rotation accounts for 92% of gains on 2-comp libraries (+4.3 pts over Legacy RWS with same warmup/budget) and 53% on 3-comp (+0.9 pts). TT-TS adds modest further improvement, most valuable on 3-comp where criticality signal is noisier (+0.8 pts). TACTICS breaks the 2-component plateau where Legacy RWS barely improves over Legacy TS (+0.8 pts). Largest gains on hardest queries (+6.1 pts on queries with Legacy RWS < 65 recovery).
- **Top-N sensitivity — TACTICS advantage grows universally at wider thresholds**: 697,600 result rows across 8 top-N thresholds (25–250). The TACTICS TT-TS advantage over Legacy RWS grows from +2.0pp at top-25 to +4.6pp at top-250 (all p<0.0001). On 2-comp: +3.1pp→+6.7pp (+16.7 additional molecules at top-250). Trend holds on 20/20 libraries. Legacy RWS crosses below Legacy TS on 2-comp libraries at ~top-150 (8/10 libraries), showing unguided thermal cycling becomes counterproductive at wider windows.
- **heated_scale=1.5 erases TT-TS advantage on imbalanced libraries**: On thrombin (130×3844, 29.6:1 ratio), TT-TS with hs=1.5 scores 73.5 (= Legacy TS, ns). With hs=1.0, all beta values significantly beat Legacy TS (+2.6 to +4.8, p<0.02). Best config β=0.5/hs=1.0 = 78.6 (p=0.0005 vs Legacy TS). Root cause: the large amine component (5 obs/reagent) has wide posteriors; inflation pushes disagreement to near-100%, making challenger selection random. TACTICS RWS = 77.0 (p=0.002 vs Legacy TS). Legacy RWS = 66.2 (broken thermal cycling on imbalanced libraries). 20 replicates + 10-rep parameter sweep (12 configs).
- **Adaptive per-component heated_scale recovers TT-TS on imbalanced libraries**: Bidirectional EMA-based adaptation (per-component, decay_rate=0.95, high_threshold=0.8, low_threshold=0.3) self-tunes heated_scale on thrombin. Adaptive TT-TS = 77.5 ± 3.6 (p=0.0002 vs Legacy TS), matching TACTICS RWS = 77.3. Fixed hs=1.5 scored 75.0 (ns). Mechanism: per-component EMA tracks disagreement rate; saturated components (>0.8) decay toward hs=1.0, under-explored components (<0.3) grow. Enabled by default in TopTwoConfig.
@@ -31,15 +31,49 @@
- **Uniform vs concentrated docking failures — implications for TACTICS reliability**: The adenine failure distribution is approximately uniform across reagents, meaning the SAR signal is preserved despite 17% information loss. This is the benign case — TACTICS recovers hits reliably. If failures were concentrated in a specific chemical subclass (e.g., all bulky aldehydes fail), that would create a systematic SAR blind spot undetectable from posteriors alone. GMIC per-component trajectories could serve as a diagnostic: anomalous convergence patterns (premature stabilization with low GMIC) may indicate concentrated failures. This is an open question for the library diagnostics paper (Paper 3).
- **Full benchmark validated with rigorous statistical framework (22 libraries, 2.18M recovery rows)**: 218,200 trials (22 libraries including adenine, 5 methods, 20 replicates), post-hoc recovery at 10 top-N thresholds (25–250). Statistical tests following Ash et al. 2025 (JCIM): repeated measures ANOVA + Tukey HSD (within-library, FWER-controlled), Friedman + Nemenyi (across-library), Cohen's d effect sizes. All 220 ANOVA tests significant. Friedman chi² > 72 at every threshold (p < 10⁻¹⁴). Overall top-100: TT-TS 86.5, RWS 85.8, Legacy RWS 82.9, Balanced-Greedy 82.0, Legacy Greedy 81.2.
- **ROCS and docking require separate statistical analyses**: Pooled Tukey HSD across all 22 libraries produces inflated paired SDs and deflated Cohen's d values because ROCS similarity and FRED docking scores have fundamentally different variance structures. Adenine's bimodal outcomes (sd=27.6 for RWS) dominate the pooled variance, masking the true effect sizes on ROCS libraries. Correct approach: Tukey HSD on 20 ROCS libraries (2,180 query-level pairs, FWER=0.05) separately from docking libraries (40 replicate-level observations). ROCS results: all TACTICS-vs-Legacy comparisons **** (p<0.0001), TT-TS vs RWS and Balanced-Greedy vs Legacy methods are ns. Docking results: TACTICS-vs-Legacy all **** with large effect sizes (d=0.79–1.23, +21–34 pts), TT-TS vs RWS ns (p=0.105) due to adenine bimodality.
-- **TT-TS adenine failure (rep 15) traced to warmup-dependent amidine_003 under-exploration**: amidine_003 contains 68/100 top-100 compounds. In rep 15, only 0.6% of search evaluations targeted amidine_003 (663/110,777) vs 4–15% in successful replicates. Both reps received ~45 warmup observations of amidine_003 with identical mean scores (-12.06), but stochastic partner pairing during warmup produced an uninformative posterior in rep 15. TT-TS's disagreement mechanism never flagged amidine_003 as uncertain → search budget directed elsewhere → 62/68 amidine_003 top hits missed. This is a 5% failure rate (1/20) driven by extreme hit concentration in a single reagent (68% of top-100). TT-TS is still far more robust than RWS (95% vs 70% success rate) because posterior disagreement detects uncertainty even when the mean is mediocre — the failure requires both a mediocre AND narrow posterior, which is rare.
+- **GMIC stabilizes within first few cycles regardless of batch size**: Diagnostic data analysis across 5 libraries shows cycle-to-cycle GMIC delta = 0.001–0.03, all components hit std < 0.1 within 0–13 cycles (0–7% of search). GMIC is computed from cumulative posteriors, not the current batch, so batch granularity does not affect signal quality. Smaller batch sizes provide more rotation decisions but the same GMIC information.
+- **TT-TS disagreement EMA rarely enters resolved region on balanced libraries**: Across all diagnostic libraries, most components spend the majority of cycles in the saturated region (EMA > 0.8). Exception: orru comp 2 enters resolved region 34–56% of cycles → high heated_scale (3.5–4.0). With 100 reagents per component, posteriors never sharpen enough for two independent draws to consistently agree. The adaptive mechanism primarily operates in one direction (decaying heated_scale), not the bidirectional feedback loop it was designed for. The thresholds (0.3/0.8) may need recalibration for large reagent counts.
- **Docking vs ROCS recovery comparable at 2% budget — Riley's 10× penalty not observed**: At 2% evaluation budget, docking libraries recover at rates comparable to ROCS (amide 97.2, quinazoline 85.2, thrombin 81.0 vs ROCS mean ~86). This appears to contradict Riley 2024's finding that docking needs 10× more budget than ROCS. Resolution: at 2% budget with libraries of 500K–20M products and 130–10,000 reagents per component, each reagent receives ~67–100+ observations. With this many observations, posterior means become informative proxies for "this reagent tends to participate in good combinations" even when the underlying mechanism is interaction-driven (non-additive). Riley's 10× penalty is expected to emerge at lower budgets (0.1–0.5%) where per-reagent observations are insufficient for posteriors to converge. Both ROCS and docking have a large enough budget at 2% that the non-additivity issue is overcome. Further evidence: thrombin receives 5.8% budget (29,220 evals, forced by warmup floor) yet has the lowest docking recovery (81.0) — more budget does not translate to more recovery when per-reagent observation density is the bottleneck (3,844 amines at ~5 obs each). Meanwhile amide at 2% (200K evals) recovers 97.2 because its 1,000 amines each get ~15 obs. Per-reagent observation density, not library percentage, is the governing factor. This is a discussion point for the manuscript, not a research question requiring further investigation.
- **Reagent posterior separation analysis explains per-library TACTICS advantage on docking**: Riley-style (mean, std) plots for all 4 docking libraries reveal that TACTICS advantage scales with hit concentration × posterior uninformativeness. Adenine (+62.1): amidine_003 has 68/100 top hits but average mean (-12.33, indistinguishable from population) — only detectable via posterior uncertainty (high std=3.00, fewer obs). Amide (+5.1): top acids have the best means AND low std (0.93) — most ROCS-like, mean-separable. Thrombin (+3.2): top dipeptides have the best means (-14.2 to -14.5) despite 3,844-reagent haystack — greedy finds them via mean exploitation, GMIC adds modest speedup. Quinazoline (+0.1 ns): all components mean-separable, hits evenly distributed, no asymmetry to exploit. The top-N trend inversion (ROCS advantage grows, docking advantage shrinks at wider top-N) follows directly: at wider top-N, more docking hits become greedy-reachable (diluted concentration), while ROCS requires exploration of second-tier reagents. Key manuscript discussion point connecting Riley's finding to per-library SAR characterization.
-- **SAR additivity governs greedy reachability — multi-top-N posterior separation analysis**: Z-scored per-reagent (mean, stdev) at top-N thresholds 25–250 across all 4 docking libraries. Quinazoline hit reagents start well-separated (z ≈ -2.0 at top-10) and erode to z ≈ -0.76 at top-1000 — greedy works at every threshold. Adenine hit reagents are barely separated even at top-10 (amidines z = -0.75, aldehydes z = -0.48) and reach z ≈ -0.13 at top-1000 (indistinguishable from population). The starting separation, not the erosion rate, determines whether greedy can succeed. TT-TS succeeds on adenine because it operates on posterior uncertainty (wide posteriors → disagreement → exploration), not reagent means — it bypasses the non-additive SAR entirely. Legacy RWS fails on docking because symmetric thermal cycling diversifies the component that needs exploitation (specific reagents must be selected repeatedly with different partners). GMIC inverts this by detecting solved vs unsolved components. Extends Riley (2024) by showing: (1) mixing severity varies across libraries, (2) mixing worsens at wider top-N, (3) TT-TS bypasses mixing via uncertainty.
-- **ROCS SAR erosion at wider top-N and Legacy RWS single-component exploration bottleneck**: Multi-top-N analysis on 4 ROCS libraries (rxn206, rxn101, petasis, poparov) shows the opposite pattern to docking: hit reagents start well-separated (z ≈ +1.4 to +3.6 at top-25) and erode toward the population mean at wider top-N as the first SAR series is exhausted and second-tier reagents (good for different structural motifs) are needed. TACTICS advantage grows at wider top-N because greedy gets stuck exploiting the first SAR series. Legacy RWS declines more steeply than all other methods because its round-robin thermal cycling can only explore one component at a time — it can never pair two exploratory choices simultaneously. On 2-comp libraries this causes the documented crossover below Legacy Greedy at ~top-150 (8/10 libraries): the exploration overhead exceeds the value of single-component discoveries. On 3-comp the onset is delayed but the same bottleneck surfaces at wider top-N. TACTICS avoids this because TT-TS posterior disagreement and GMIC-weighted rotation operate independently per component, enabling coordinated multi-component exploration in a single iteration.
+- **SAR additivity governs greedy reachability — multi-top-N posterior separation analysis**: Z-scored per-reagent (mean, stdev) at top-N thresholds 25–250 across all 4 docking libraries. Quinazoline hit reagents start well-separated (z ≈ -2.0 at top-10) and erode to z ≈ -0.76 at top-1000 — greedy works at every threshold. Adenine hit reagents are barely separated even at top-10 (amidines z = -0.75, aldehydes z = -0.48) and reach z ≈ -0.13 at top-1000 (indistinguishable from population). The starting separation, not the erosion rate, determines whether greedy can succeed. TT-TS succeeds on adenine because it operates on posterior uncertainty (wide posteriors → disagreement → exploration), not reagent means — it bypasses the non-additive SAR entirely. Legacy RWS fails on docking because symmetric thermal cycling diversifies the component that needs exploitation (specific reagents must be selected repeatedly with different partners). GMIC inverts this by detecting solved vs unsolved components. Extends Riley (2024) by showing: (1) mixing severity varies across libraries, (2) mixing worsens at wider top-N, (3) TT-TS bypasses mixing via uncertainty.
+- **ROCS SAR erosion at wider top-N and Legacy RWS single-component exploration bottleneck**: Multi-top-N analysis on 4 ROCS libraries (rxn206, rxn101, petasis, poparov) shows the opposite pattern to docking: hit reagents start well-separated (z ≈ +1.4 to +3.6 at top-25) and erode toward the population mean at wider top-N as the first SAR series is exhausted and second-tier reagents (good for different structural motifs) are needed. TACTICS advantage grows at wider top-N because greedy gets stuck exploiting the first SAR series. Legacy RWS declines more steeply than all other methods because its round-robin thermal cycling can only explore one component at a time — it can never pair two exploratory choices simultaneously. On 2-comp libraries this causes the documented crossover below Legacy Greedy at ~top-150 (8/10 libraries): the exploration overhead exceeds the value of single-component discoveries. On 3-comp the onset is delayed but the same bottleneck surfaces at wider top-N. **[Corrected 2026-06-04]** The single-component-exploration bottleneck is SHARED — TACTICS does NOT avoid it: code (`roulette_wheel.py:261`, `top_two_selection.py:131`) shows all three methods (Legacy RWS, TACTICS RWS, TACTICS TT-TS) heat exactly one component per cycle (`is_heated = component_idx == self.current_component_idx`), so no single combination explores two components at once. The per-component signals (GMIC, disagreement EMA) are computed independently per component, but the heating (σ-inflation / temperature) is applied to ONE component per cycle. TACTICS improves only the DIRECTION of the single heated slot (GMIC-targeted vs blind round-robin), not the simultaneity. (The prior "coordinated multi-component exploration in a single iteration" claim was wrong; the linked Heptabase card is amended. See Open Questions: simultaneous multi-component exploration.)
- **TT-TS diagnostics schema was silently dropping mechanism data**: The saved full benchmark diagnostics for TT-TS trials were missing `disagreement_ema`, `heated_scale`, `effective_scale` (5 TT-TS-specific columns dropped by GMIC schema coercion in `sampler.get_diagnostics()`). Fix: added `_TTTS_DIAGNOSTICS_SCHEMA` (12 columns), detection via `disagreement_ema` key, and `criticality` alias in `TopTwoSelection.get_component_state()`. Added `compute_disagreement_convergence()` and `compute_scale_adaptation()` analysis functions, plus 3 TT-TS plot functions in `diagnostic_plots.py`. 43 tests pass (13 new). Existing RWS/BayesUCB diagnostics unaffected.
- **Adaptive heated_scale runaway on balanced 3-comp libraries — cosmetic, not functional**: On 3-comp libraries with balanced reagent counts (100×100×100), one component's posteriors converge early, dropping disagreement EMA below `low_threshold=0.3`. The adaptive rule inflates heated_scale to 2–5× before EMA self-corrects. Consistent across all 5 replicates on orru and mannich (structural, not stochastic). Originally interpreted as a pathology requiring a growth cap fix (`max_growth_per_step=0.1`). Per-query diagnostic analysis (2026-04-01) showed this has no measurable recovery impact — the smaller TACTICS advantage on 3-comp libraries is explained by easier SAR (higher min_GMIC, SNR 12-42), not mechanism malfunction. Growth cap reverted to None (disabled).
- **GMIC direction is query-specific — per-query diagnostic analysis confirms adaptive mechanism**: Diagnostic benchmark analysis (110 trials, 5 libraries, 11 queries, 5 replicates) revealed that GMIC direction varies per query molecule, not per library. On rxn206, query_008 identifies c1 as flexible throughout while query_090 identifies c0 (opposite direction). 5/9 ROCS queries show GMIC redirection mid-search (most-flexible component changes) with 80-100% cross-replicate consensus. The "near-uniform" rotation probabilities reported at the library level are an artifact of averaging across queries with opposing GMIC directions. Per query, the signal is consistently directional. CATS multiplier is active on 43-56% of iterations, with temperature modulation ranging from 0.025 to 0.20 (8:1 vs 2:1 from alpha/beta alone). min_GMIC (the weakest component) is the strongest predictor of TACTICS advantage: queries with min_GMIC < 1.0 show large gains, queries with min_GMIC > 1.5 show negligible gains regardless of component count.
- **Expanded docking benchmark — 4 libraries, 3 protein families, TACTICS advantage scales with difficulty**: Added amide-JNK3 (10M, 2-comp, 10:1 ratio, 3.3% failures) and quinazoline-JNK3 (20M, 3-comp, 2.1:1 ratio, ~0% failures). Full 24-library benchmark (2,184,000 recovery rows, 5 methods, 20 replicates, 10 top-N thresholds). Docking top-100: TT-TS 97.2 (amide), 85.2 (quinazoline), 81.0 (thrombin), 93.9 (adenine). TACTICS advantage scales with imbalance × failure rate: +0.1 on balanced quinazoline (ns) to +62.1 on hard adenine (****). Legacy RWS consistently worst on docking. Friedman chi²=73.1, p<10⁻¹⁴ across 24 libraries. 2×2 design: 2-comp vs 3-comp × 2 protein families each.
+- **Adenine docking failures are uniform — SAR signal preserved despite 16% information loss**: 4,909,597/5,865,888 products scored (16.3% failure). Active reagents have 80–92% docking success rate uniformly across all three components. 34 reagents (2 amidines, 14 isocyanides, 18 aldehydes) have 0% success due to enumeration/Omega2 failures, not docking failures. Amidine_003 at 40.7% is the sole meaningful outlier among active reagents.
+- **Quinazoline -0.75pp TT-TS shift between pre/post-fix benchmarks is replicate noise**: Pre-fix TT-TS 85.2% vs post-fix 84.5% on quinazoline top-100. z=1.26, p=0.208. Diagnostics confirm identical GMIC trajectories and heating fractions between runs. Shift from 2/20 replicates landing one discrete step lower. Neither rotation fix nor EMA CATS affects TT-TS on quinazoline (GMIC uniformly >2.0, converges within first few cycles).
+- **Legacy method preference reflects SAR structure — RWS for ROCS, Greedy for docking**: Per-library win count (top-100, 24 libraries). 2-comp ROCS: Legacy Greedy vs Legacy RWS split 4/6, mean |Δ|=1.1pt (tied — balanced SAR). 3-comp ROCS: Legacy RWS wins 10/10, mean +3.7pt (multi-cluster SAR benefits from diversity). 2-comp docking: Legacy Greedy wins 2/2, mean +9.0pt (mean-separable SAR; thermal cycling diversifies away from exploitable reagents). 3-comp docking: mixed — RWS wins on adenine (+13.7, non-additive SAR, round-robin bootstraps amidine_003), Greedy wins on quinazoline (+2.3, mean-separable). TT-TS dominates all regimes: amide 97.3 vs Greedy 95.4 (+1.9), thrombin 81.8 vs 77.3 (+4.5), because it combines greedy-like exploitation on cooled-draw with uncertainty-driven exploration on heated-draw.
+- **Adenine TT-TS recovery is two-compartment — permanent +12pp advantage from amidine_003 alone**: On adenine at top-10000, TT-TS = 71.1% vs Legacy Greedy = 58.6%. Decomposition: TT-TS recovers 79.1% of amidine_003's 1,315 top-10000 hits (Greedy 1.7%) and 69.9% of "other amidines" hits (Greedy 67.2%). The +12pp total advantage comes entirely from amidine_003. At top-25 (100% in amidine_003), TT-TS = 99.4%; at top-10000 (13% in amidine_003), the mix shifts toward greedy-reachable "other" reagents and TT-TS converges toward greedy-like behavior. The decreasing-fraction-with-top-N signature is a ceiling effect from best-arm identification saturating narrow windows, not a mechanism failure. TT-TS evaluates ~9,685/202,272 (4.8%) of amidine_003's combination space per replicate.
+- **iso_db_472 is discovered via cooled-draw posterior updating, not GMIC**: On adenine, isocyanide_db_472 contains 46/100 top-100 hits (44 paired with amidine_003 specifically — pair-level non-additive SAR). GMIC for the isocyanide component stays at 0.000 throughout search (observation density bottleneck — 688 reagents × 100K budget). Discovery mechanism: (1) TT-TS disagreement detects amidine_003 via wide posterior std → a003 gets heated-draw exposure, (2) during amidine-heated cycles, cooled-iso draw is stochastic across overlapping posteriors, (3) a003 × iso_db_472 pairs score ~-15 → both posteriors update sharply, (4) iso_db_472 becomes cooled-draw argmax → pair locks in. Discovery is **Bayesian posterior-level**, operating below GMIC's component-level resolution. iso_db_472's ground-truth mean is z=-1.43 (rank 8/674) but the full value is only realized paired with a003, which Greedy never reaches.
+- **GMIC has structural observation-density threshold — fails when reagent count > ~5-10% of budget**: GMIC = Var(μ_i) / mean(σ²_i) where the denominator averages variance across all reagents in a component. When 85-90% of reagents remain at warmup variance (unexplored), the denominator inflates 2-12× and drowns signal. Quantitative threshold on adenine (~100K search budget): amidine (27 reagents, ~3,700 obs/reagent uniform) → GMIC activates; aldehyde (276 reagents, ~360 obs) → partial; isocyanide (688 reagents, ~145 obs) → GMIC = 0.000 throughout. Rule of thumb: GMIC needs reagent count ≤ ~5-10% of total observation budget for signal to emerge. Open question line 131 (observation-weighted GMIC) is the structural fix.
+- **CATS multiplier fires on only ~53% of (cycle, component) rows; misalignment is the modal failure mode**: Diagnostic benchmark (98,730 component-rows across 5 libraries × RWS × 5 reps). Pooled breakdown: multiplier amplifies (heated+flexible OR cooled+critical) on 52.7%; no-op due to rotation/GMIC misalignment on 44.0% (heated+critical 16.9%, cooled+flexible 27.1%); diversity-mode (gate closed) on 3.2%. Rotation success per heated cycle ranges from 62% (adenine, sustained structural asymmetry) to 42% (quinazoline, where rotation picks the *critical* component more often than the flexible — but the four-case truth table no-ops on misaligned cycles, so the error-correction property prevents this from converting into recovery loss). Late-search rotation alignment degrades on balanced libraries (orru: 77% early → 46% late, quinazoline: 62% → 35%) but recovery is preserved because the architectural floor (asymmetric α/β + no adaptive α inflation) carries the work when GMIC stops directing.
+- **Adenine library expanded with amidine_035 — FRED ground truth re-run and merged**: `data/scores/adenine.parquet` updated 2026-04-30 with 174,001 new amidine_035 × isocyanide × aldehyde products. Post-merge totals: 5,083,598 rows (was 4,909,597), 28 unique amidine prefixes (was 27), 30 listed in `data/reagents/adenine/adenine_reagent_0.smi`. The two amidines absent from the data (`amidine_022`, `amidine_029`) are not covered by the adenine reaction SMARTS — same SMARTS-coverage gap previously reported as "2 amidines have 0% success" (line 47), confirmed unchanged. amidine_035 yield 174,000/202,272 = ~86%, matching the established 14–17% omega2/FRED failure rate for other amidines. Score range -20.89 to 21.78, mean -13.10, no nulls. All adenine benchmark recovery numbers in lines 30, 47, 51 reflect the pre-merge 4.9M-row data; rerun staged as `jobs/adenine_rerun_amidine_035.sub` (5 methods × 20 reps × adenine only, output `outputs/adenine_rerun_amidine_035/`) but not yet executed. Pre-merge backup retained at `data/scores/adenine.parquet.bak`.
+- **HYBRID adenine campaign complete and parquet built**: Second adenine docking campaign with HYBRID (OpenEye) against the Lenselink-prepared A2A receptor (`data/docking/adenine/a2ar-lenselink.oedu`). 543/580 batches docked successfully (~5.4M unique products with stereoisomer dedup); the 37 missing batches are pure amidine_022 (batches 348–366) + amidine_029 (batches 445–462) — the SMARTS-uncovered amidines whose enumerated SMILES are structurally invalid (omega2 produces empty conformer files; HYBRID then fails with "Database has no molecules"). These batches are not recoverable and are correctly absent from the parquet. The `merge_full_hybrid.py` script had a 2-column schema bug (HYBRID score files have 4 columns: Title, HYBRID Chemgauss4 score, Receptor Title, Receptor File); a corrected merge ran on 2026-05-01 producing `data/scores/adenine_hybrid.parquet` (5,203,413 rows, 28 unique amidines, 51.3 MB, score range -19.46 to 24.52, mean -8.16). Same SMARTS-coverage gap as FRED (amidines 022, 029 absent). Top-100 dominated by **aldehyde_137 (70 hits)** + aldehyde_430 (20 hits) — different SAR landscape from FRED (amidine_003 + amidine_035 dominant); aldehyde-axis SAR concentration distinguishes the two receptor preparations.
+- **HYBRID benchmark — TT-TS dominant on a second docking scoring function**: 100 trials (5 methods × 20 reps × 1 query, 0 errors, 6.6 min runtime), `outputs/adenine_hybrid_benchmark/`. Top-100 results: **TT-TS = 94.5 ± 5.1** (winner; no bimodal failures; 11/20 reps recover exactly 90 = 70 ald137 + 20 ald430; 9/20 reps recover exactly 100 = the same 70+20 + 10 tail-aldehyde hits — deterministic recovery of dominant hubs in every replicate); TACTICS RWS = 86.4 ± 14.3 (1 catastrophic + 3 partial bimodal failures via aldehyde_137 GMIC mis-rotation, 902 ald137 evals vs 18,853 in successful reps); Legacy RWS = 84.0 ± 5.6; Legacy Greedy = 29.8 ± 1.4. Best-TACTICS − best-Legacy = +10.5 pp. **Cross-scoring-function robustness (reframed 2026-06-06)**: a second docking scoring function (HYBRID/Lenselink-A2A vs FRED/4EIY) on the same adenine chemistry shifts the top-100 hub from the amidine axis (FRED: amidine_003 + amidine_035) to the aldehyde axis (HYBRID: aldehyde_137 + aldehyde_430), zero top-100 overlap. Legacy methods are sensitive (Legacy Greedy 46 → 30; Legacy RWS bimodal), while TACTICS TT-TS stays consistent and is top on both (gate-fixed FRED 95.6, HYBRID 94.0; 0 bimodal failures). Modest within-docking robustness point — FRED and HYBRID are similar OpenEye Chemgauss4 variants. The earlier "different scoring function → different optimal method (FRED→RWS, HYBRID→TT-TS)" / cross-docking-pipeline-framework-validation framing is RETIRED (it rested on the now-removed TT-TS GMIC min-observation gate artifact; post-gate-fix TT-TS wins both). Result merged into the "never the worst" card (32f0a5f5).
+- **TACTICS methods are never the worst on any (library × top-N) cell — universal-floor property**: Across 28 libraries × 4 reported top-N (top-100, 250, 1000, 2500) = 112 benchmark cells at the 2pct canonical budget, TACTICS TT-TS and TACTICS RWS are each *never* the worst-performing method on any cell. Per-library winner counts: TT-TS wins 21–23/28 at every reported top-N; combined TACTICS (TT-TS or T-RWS) wins 26–27/28; the single non-TACTICS-winner library is **amide** at wider top-N (saturated 94–98% across all methods, noise picks the winner). Worst-method count: at top-2500, Legacy RWS is worst on 26/28 libraries — symmetric thermal cycling becomes counterproductive once first-cluster reagents are resolved, wasting budget on saturated mean-separable terrain. This extends the previously-documented 2-comp ROCS L-RWS-vs-Greedy crossover into the docking aggregate. The honest framing is the never-worst floor plus majority-winner, not strict per-cell dominance (counterexamples exist, e.g., adenine_hybrid at top-25 where L-RWS wins); the property holds across both structure-based (docking) and ligand-based (ROCS) scoring.
+- **Budget-sensitivity benchmark — TACTICS dominance robust at 1pct; gap *grows* at lower budget**: `outputs/full_benchmark_1pct/` (218,800 trials, 28 libraries, 5 methods, 20 reps, budget = 1% of library — half the canonical 2pct). Aggregate top-100: TACTICS TT-TS 81.0 ± 18.9 (winner), TACTICS RWS 80.1 ± 18.9, Legacy RWS 75.5, Bal-Greedy 75.2, Legacy Greedy 74.0. TACTICS-vs-Legacy-RWS lead = **+5.5 pp on ROCS (vs +3.5 at 2pct)** and **+10.1 pp on docking (vs +8.5 at 2pct)**. Best-TACTICS recovers ≥50% top-100 on every benchmark library at 1pct (worst: adenine_hybrid T-RWS = 50.7%); best-Legacy fails to clear 50% on adenine_hybrid (L-RWS = 31.1%, L-Grdy = 27.6%). Friedman χ²=66–80 at every top-N (p < 10⁻¹³); 958/1120 planned-Welch pairs Bonferroni-significant. The TACTICS algorithmic contribution is robust to budget pressure — criticality-directed allocation does proportionally *more* of the work when warmup leaves the criticality signal noisy.
+- **Bimodal-failure concentration at low budget is the manuscript-relevant cost of halving the budget**: Variance increase from 2pct → 1pct is *entirely concentrated* in adenine + adenine_hybrid (the hidden-hub docking pair). TT-TS docking aggregate sd 18.9 (1pct) vs 15.7 (2pct); RWS docking 18.9 vs 15.5. Per-library budget cost at top-100: adenine TT-TS 87.3 → 61.4 (−26 pp), RWS 94.5 → 61.2 (−33 pp); adenine_hybrid TT-TS 94.5 → 43.4 (−51 pp), RWS 86.4 → 50.7 (−36 pp); amide_hybrid TT-TS −12 pp; all ROCS libraries lose 5–6 pp uniformly. Quinazoline_hybrid, thrombin, thrombin_hybrid are essentially flat between 1pct and 2pct — those libraries are saturated at 1pct already. **2pct is the smallest budget at which the hidden-hub docking pair clears the bimodal-failure regime** (best-TACTICS ≥94% on both). This decision rule justifies 2pct as the canonical operating point for the manuscript.
+- **3pct budget benchmark — TACTICS dominance at intermediate operating point**: Top-100 aggregate (28 libraries, 5 methods, 20 reps, 218,800 trials): TT-TS = 90.7 ± 12.7 (winner), TACTICS RWS = 90.3 ± 12.6, Legacy RWS = 87.8 ± 12.7, TACTICS Balanced-Greedy = 86.5 ± 16.8, Legacy Greedy = 85.9 ± 17.2. TACTICS-vs-Legacy-RWS gap is +2.8 pp on ROCS, +7.7 pp on docking. Within-TACTICS Nemenyi at top-100: TT-TS vs TACTICS RWS p = 0.90 (tied). Per-library winner at top-100: TT-TS on 19/28 libs, TACTICS RWS on 7/28. 95 of 112 Welch + Bonferroni-4 planned pairs Bonferroni-significant. `outputs/full_benchmark_3pct/`.
+- **4pct budget benchmark — saturation pattern confirmed**: Top-100 aggregate (same setup as 3pct): TT-TS = 93.0 ± 10.7, TACTICS RWS = 92.7 ± 10.7, Legacy RWS = 90.2 ± 11.1, TACTICS Balanced-Greedy = 89.0 ± 15.0, Legacy Greedy = 88.5 ± 15.5. 4pct → 3pct delta ≈ +2 pp uniform — saturation visible on most libraries. `outputs/full_benchmark_4pct/`.
+- **TT-TS vs TACTICS RWS is statistically tied at top-100 across budgets; TT-TS pulls ahead at wider top-N**: Nemenyi across-library rank test at top-100 returns p ≥ 0.29 at every measured budget (1/2/3pct). Within-library Tukey HSD at top-100 is significant on only 2–6/28 libraries depending on budget — adenine + amide HYBRID are the only libraries with Cohen's d > 0.9 differentiating the two methods. At top-2500 TT-TS sig > TACTICS RWS on 14/28 (3pct), 16/28 (2pct), 19/28 (1pct) libraries — the gap grows with recovery depth and with budget tightening. Practical framing: cite TT-TS as the per-library winner more often (19/28 at top-100, 22/28 at top-500), with materially higher worst-case reliability floor on hidden-hub libraries (adenine_hybrid 2pct: TT-TS min = 90 vs TACTICS RWS min = 26, a +64 pp floor advantage), rather than as a higher-mean winner — the aggregate-mean difference is within ±1 pp at the canonical reporting threshold.
+- **Polars + Python multiprocessing on Linux requires `forkserver`, not `fork` or `spawn`**: Linux default `fork()` deadlocks polars workers silently (rayon thread pool mutexes inherit into child without owning threads → first polars op hangs forever; the original 32-worker posthoc run hung 16+ hours with AveCPU = 1m12s before being cancelled). `spawn` resolves the deadlock but multiplies memory by worker count (each worker re-imports polars + numpy + scipy at ~1 GB overhead × N workers, so an 8-worker spawn run OOMed at 192 GB in 7 minutes). `forkserver` shares imports via COW from a clean intermediate process and combines spawn's deadlock-safety with fork's memory efficiency — confirmed working in `examples/run_posthoc_analysis.py` after 2026-05-25 patch (3pct/4pct posthocs completed successfully). `examples/run_diagnostic_benchmark.py` already used forkserver and was the empirical reference.
+- **Integrated 25-library benchmark — TACTICS beats Legacy on 24/25**: `outputs/full_benchmark_ema_cats/` extended on 2026-05-03 with post-merge adenine (FRED 4EIY, replacing pre-merge) and adenine_hybrid (HYBRID Lenselink-A2A, new library). Manifest: 218,500 trials across 25 libraries. Recovery summary: 2,185,000 rows (10 top-N × 25 libraries × ~8,740 trials/library). Best-TACTICS minus best-Legacy at top-100: 24/25 positive (range +0.1 to +18.8 pp); only orru tied at −0.1. Aggregate top-100 ranking: TACTICS TT-TS = 86.4 ± 15.7 (winner), TACTICS RWS = 85.9 ± 15.5, Legacy RWS = 82.9, Bal-Greedy = 82.0, Legacy TS = 81.1. Statistical framework: 248/250 within-library ANOVAs significant; 1,743/2,480 Tukey HSD pairwise comparisons significant (326 with large Cohen's d ≥ 0.8); all 10 across-library Friedman tests χ² > 66, p < 10⁻¹³. Post-gate-fix, TACTICS TT-TS is the top method on both adenine setups (FRED ≈ 95.6, HYBRID ≈ 94.0; 0 bimodal failures); the earlier "FRED axes → RWS, HYBRID axes → TT-TS" cross-pipeline method-flip framing is RETIRED (it rested on the now-removed TT-TS GMIC-gate artifact). Top-100 SMILES exported for both adenine setups (`outputs/adenine_fred_top100.csv`, `outputs/adenine_hybrid_top100.csv`); zero product-code overlap between the two top-100 sets. Manuscript notebook (`tutorials/manuscript_plots.py`) updated for adenine_hybrid in all per-library plots, SAR plots (parser shared with adenine), and a parallel "Adenine HYBRID — Top-N sensitivity" cell mirroring the FRED adenine cell. Pre-merge adenine search shards archived at `outputs/full_benchmark_ema_cats/search/_adenine_pre_merge/`.
+- **Budget sensitivity sweep 1-5% fully characterized + aggregation methodology corrected**: TACTICS-vs-Legacy advantage preserved at every budget (1%, 2%, 3%, 4%, 5%); largest advantage at constrained budgets (1-2%). Corrected per-library best-T computation (avg across libraries of per-library `max(TT-TS, T-RWS) − max(L-RWS, L-Grdy)`) gives different DOCK conclusions than prior aggregate-then-max. At 2% canonical: ROCS 2-comp +4.66±0.90, ROCS 3-comp +1.81±0.82, DOCK 2-comp +6.84±3.94, DOCK 3-comp +7.32±8.91 (3-comp actually exceeds 2-comp due to adenine bimodal-failure-threshold peak at +18.8 pp). DOCK aggregation has wide 95% CIs (±2-9 pp) reflecting per-library heterogeneity; ROCS aggregation is tight (±0.7-1 pp) reflecting per-library uniformity — the wide-vs-tight contrast is itself a manuscript finding. Two figures implemented as Section 4 of `tutorials/manuscript_plots.py` (`sensitivity_rocs.pdf`, `sensitivity_dock.pdf`), 2×2 line+bar panels each.
+- **Chained-decomposition attribution of ROCS recovery gains over Klarich-style TS**: Total TACTICS-vs-Klarich recovery gain pooled across 20 ROCS libraries is +5.25 pp at top-100 and stable at ~+5.2 pp across the full top-N range (25–250). At top-100 the credit splits 33% / 67% between Zhao 2025's RWS contribution (+1.73 pp) and TACTICS' algorithmic additions (+3.53 pp). Credit redistributes monotonically with top-N: by top-250 Zhao's contribution shrinks to +0.14 pp (3%) and TACTICS carries +5.05 pp (97%). On 2-component ROCS libraries, Zhao's contribution crosses negative at top-100 (−0.26 pp) and reaches −2.28 pp at top-250 — Legacy RWS underperforms Legacy Greedy at wider top-N due to symmetric thermal-cycling pathology — while TACTICS recovers this loss and adds further positive lift, keeping the net TT-TS-vs-Klarich gap at +5.0 pp. This is the strongest "what did TACTICS add beyond Zhao 2025?" picture in the manuscript and identifies a previously-undocumented failure mode of the field's state-of-the-art method.
+- **TT-TS and TACTICS RWS are practically equivalent on ROCS at the per-library level**: Per-cell empirical analysis across 200 (ROCS library × top-N) cells at the 2% canonical budget shows TACTICS RWS beats TT-TS on only 21/200 cells, with maximum gap 0.79 pp (poparov, top-25) ≈ 0.2 compounds. Persistent T-RWS wins are confined to one library (`betti`, 3-comp, ~0.5 pp gap at every top-N); narrow-top-N flips on `orru`/`poparov` (top-N 25–100, gap closes by 125); all other flips are < 0.1 pp (replicate noise). The maximum gap on any cell is well below the manuscript's ~2 pp practical-significance threshold, justifying headlining TT-TS as the TACTICS method on ROCS without a per-library method-selection caveat. The two TACTICS methods diverge on docking libraries (see hidden-hub analysis); the ROCS equivalence is specific to the similarity-scoring regime.
+- **Docking SAR landscape is library-dependent — enriched-hub interaction framework (Paper 1 docking section)**: each docking library is characterized post-hoc by its dominant enriched hub on two ground-truth axes — average compound quality (a reagent's mean = its average over all partner combinations) and partner-specificity (its score σ = variance over partners). TACTICS advantage over Legacy scales with hidden-hub structure (mean-mediocre and/or high-σ hubs that greedy's mean-exploitation misses); broadly-good low-σ libraries (amide-FRED, quinazoline) are greedy-reachable and all methods converge. No single-hub (mean, σ) marginal cleanly classifies greedy-easy vs hard (amide_hybrid hub z(mean) −2.20 is *better*-separated than quinazoline's −1.67 yet greedy fails amide_hybrid), so the landscape map is a structural summary/companion to the SAR scatters, NOT a classifier. Probe: `experiments/_debug/docking_landscape_probe.py` (M1 additive-R², M2 eff_hubs, M3 mean→hit ρ, M4 σ-discriminant, M5 GMIC+two-fraction, M6 per-method hub allocation; all M6 recovery numbers reproduce the canonical benchmark exactly).
+- **GMIC diagnostics characterized (MECH-2/4/5)**: (a) GMIC rises on the critical/exploited (high-GMIC, cooled) component as exploitation sharpens its mean-separation but stays flat on the flexible/explored (low-GMIC, heated) component because its reagent means are genuinely similar — exploration cannot manufacture mean-separation that isn't there (quinazoline negative control: all components rise together, GMIC inert, all methods tie ~85%); TT-TS separates components more sharply than RWS. (b) GMIC direction adapts per query on ROCS (groebke uses all 3 components as the flexible one across queries 24/51/34; rxn206 50/59) — a fixed rotation cannot match this.
+- **Docking method-winner framework — three greedy-failure points — selection / pairing / estimation (descriptive, post-hoc)**: On the 8 docking libraries, whether TACTICS beats best-Legacy is governed by three INDEPENDENT axes (any one sufficient): coverage failure = `max(0, 1.5 + z_mean)` of the dominant hit-holding hub (mean-blind hub → greedy starves it; adenine amidine_003 z_mean −0.58); allocation failure = `z_std × (1 − top1_share)` (partner-specific + dispersed → greedy can't cover the spread; amide_hybrid σ +6.8 / top1 0.13 → 5.9); reagent imbalance = `max/min reagent count` (search-dynamics, NOT SAR: under-observed large component → Legacy RWS thermal cycling breaks + greedy exploits noisy estimates; GMIC routes; thrombin 30:1). ≈0 on all three → all methods tie (quinazoline). WHICH TACTICS method is set by hidden-hub σ-character: wide-σ hubs → TT-TS (σ-disagreement specialist); a narrow-σ mean-good co-hub STABILIZES RWS to parity (its mean-OR-σ Boltzmann generalist reliably catches the mean-good co-hub: adenine FRED RWS 78.5→94.5) but does NOT make it win — TT-TS keeps the reliability floor either way (96.0→95.5). Post-gate-fix there is NO docking library where RWS reliably beats TT-TS; the earlier 'amidine_035 co-hub flips the winner to RWS' reading was a TT-TS-only GMIC min-observation-gate artifact (3 of TT-TS's 4 adenine failures), now removed. The 1.5 (conventional |z|≤1.5 separability threshold, carried from the dropped two-fraction metric) and the `z_std × (1 − top1)` composite are CONSTRUCTED and fit to the 8 libraries — descriptive landscape axes, NOT a prospective classifier; "run both, take max" stands. Mechanisms co-occur (adenine is coverage-failure AND 24:1 imbalanced); ROCS is a different scoring class (mean-separable → exploration-efficiency story, TT-TS≈RWS). Figure: `tutorials/manuscript_sar_plots.py` (landscape + z* sensitivity + greedy-reach cells). Card `66c29afe` rewritten as 'Three Greedy-Failure Modes'; supersedes the retired two-fraction framework.
+- **GMIC signal→decision→result — criticality routes component exploration (diagnostic)**: New 3-row diagnostic `plot_gmic_directed_exploration` makes GMIC component-routing visible end to end: SIGNAL (per-component GMIC) → DECISION (explored strip, with colorbar key) → RESULT (cumulative per-component coverage). Thrombin exemplar: GMIC reads acids resolved (≈2.0 → exploit → coverage saturates 100% by ~cycle 10), dipeptides unresolved (≈0.6 → explore ~63–69% of cycles → coverage climbs to ~27%); RWS and TT-TS route identically. In `src/TACTICS/library_analysis/diagnostic_plots.py` + cell in `tutorials/diagnostic_benchmark_plots.py` (loads `outputs/full_benchmark_5pct` thrombin).
+- **Thrombin is mean-accessible yet TACTICS wins via exploration efficiency**: Top dipeptides have clearly-best means (greedy CAN reach them, z_mean ≈ −14), yet TT-TS 81.8 > Legacy Greedy 77.3 (FRED). Cause: greedy acts on ESTIMATES, and the 3844-dipeptide component gets ~5 warmup obs each → noisy ranking → premature exploitation + under-exploration of the haystack; GMIC routes exploration to the under-resolved component. "Mean-accessible (ground truth) ≠ mean-identifiable within budget (estimates)." Distinct from Legacy RWS BREAKAGE (thermal cycling flattens on the under-resolved component → 68.8, worst). Advantage is modest (+4.5) because greedy is "right but slow"; contrast adenine (mean-blind, +19).
+- **TACTICS is reliably high on docking — 82–98% top-100 recovery, much lower variance than Legacy**: Across all 8 docking libraries at the 2% canonical budget, best-TACTICS recovers 82–98% of the top-100 with mean replicate σ ≈ 1.9 vs ≈ 6.3 for the best Legacy method. The variance gap is concentrated in the hidden-hub adenine pair where Legacy bimodal-fails (Legacy-RWS σ 19.8 adenine, 15.0 adenine_hybrid vs TACTICS 2.1 / 5.1); TACTICS is lower-variance on 6/8 libraries (the 2 exceptions are at-ceiling, σ≈0). The reliable recovery FLOOR — not merely a higher mean — is the docking headline for a one-shot screen. Heatmap: `outputs/full_benchmark_ema_cats/analysis/recovery_summary.parquet`.
+- **TT-TS GMIC min-observation gate was an artifact — removal validated, canonical 2% updated**: Removing the TT-TS-only GMIC min-observation gate (`TopTwoSelection._calculate_gmic` returned 0 when a component's least-observed active reagent had <5 obs; RWS had none) lifts adenine TT-TS top-100 87.3±16.2 → 95.5±3.3 (bimodal failures gone, now beats RWS 94.5); 28-lib aggregate unchanged (TT-TS 87.0→87.3), surgical to adenine. Validated by `full_benchmark_nogate` (218,800 trials, job 52894249); canonical `full_benchmark_ema_cats` updated in place by splicing gate-fixed TT-TS search (`examples/splice_gatefixed_ttts.py`; non-TT-TS byte-identical, 0/112 cells changed) + posthoc rerun.
+- **Docking: TACTICS holds an 81–98% recovery floor; the Legacy methods are dataset-specific (cross-library dispersion, not within-library)**: best-TACTICS top-100 spans **81–98%** across the 8 docking libraries (cross-library σ **7.2**), while Legacy Greedy swings **30–97%** (cross-library σ **25.1** = 3.5× TACTICS) and Legacy RWS 68–97% (σ 9.7, + bimodal failures); docking advantage **+7.1 pp** vs ROCS **+3.3 pp** at top-100. The variance the budget-sensitivity panels show as wide DOCK error bars is *cross-library dispersion of the Legacy methods*, not TACTICS instability — so the docking headline is TACTICS-consistency, NOT "TACTICS performance is library-dependent" (the library-dependent property is the ground-truth SAR structure). FRED-vs-HYBRID is treated as different ground truths, not a comparison (no claims). Verified 2026-06-08 from `full_benchmark_ema_cats` recovery_summary; figure = docking ranking heatmap.
+- **Docking layer diagnostics — per-library exploration mechanism verified end-to-end**: (a) **adenine** — component-level GMIC reads the amidine as resolved (≈1.9, exploited) and is BLIND to amidine_003 within it; TT-TS's *adaptive heated_scale inflates specifically on the amidine* (≈2.5 early → 1.4; disagreement 0.41→0.61), forcing the reagent-σ exploration that surfaces the hidden hub — the reason TT-TS > TACTICS RWS here (concretizes the GMIC observation-density bottleneck; isocyanide/aldehyde explored without inflation). (b) **thrombin** — strongest GMIC routing of any docking library: dipeptides stay low-GMIC (≈0.6, never resolve within budget at 3,844 reagents), explored ~65% of cycles with CATS pinned at 1.34, acids exploited (≈1.95, CATS 0.83). (c) **quinazoline** — flat per-component GMIC (spread 0.30 vs adenine 0.88 / thrombin 1.40) → no routing asymmetry → all methods tie (negative control). Verified 2026-06-08 from `full_benchmark_nogate` diagnostics; adenine/thrombin cards carry a "Layer diagnostics — what the figures show" paragraph.
+- **TACTICS RWS is bimodal on hidden-hub docking — 20-rep mean flips the RWS-vs-Legacy ordering**: On adenine/adenine_hybrid, TACTICS RWS recovers ~90 when its CATS exploration works but catastrophically fails (~16–36 top-100) on a fraction of reps; with 20 reps the failure count (hence the mean) is RNG-draw-dependent enough to flip TACTICS-RWS-vs-Legacy-RWS between independent runs (adenine_hybrid RWS 86.4±14.0/1-fail in ema_cats vs 77.5±26.1/4-fail in nogate; TT-TS 94.0±5/0-fail, stable in both). Report RWS on these libs with σ + catastrophic-failure rate, not bare means; reinforces TT-TS as the reliable hidden-hub method. [2026-06-09: card retitled 'Why TACTICS RWS Is Bimodal on Hidden Hubs' + a Mechanism section added — the catastrophic failures are a reagent-shape-agnostic Boltzmann lever meeting a within-component mean-blind hub that the component-level GMIC signal is structurally blind to ((a) blind by construction — mean-average reagent → no between-reagent spread → component reads resolved; (b) obs-density-bottlenecked → GMIC≈0/noisy on large components; (c) RWS surfaces the mean-blind hub only via a stochastic σ-tail lottery → bimodal); TT-TS's σ-disagreement lever is reagent-shape-aware → immune.]
+- **Gate-fix recovery impact is budget-dependent — surgical to a DIFFERENT library at each tight budget, inert once saturated**: Recomputed TT-TS top-100 recovery gated→gate-fixed directly from search shards (validated diff=0.000 vs published `recovery_summary`; gated baselines match STATUS lines 60–61 exactly), all 5 budgets. **2%**: only adenine moves (87.3→95.5, **+8.2**, σ 16→3 — the robust bimodal-failure-elimination; adenine_hybrid −0.5, all else ≤1.2). **1%**: the dominant mover is **adenine_hybrid (43.4→65.8, +22.4)** — in best-T terms 50.7→65.8 since gate-fixed TT-TS overtakes T-RWS (which was the 1% best at 50.7); adenine is −4.8 (61.4→56.6) but that is within-bimodal RNG (both ~57–61%, σ≈20), NOT a real regression; all else ≤0.4. **3/4/5%**: inert (all docking ≤0.7pp, dBestT≈0; adenine_hybrid saturates 99.9–100, adenine ~97–98). This **fills the gap in the 2026-06-04 same-seed "surgical" check** (which verified docking@3/4/5% ≤0.6pp + ROCS@1% ≤0.64pp but NOT docking@1%): the fix is "surgical to adenine at 2%, surgical to adenine_hybrid at 1%, inert ≥3%." **Plot consequence**: every 2%-canonical figure already reflects the fix via the spliced `full_benchmark_ema_cats` (only adenine's per-library cell moved); the ONLY figure still pending update when the 1/3/4/5% splice lands is the docking budget-sensitivity panel `sensitivity_dock.pdf`, and only at its 1% 3-comp point (adenine_hybrid best-T +15pp) — which STRENGTHENS the ≥50% top-100 floor-at-1% claim (worst case was adenine_hybrid 50.7 → now 65.8). `sensitivity_rocs.pdf` and all ROCS figures are unaffected at every budget. `experiments/_debug/gatefix_recovery_delta{,_345}.py`, `gatefix_45_dockonly.py`.
+- **Quinazoline per-method spread is real but the docking heatmap rendering hides it; TT-TS ≈ TACTICS RWS (ns)**: On quinazoline (FRED, 2%) top-100 the five methods are TT-TS 85.0 / TACTICS RWS 85.5 / Legacy Greedy 85.4 / Balanced-Greedy 85.3 / **Legacy RWS 83.1** — four cluster within ~0.5pp; Legacy RWS is the lone per-method laggard (−2.3pp) and its deficit GROWS with top-N (−4.1pp at top-250: L-RWS plateaus 83.1→83.2 while Greedy climbs 85.4→86.8 — symmetric thermal cycling wasting budget on mean-separable SAR, consistent with the L-RWS-worst-on-docking entries). TT-TS vs TACTICS RWS is statistically TIED (paired mean diff −0.55 compounds, 95% CI [−1.85,+0.75], Tukey p_adj=0.63, Cohen's d −0.20); the only significant top-100 pair is every method beating Legacy RWS (+1.9, p_adj=0.0001, ****). The "all methods similar on quinazoline" narrative is a BEST-OF-CLASS claim (best-TACTICS 85.5 ≈ best-Legacy=Greedy 85.4, +0.1 ns), NOT all-five-methods-equal. `docking_ranking_heatmap.pdf` LOOKS uniform only because it is top-100, integer-rounded (`f"{v:.0f}"`), on a GLOBAL 0–100 viridis scale (a 2–3pp within-library spread ≈ one shade); a 4–5pp T-RWS-vs-L-RWS gap visible in other plots is at top-50/250 (+4.4/+4.1), not top-100 (+2.4). quinazoline_hybrid IS genuinely all-methods-equal (97.0, at ceiling).
## Active Hypothesis
@@ -49,23 +83,21 @@
Only ONE active hypothesis per worktree at a time.
-->
-**Fixing the rotation-evaluation ordering and batch-granularity CATS adaptation should produce modest to medium improvement in TACTICS RWS recovery on 3-component libraries.**
+**Active thread (2026-06-03): gate fix RESOLVED; next = gate-fixed budget sweep.** The TT-TS GMIC gate fix is validated and the canonical 2% (`full_benchmark_ema_cats`) now carries gate-fixed TT-TS (see Established Results + Session Log 2026-06-03). The budget-sensitivity sweep below must be refreshed with gate-fixed TT-TS at 1/3/4/5%: `jobs/ttts_sweep_{array,merge}.sub` (TT-TS-only, 28 libs, fractions 1/3/4/5%, diagnostics on, posteriors skipped) are prepared but NOT submitted — run `./scripts/submit_benchmark.sh ttts_sweep` on HPC3, then per fraction `python examples/splice_gatefixed_ttts.py --target outputs/full_benchmark_Fpct --ttts-source outputs/ttts_sweep_Fpct --apply` + re-posthoc (non-TT-TS is gate-independent, reused; 2% already done via ema_cats).
-**Observations motivating this hypothesis (2026-04-04/05)**:
+**Half-budget union hypothesis (2026-05-22) — RESOLVED, does not generalize**. Union of `TT-TS-1pct ∪ RWS-1pct` top-N picks loses to best-single-2pct on 63.5% of (library × query × top_n) cells (mean diff −0.6 to −1.0 pp depending on top-N). Catastrophic on hidden-hub docking (adenine_hybrid −23 pp, adenine −20 pp): both methods *fail on the same replicates* at 1pct (warmup-observation density below the bimodal-failure threshold), so the union of two failures is still a failure. Small wins on imbalanced mean-separable docking (thrombin +1.9 pp, thrombin_hybrid +3.4 pp, both 100% win rate) and a few easy ROCS libraries — but the wins are an artifact of *each individual method already being saturated at 1pct on those libraries*, not of genuine diversification. The framework prediction (TT-TS catches σ-channel hubs, RWS catches GMIC-channel hubs → union helps most on hidden-hub libraries) is **wrong** because budget reduction shifts both methods below the criticality-signal density floor before any diversification benefit materializes. Records as [DEAD END] heptabase card; the original active-hypothesis prediction in this section is retracted.
-1. **Rotation-evaluation ordering is wrong**: In `sampler.search()`, `rotate_component_weighted()` (step 2) runs BEFORE `evaluate_batch()` and posterior updates (step 5). The rotation decision for cycle N+1 uses posteriors from cycle N-1 — the current batch's 100 evaluations are not incorporated. The rotation GMIC is one batch stale. Cycle-to-cycle GMIC changes average 0.008 (0.019 in early search where the lag matters most).
+**Active hypothesis (added 2026-05-22, session 3; expanded 2026-05-25 to five-budget sweep): Budget-sensitivity sweep — characterize the operating point.** The TACTICS-vs-Legacy algorithmic advantage is robust to (and slightly amplified by) budget reduction on every library class except the hidden-hub docking libraries, where the advantage persists but absolute recovery collapses due to per-reagent observation density falling below the criticality-signal floor. 1pct establishes that TACTICS dominance holds at constrained budget (≥50% top-100 floor on every library, +5.5/+10.1 pp lead over Legacy RWS on ROCS/docking aggregate). 2pct is the smallest budget at which hidden-hub libraries clear the bimodal-failure regime (canonical full_benchmark_ema_cats reference). 3pct and 4pct confirm saturation in the +4–5 pp / budget step range; 5pct is the abundant-budget bookend. As of 2026-05-25: 1pct ✓, 2pct ✓, 3pct ✓, 4pct ✓, 5pct rerun in progress (job 52673429, ~9h estimated remaining). Hypothesis is substantially confirmed at 4/5 budgets — TACTICS-vs-Legacy gap preserved at every budget tested, monotonically compressing from +5.5 pp (1pct ROCS) → +2.8 pp (3pct ROCS), and from +10.1 pp (1pct docking) → +7.7 pp (3pct docking). 5pct expected to be flat over 4pct on most libraries.
-2. **CATS multiplier is effectively noise on balanced 3-comp libraries**: Median multiplier = 1.000, P5-P95 range = [0.90, 1.17]. CATS contributes ~10% of the total temperature differential between heated and cooled components. Rotation (which component gets alpha vs beta) creates a 13× larger shift in Boltzmann selection probability than CATS at P90. The multiplier fluctuates cycle-to-cycle without developing persistent trends because it responds to instantaneous relative_gmic that barely changes per batch (std = 0.12–0.16 on these libraries).
+**Approach**: Five matched 28-library benchmarks (5 methods × 20 reps × top-N ∈ {25, 50, …, 2500}) at budget fractions 0.01 / 0.02 / 0.03 / 0.04 / 0.05. 1pct, 2pct, 3pct, 4pct posthocs all complete. 5pct array re-running on HPC3 with `--mem=192G` per task (the original 5pct array silently OOM-killed 2 of 14 tasks at the 96 GB ceiling during the per-library merge step — see [BUG] Slurm .sub Shells Mask Python Exit Codes). Compare same-method recovery curves across budgets and TACTICS-vs-Legacy gap across budgets; characterize headroom-to-ceiling per library.
-3. **Batch granularity suppresses CATS responsiveness**: CATS gets 195 update opportunities (one per cycle) across the entire search. All 100 compounds in a batch use identical temperatures. The divergence gate fires on 33% of early iterations (where adaptation would be most valuable) because batch updates produce larger posterior jumps than the per-iteration updates the gate was designed for.
+**Positive outcome (already partially observed)**: TACTICS-vs-Legacy gap *grows* at 1pct on most libraries (ROCS +5.5 vs +3.5 pp; docking +10.1 vs +8.5 pp on the aggregate; 2 of 3 hidden-hub libraries show gap *expansion*). ≥50% top-100 recovery floor for best-TACTICS holds on every benchmark library at 1pct. 3pct expected to show flat-or-near-flat gains over 2pct on most libraries, modest gains on the few libraries with residual headroom (adenine FRED TT-TS).
-4. **GMIC-weighted rotation IS working but the effect is small**: Most-flexible component heated 35.6–36.5% of the time vs 33.3% round-robin. This translates to +4–6 extra heating rounds (400–600 compounds) directed to the right component. On betti, the most-flexible component gets 69.6 heating rounds vs 60.6 for most-critical (round-robin: 65 each).
+**Manuscript framing implication**: Results section opens with the 1pct constrained-budget benchmark (TACTICS dominance at low budget, ≥50% floor everywhere, gap-expansion at low budget), then introduces 2pct as the operating point chosen to clear hidden-hub bimodal-failure threshold (with 3pct confirming saturation), then runs the canonical 28-library analysis at 2pct. This frames 2pct as a justified empirical choice rather than an assertion inherited from Zhao 2025.
-**Approach**: Two separable changes: (a) move `rotate_component_weighted` after evaluation/posterior update so the rotation decision for cycle N+1 uses cycle N's posteriors, (b) replace instantaneous relative_gmic with EMA-smoothed relative GMIC that accumulates directional signal across cycles. Test with 3-comp ROCS benchmark (10 libraries, 109 queries, 20 replicates): current vs ordering fix vs ordering fix + EMA CATS.
+Method-improvement directions that emerged from mechanism decomposition (2026-04-16) are scoped into **Paper 1b — TACTICS v2 Method Refinements** in the Future Manuscript Pipeline below. They are not in scope for the current manuscript.
-**Expected outcome**: Modest to medium improvement in recovery. Libraries with growing inter-component GMIC spread (poparov: 0.162→0.533, betti: 0.088→0.318, passerini: 0.150→0.397) are most likely to benefit. Libraries with uniformly easy SAR (orru, dobener) will not improve because the problem is absence of inter-component differential, not mechanism responsiveness. A null result would confirm that GMIC-weighted rotation alone is the full mechanism and CATS should be simplified or removed for batch processing.
-
-**Previous state (2026-04-02)**: Mechanism analysis and benchmark validation were complete. Per-query diagnostic analysis had confirmed GMIC adaptively detects which components need exploration. The smaller 3-comp advantage was attributed to easier SAR. This session's deeper temperature decomposition revealed that the CATS multiplier is not contributing meaningfully on these libraries due to batch processing constraints, refining the previous assessment that "CATS provides meaningful temperature modulation."
+**Docking-benchmark expansion — RETIRED 2026-05-31 (not pursued for Paper 1).** The 2026-05-30 engineering thread to add new combinatorial docking datasets (sEH ~60M + KinDEL MAPK14/DDR1 + ROCS-libs-docked-on-JNK3) was evaluated and dropped. (1) **sEH dead-ended**: the overnight pilot showed DEL read-count is strongly mean-separable (greedy beats TACTICS) — see Ruled Out / Dead Ends (card 24bd9c07); retained only as the TACTICS-v2/Paper-1b collapse-to-greedy diagnostic dataset. (2) **KinDEL declined**: no files local (fetch-from-S3, CC BY-NC-SA), MAPK14/DDR1 are redundant with JNK3 (kinase family already covered), single-pipeline (does not extend the FRED/HYBRID cross-pipeline contrast), and its enrichment/Kd objective carries the same mean-separability risk as sEH. (3) **JNK3-ROCS** was lowest-priority and is not needed — the 8 existing docking libraries already span the structural axes (2- vs 3-comp, balanced vs imbalanced, mean-separable vs hidden-hub, 2 protein families × 2 pipelines), and the cross-pipeline contrast (amide greedy-collapse 95→47 FRED→HYBRID, TT-TS +8.8 pp over Legacy) already rebuts "adenine carried it." Execution plan preserved for reference in `docs/handoffs/2026-05-30_docking-benchmark-expansion.md` (now superseded); recovered SMARTS recipes in `research/blueprints/seh_caix_smarts.md`. **New focus: docking narrative + figures.**
## Session Log
@@ -76,26 +108,53 @@
docs/handoffs/ and keep only the most recent 10 here.
-->
-- **2026-03-11**: _Template created. No experiments run yet._
-- **2026-03-14**: Implemented TT-TS and Bayesian additive decomposition. Ran full benchmark (65,430 trials, 21 libraries). TT-TS significantly outperforms RWS on 6/21 libraries (+0.6 pts overall). Additive decomposition adds zero value — removed from codebase. Ran manuscript benchmark (87,240 trials, 4 methods, 20 libraries). Key finding: GMIC criticality-weighted rotation is the primary mechanism (+4.3 pts on 2-comp, +0.9 on 3-comp over Legacy RWS), breaking the 2-component plateau. TT-TS adds modest gains especially on 3-comp. Next: manuscript preparation.
-- **2026-03-15**: Ran top-N sensitivity benchmark (697,600 rows, 8 thresholds 25–250). TACTICS advantage grows monotonically at wider top-N on all 20/20 libraries — criticality is more valuable for broad hit landscape than needle-in-haystack. Key discovery: Legacy RWS crosses below Legacy TS on 2-comp at ~top-150 (8/10 libraries), showing unguided thermal cycling is counterproductive at wider windows. Created marimo notebook for manuscript plots. Next: manuscript figure generation.
-- **2026-03-16**: Investigated TT-TS underperformance on thrombin (29.6:1 imbalance). Parameter sweep showed heated_scale is the sole cause: hs=1.0 recovers 78.6 (p=0.0005 vs Legacy TS), hs=1.5 = 73.5 (ns). Implemented bidirectional per-component adaptive heated_scale via EMA disagreement tracking in TopTwoSelection. Validated: adaptive TT-TS = 77.5 (p=0.0002 vs Legacy TS), matching RWS = 77.3. Fixed Legacy RWS budget bug on thrombin (negative nsearch from hardcoded 2%). Added TACTICS Balanced-Greedy as 5th benchmark method to isolate framework gains. Adopted naming convention: "Framework Warmup-Selection (modifiers)". Manuscript benchmark ready for cluster re-run with 5 methods.
-- **2026-03-16 (session 2)**: Analyzed completed 5-method manuscript benchmark (114,450 trials, 21 libraries). Adaptive TT-TS confirmed no regression: mean delta -0.02 vs fixed hs=1.5 across 20 ETS libraries, thrombin +2.1. Balanced warmup isolated: +1.5 on 2-comp (sig 10/11), +0.4 on 3-comp (sig 1/10) via greedy alone. Key insight: Legacy RWS is *worst* method on thrombin (68.8, below greedy 73.2) — thermal cycling is counterproductive on imbalanced libraries. Both TACTICS TT-TS (77.5) and RWS (78.2) still use Enhanced warmup, inheriting a 30:1 posterior observation imbalance. New hypothesis: Balanced warmup + TT-TS/RWS should further improve thrombin by eliminating this artifact.
-- **2026-03-16 (session 3)**: Ran thrombin warmup comparison (7 methods × 20 replicates, 29,220 budget). **Hypothesis rejected**: Balanced warmup significantly hurts — Balanced-TT-TS=72.5 vs Enhanced-TT-TS=78.0 (-5.5, p<0.0001), Balanced-RWS=73.5 vs Enhanced-RWS=78.5 (-5.0, p<0.0001). Also ran Enhanced-Greedy=73.2 (≈Balanced-Greedy=74.4, ns). Key mechanistic insight: Enhanced warmup's over-sampling of the small component pre-solves acid ranking; GMIC detects the differential posterior quality and redirects search to amines. Without GMIC the asymmetry is inert; with Legacy RWS's round-robin cycling it's harmful. The three-way interaction (warmup × information asymmetry × adaptive rotation) is the core finding. Manuscript methods confirmed as optimal. New open question: direct uncertainty-driven diversity sampling as alternative to thermal cycling (future work, not this manuscript).
-- **2026-03-17**: Spun out `library-diagnostics` worktree to investigate library diagnostic framework for follow-up manuscript. Candidate axes: Free-Wilson additivity, gap hardness, effective reagent rank, NK ruggedness. Goal: prescriptive diagnostics that help users design libraries amenable to TACTICS screening.
-- **2026-03-19**: Created full-results benchmark infrastructure (`_full_results_worker.py`, `run_full_benchmark.py`) that saves raw search data for post-hoc analysis. Ran 218,100 trials (21 libraries, 5 methods, 20 replicates) on HPC3 with 12 shards. All trials completed, zero errors, ~32 GB output. Merge job OOM-killed at 64G due to unnecessary dedup — fixed merge to skip dedup (shards disjoint by construction), bumped to 96G+16 CPUs. Recorded future manuscript pipeline (Papers 2a-4) in STATUS.md. Spun out `reagent-expansion-DrugEx` worktree for DrugEx integration / vendor mapping work (collaboration with Leiden University).
-- **2026-03-21**: Added aggregated Tukey HSD pairwise CI plot to manuscript_plots.py notebook. Discovered pooling ROCS and docking in one Tukey HSD inflates variance (adenine bimodality sd=27.6 dominates) — split into separate ROCS (20 libs, 2,180 query-level pairs) and docking (2 libs, 40 replicate-level obs) analyses. Traced TT-TS adenine rep 15 failure to amidine_003 under-exploration (0.6% search budget vs 4–15% in successful reps, caused by uninformative warmup posterior). Planned docking benchmark expansion: amide 10M + quinazoline 20M from Walters repo, plus ideally 1 additional target protein. Quinazoline enables cross-scoring-function comparison with ROCS niementowski. Next: prepare docking datasets, then manuscript writing.
-- **2026-03-21 (session 2)**: Built docking benchmark infrastructure for expanded libraries targeting JNK3 (PDB 2ZDT, same receptor as Klarich 2024). Selected diverse reagent subsets using BitBIRCH clustering + seed-anchored MaxMin diversity picking (`experiments/reagent_selection/select_reagents.py`). Amide library: 1,000 acids × 10,000 amines = 10M (reaction SMARTS from Walters repo, not rxn101). Quinazoline library: 376 aminobenzoic × 300 acids × 177 amines = 20M (niementowski 100-per-component reagents as seeds, expanded via BitBIRCH). Both libraries enumerated on HPC3 (10M amide 100% success, 20M quinazoline 100% success). Test docking of batch_00000 successful for both: amide 9,980 docked in 8 min, quinazoline 11,155 in 5 min (FRED Chemgauss4, 48 MPI cores, 200 conformers). Bumped to 400 conformers for production runs. Docking target is JNK3 (kinase), giving three protein families (thrombin=protease, adenine=GPCR A2B, JNK3=kinase). Scripts: `data/docking/{enumerate_library.py, dock_batch.sub, submit_docking.sh}`. Next: submit full docking arrays, then process scores into TACTICS benchmark format.
-- **2026-03-19 (session 2)**: Added adenine docking library (29×688×294, 4.9M scored products, FRED Chemgauss4, 17% dock failures). Standardized all data formats: thrombin and adenine converted to `{lib_id}.parquet` with `[Product_Code, Scores]`, reagent files renamed to `{lib_id}_reagent_{i}.smi`, reactions.csv updated with thrombin + adenine SMARTS. Refactored `run_full_benchmark.py` to auto-discover libraries from data directory (eliminated hardcoded library sets and thrombin special-casing). Ran adenine benchmark (100 trials, 20 reps, 5 methods). Created `run_posthoc_analysis.py` with rigorous statistical framework (ANOVA + Tukey HSD + Friedman + Nemenyi + Cohen's d) following Ash et al. 2025. Debugged two post-hoc issues: (1) merged + shard files both loaded causing double-counting — fixed by preferring merged file; (2) NaN scores from docking failures sorted to top of results by Polars — fixed by filtering NaN before ranking. Key finding: TT-TS is exceptionally robust to docking failures (93.9 on adenine, 19/20 reps at 97), while RWS shows bimodal behavior and greedy plateaus at ~32. The uniform distribution of failures across reagents preserves SAR signal — concentrated failures would create blind spots.
-- **2026-03-24 to 2026-03-26**: Deep analysis of Riley (2024) posterior separation methodology and connection to TACTICS results on both docking and ROCS. Major manuscript_plots.py overhaul (pushed to GitHub): (1) data path moved to `outputs/manuscript_analysis/post_hoc`, (2) posterior separation plots: continuous colormap by hit count, z-scored axes, data-driven colorbar, (3) multi-top-N Riley-style grid plots (top-25–250) for all 4 docking + 4 ROCS libraries, (4) gain decomposition split into ROCS (baseline=Legacy RWS) and Docking (baseline=Legacy Greedy) with scoring-type-appropriate baselines, (5) per-library grouped difference bars for TT-TS and RWS(GMIC) vs appropriate legacy baseline, (6) docking top-N sensitivity fixed to use Legacy Greedy baseline. Key findings: SAR additivity governs greedy reachability (docking quinazoline z≈-2.0 vs adenine z≈-0.75); ROCS SAR erodes at wider top-N as first series exhausts; Legacy RWS single-component exploration bottleneck explains 2-comp crossover. Detailed mechanism walkthrough of GMIC (3 layers: weighted rotation, divergence gate, CATS multiplier) and TT-TS (posterior widening/contraction, two-sample disagreement, adaptive EMA). Established correct terminology: TT-TS uses "posterior widening" and "contraction" not "heating/cooling" — no Boltzmann temperature involved. Created 4 Heptabase cards: (1) docking SAR additivity, (2) ROCS SAR erosion + RWS bottleneck, (3) RWS GMIC mechanism walkthrough, (4) TT-TS GMIC mechanism walkthrough. Next: run instrumented searches on HPC3 with track_diagnostics=True to generate per-iteration GMIC/disagreement data for mechanism visualization plots.
-- **2026-03-26 (session 2)**: Fixed TT-TS diagnostics schema mismatch — `disagreement_ema`, `heated_scale`, `effective_scale` were silently dropped when TT-TS records were coerced into GMIC schema. Added `_TTTS_DIAGNOSTICS_SCHEMA`, detection logic, `criticality` alias, 2 analysis functions (`compute_disagreement_convergence`, `compute_scale_adaptation`), 3 plot functions, 13 new tests (43 total). Built diagnostic benchmark infrastructure: `run_diagnostic_benchmark.py` (110 trials: adenine, quinazoline, rxn206, mannich, orru × RWS/TT-TS × 5 reps), ROCS query selection at p10/p50/p90 of advantage distribution, and Slurm job file using `stage_benchmark_data.sh` convention. Selected libraries span the difficulty spectrum: rxn206 (+7.6 advantage), mannich (+3.9), orru (-0.4, negative control). New active hypothesis: mechanism activation scales with query difficulty. Next: push to HPC3, submit `sbatch examples/run_diagnostic_benchmark.sub`, then build visualization notebook from results.
-- **2026-03-27**: Analyzed diagnostic benchmark results (110 trials, all 5 libraries, TT-TS diagnostics fully captured). Original hypothesis partially rejected: mechanism does NOT show sustained activation on hard queries — it fires early and quiets down. Discovered adaptive heated_scale runaway on balanced 3-comp libraries: one component converges fast → EMA drops below 0.3 → uncapped growth inflates scale to 2–5× → trapped in neutral zone. Consistent across all replicates (structural, not stochastic). Considered 3 fixes: (1) raise low_threshold — unsafe, suppresses beneficial inflation on adenine, (2) max_growth_per_step=0.1 — safe, caps runaway without preventing beneficial growth, (3) GMIC convergence gate — unsafe, can't distinguish additive from non-additive SAR. Implemented fix 2 as new default. Added `--methods` filter to `run_full_benchmark.py`. Prepared TT-TS-only validation benchmark (24 libs, 20 reps, 480 trials). Next: submit `sbatch examples/run_ttts_validation.sub`, compare against existing baseline.
-- **2026-03-27 (session 2)**: Built unified benchmark infrastructure. Replaced 8+ copy-pasted benchmark scripts with single `examples/run_benchmark.py` (run/estimate/merge subcommands) + `_benchmark_config.py` + `_benchmark_slurm.py`. Created `/benchmark` skill. Discovered TT-TS validation job (50251973) has insufficient wall time.
-- **2026-03-22**: Completed brute-force docking for amide (10M) and quinazoline (20M) on HPC3. Converted scores to parquet. Ran 5-method benchmark (200 trials, 20 reps). Post-hoc analysis complete (24 libraries, 2.18M recovery rows).
-- **2026-04-02**: Investigated multi-component heating hypothesis. Diagnostic data showed 2+ components below GMIC 1.0 simultaneously is rare on 3-comp (3% of query-phases). More importantly, 2-comp libraries where BOTH components have GMIC < 1.0 still benefit strongly from single-component heating (+7.6 on rxn206) — the explore/exploit contrast itself drives gains. Deferred continuous temperature gradient to follow-up study. Built manuscript-quality diagnostic plot functions (`plot_rws_diagnostic`, `plot_ttts_diagnostic`, `plot_min_gmic_vs_advantage`) in `diagnostic_plots.py` with 95% CI bands across replicates, rolling-averaged temperature, divergence gate background shading, and scalable colour palette. Created marimo notebook `tutorials/diagnostic_benchmark_plots.py` rendering all 21 diagnostic figures. Cleared active hypothesis — manuscript ready for writing.
-- **2026-04-04/05**: Two-session deep dive into GMIC mechanism on 3-comp ROCS libraries. (1) Decomposed GMIC signal/noise: noise denominator inflated 2-12× by unexplored reagents (recorded as Paper 3 open question + Heptabase card). (2) CATS multiplier is effectively noise on balanced 3-comp: median=1.000, contributes ~10% of temperature differential, 13× smaller Boltzmann shift than rotation alone. (3) Identified rotation-evaluation ordering bug: rotation for N+1 uses N-1 posteriors (100 evals stale). (4) Batch granularity suppresses CATS: 195 update opportunities vs 19,500 at batch=1, divergence gate fires 33% in early search. (5) Mechanism decomposition: GMIC rotation produces +2.2 net over legacy cycling on mannich (positive 79% of queries), -0.8 on orru (uniformly easy SAR). TACTICS advantage is net-positive across the query distribution: gains on hard queries, neutral on easy. New active hypothesis: fixing rotation ordering + EMA-smoothed CATS should give modest-to-medium improvement.
-- **2026-04-04**: Decomposed GMIC signal/noise dynamics on all 10 three-component ROCS libraries using full benchmark diagnostics (1.27M rows per library, 109 queries × 20 replicates × 195 cycles). Key finding: GMIC noise denominator is inflated 2-12× above expected (uniform observation) levels because ~85-90% of reagents remain at warmup-level variance while directed search concentrates observations on ~10-15%. This is a limitation of GMIC: it underestimates SNR for the exploited subpopulation. Per-query trajectory classification: signal crash magnitude predicts GMIC direction (r=-0.595, mannich). DOWN trajectories = false positive SAR from warmup (92% signal crash, self-correcting). UP trajectories = real SAR (35-47% signal retained). Mannich (Δ=+2.5): 171 UP, 45 DOWN, 111 FLAT → differential GMIC drives advantage. Orru (Δ=-0.5): 268 UP, 26 DOWN → uniformly easy SAR, no differential. Created Heptabase card with proposed observation-weighted GMIC variant and reagent usage fraction diagnostic plot. Recorded as Paper 3 open question; Paper 1 discussion point (TACTICS outperforms legacy even with conservative estimates).
-- **2026-04-01**: Deep dive into GMIC-RWS mechanism using diagnostic benchmark data (110 trials, 5 libraries, 11 queries). **Key findings**: (1) GMIC direction is query-specific — different query molecules on the same library produce opposite GMIC directions (rxn206 query_008 vs query_090). The "near-uniform" rotation probabilities are an artifact of averaging across queries with opposing directions. (2) 5/9 ROCS queries show GMIC redirection mid-search, tracking the transition from first SAR series to second-tier reagents. (3) The CATS multiplier provides meaningful modulation even on balanced libraries (43-56% of iterations active, range [0.025, 0.20] effective temp). The alpha/beta ratio defines both the base asymmetry AND the CATS dynamic range — they are one integrated mechanism. (4) min_GMIC (weakest component) predicts TACTICS advantage: <1.0 → large gains, >1.5 → negligible. (5) The smaller 3-comp advantage is due to easier SAR (higher min_GMIC), not mechanism malfunction or the heated_scale "runaway." Retired the max_growth_per_step hypothesis, disabled the growth cap (default=None). Created 2 Heptabase cards: mechanism walkthrough and diagnostics analysis. New hypothesis: multi-component heating to address the structural single-component constraint on 3+ comp libraries.
+_Entries before 2026-05-03 archived to `docs/handoffs/session-archive-{2026-03,2026-04,2026-05}.md`._
+
+- **2026-06-09 (cont.)**: Corrected the docking Q2 (TT-TS-vs-RWS) narrative for the post-gate-fix world + supplied the RWS-bimodality mechanism. (1) **Trashed the 'Why the Winner Flips' card `79065473`** (soft-delete, restorable) — the 'flip' it explained (adding amidine_035 flips adenine FRED winner TT-TS→RWS) was ~75% a TT-TS-only GMIC min-observation-gate artifact: pre-gate-fix TT-TS 87.3 (3–4 bimodal fails) < RWS 94.5, but the gate caused 3 of those 4 fails; post-gate-fix TT-TS 95.5 ≥ RWS 94.5 — no flip. The amidine_035 counterfactual was already retired 2026-06-03; the card was a leftover straggler (so was the framework's Q2 paragraph). (2) **Corrected the Q2 paragraph of the framework card `66c29afe`**: a narrow-σ mean-good co-hub STABILIZES RWS (adenine FRED 78.5→94.5; its Boltzmann generalist reliably catches a mean-good reagent) up to parity but does NOT make it win; TT-TS holds the reliability floor (96.0→95.5, stable either way); post-gate-fix no docking library has RWS reliably beating TT-TS. Established Result line ~67 amended to match. (3) **Rewrote `19f566ab`** — retitled *'Why TACTICS RWS Is Bimodal on Hidden Hubs'* (was '…20-Rep Mean Flips the RWS-vs-Legacy Ordering') and added a **Mechanism** section answering 'why does the more sophisticated GMIC-RWS still catastrophically fail?': the hub is a within-component, mean-blind, high-σ reagent; GMIC is a COMPONENT-level signal that is (a) blind to it by construction (mean-average → no between-reagent spread → component reads resolved → heat routed away), (b) weakest exactly here (obs-density bottleneck → GMIC≈0/noisy on large components — HYBRID failure reps give the aldehyde-137 component 902 evals vs 18,853), and (c) RWS's within-component Boltzmann lever is reagent-shape-AGNOSTIC, surfacing a mean-blind hub only via a stochastic σ-tail lottery → bimodal. TT-TS's σ-disagreement lever is reagent-shape-AWARE → reliably surfaces the hub → immune; GMIC+CATS only removes legacy RWS's symmetric-cooling pathology (TACTICS RWS fails less than legacy), not the within-component lottery. **Working/Discussion set now 10 cards.** **Next** (unchanged): wire href cross-links `66c29afe`→case studies; place the docking set into §3.2.x whiteboard sections.
+- **2026-06-09**: Discussion-card restructuring toward a manuscript whiteboard outline + working-set tagging. (1) Tagged all 12 TACTICS Discussion-section cards (`Project-TACTICS-Primary-Engine` ∩ `Section-Discussion`) with the transient `working` tag as the session's under-discussion set (`lab-notebook` skill, `research_card.py working`). (2) **Docking discussion consolidated to a single framework card.** Kept `[DISCUSSION] Docking Reagent-Score Landscape — Three Greedy-Failure Modes` (`66c29afe`) as the sole interpretation/framework card — it already subsumes the metric reading + Q1 coverage/allocation/imbalance + Q2 σ-character + the amide→amide_hybrid σ-control + the reliability headline + the z*=1.5 defense. **Trashed 3 redundant explanation cards** (soft-delete, restorable): `63e65a69` (Reading the Docking Landscape — mean/σ), `01ad47d0` (SAR Additivity & Posterior Separation), `d868cd7e` ([RESULT] Docking SAR Landscape Is Library-Dependent — its "library-dependent performance" framing already retired for the consistency result `ca8961ba`). New structure: ONE failure-mode framework (`66c29afe`) → **case studies as examples of each mode** — adenine=coverage (`523bd90b`), amide_hybrid=allocation (`359f603e`), thrombin=imbalance (`548d7583`), quinazoline=greedy-works/negative control (`c78d8256`); `79065473` (Why the Winner Flips) was initially kept here as the Q2 elaboration, then **trashed later the same day (see the 2026-06-09 (cont.) entry)**. Unique content to re-home during the prose pass: `01ad47d0`'s Riley-2024 connection + z-erosion tables → into `66c29afe` or the Methods SAR card `ebee37ac`; `d868cd7e`'s ranking-heatmap figure → `ca8961ba`. (3) **ROCS `[DISCUSSION] Legacy RWS as Baseline` (`e6614eb7`) flagged inaccurate** — its blanket "Legacy RWS is THE legacy baseline" conflicts with the best-TACTICS − best-Legacy methodology (its own body shows 2-comp ROCS is a 4/6 Greedy↔RWS split, so best-Legacy is sometimes Greedy); the valid kernel (Legacy RWS = the Zhao-2025 comparator for the *incremental* contribution) is already carried by the chained-decomposition card `05cf848e`. Resolution: **TRASHED 2026-06-09** (user-confirmed) — redundant with the chained-decomposition card `05cf848e`, and best-of-class (best-TACTICS − best-Legacy) already neutralizes the compare-to-RWS-not-TS concern. ROCS discussion set is now 5 cards (`1e5691da` Zhao open-question centerpiece, `bf67bcd4` top-N decay, `cf1c14a9` layer diagnostics, `d7996148` practical-vs-statistical, `cfc20ceb` efficient-learning/narrow-top-N). Goal stated: whiteboard = initial outline → text-based outline → manuscript draft. **Next**: wire href cross-links from `66c29afe` to the 4 case-study cards (currently inline-prose ties only); place the consolidated docking set into the §3.2.x whiteboard sections.
+- **2026-06-08**: Built the docking manuscript §3.2 card set + a reusable card-pipeline workflow; verified the docking mechanism narratives against raw data. (1) **Global `lab-notebook` skill** (`~/.claude/skills/lab-notebook/`: `SKILL.md` + `scripts/research_card.py` with create/triage/promote/link/working + a conventions reference) — one-command Heptabase card create+tag+cross-link+place, config-driven per project (`/.claude/lab-notebook.json` = project_tag + manuscript/lab-notebook whiteboard ids; user-global `~/.claude/lab-notebook.json` = workspace). Implements the capture→triage→promote lifecycle (Type-Experiment lab-notebook lane → triage → Type-Manuscript). (2) **Docking Results §3.2 drafted as 8 cards** (budget → heatmap → case studies → discussion): NEW — *Docking Budget Sensitivity* (§3.2.1), *TACTICS Is Well-Suited and Consistent on Docking* heatmap (§3.2.2), *Amide-HYBRID — Allocation Failure* (§3.2.4), *Quinazoline — Greedy-Accessible Negative Control* (§3.2.6); AMENDED — adenine (`523bd90b`) + thrombin (`548d7583`) each got a "Layer diagnostics — what the figures show" paragraph; RECLASSIFIED to [DISCUSSION]+Section-Discussion — *SAR Additivity and Posterior Separation* (`01ad47d0`, was [RESULT]) and *Three Greedy-Failure Modes* (`66c29afe`, was [METHOD]) as the two halves of the docking discussion. Case studies map to failure modes: adenine=coverage, amide_hybrid=allocation, thrombin=imbalance, quinazoline=control. (3) **Earlier this session**: ROCS Tukey card `a306fcfc` repurposed into the main-text bar-plot significance card (R3 `8b668f51` retired/merged in); ROCS §3.1.7 layer-diagnostic cards created (`1930687a` result + `cf1c14a9` discussion). (4) **Empirical (data-verified)**: docking consistency reframe (81–98% σ7.2 vs Greedy 30–97% σ25.1, 3.5×; +7.1 vs +3.3 pp ROCS); adenine σ-channel hidden-hub finding (heated_scale 2.5 on amidine, GMIC blind); thrombin GMIC dipeptide routing; quinazoline flat GMIC (0.30); amide_hybrid allocation (greedy 47 / TT-TS 84). **FRED-vs-HYBRID narrative dropped (no claims); amide_hybrid hybrid-only in main text.** (5) **4% re-run COMPLETE** (218,800 trials, 0 short cells; recovery_summary current; budget plots `manuscript_plots_{docking,ROCS}.py` point at `outputs/full_benchmark_4pct/...`) — 4% docking gaps shift slightly on the complete data (2-comp +5.5, 3-comp +2.4 vs the card's +5.1/+3.1). (6) **Notebook fixes** (`diagnostic_benchmark_plots.py`): MECH-4 + docking layer plots now label components with reaction chemistry (quinazoline = Aminobenzoic acid/Amine/Carboxylic acid; adenine = Amidine/Isocyanide/Aldehyde; thrombin = Acids/Dipeptides); fixed an imports-cell corruption (imports wrapped in `mo.md` → "mo undefined"; `ast.parse`/import both still passed). (7) **Conventions recorded** (memory): reference cards by title not UUID; every [RESULT] card needs a figure/table + a "what to take from the figure" paragraph, pure-explanatory cards → Discussion. **Next**: place the 8 docking cards into their §3.2.x whiteboard sections + retire *Docking SAR Landscape Is Library-Dependent* (`d868cd7e`, library-dependent framing dropped, pending — needs explicit trash authorization); reconcile the §3.2.1 budget card's 1–5% series against the current gate-fixed per-budget recovery_summaries; confirm/generate amide_hybrid reagent-score-distribution + layer figures on the board.
+- **2026-06-06**: Manuscript Heptabase card cleanup + STATUS reconciliation. Trashed 7 outdated/false [RESULT] cards — EMA-smoothed-CATS reliability, Statistical-Framework-Validation, GMIC-Contribution-92%, FRED-vs-HYBRID per-amidine coverage, TT-TS-adenine-failure (rep-15), min_GMIC-predictor, amidine_035-counterfactual (all soft-deleted/restorable, removed from the TACTICS-I whiteboard). Rewrote the never-worst card `32f0a5f5`: dropped the 'Pareto Floor' framing, reframed to TACTICS-beats-Legacy across structure-based (docking) + ligand-based (ROCS) scoring, merged in the FRED/HYBRID cross-scoring side note (legacy sensitive / TT-TS consistent; the cross-pipeline 'method-flip' framing is RETIRED as a gate artifact). Reclassified the gate-fix card `a5457730` from manuscript [RESULT] → experiment card (dropped prefix; Type-Manuscript + Section-Results → Type-Experiment). Corrected the false 'EMA eliminated the bimodal failures' attribution in the two EMA [METHOD] cards `4e272cf7`/`78aef3bb` (credit → gate removal; RWS still bimodal). Created recovery-performance reference card `6a0e97d5` (docking + ROCS top-100 at 1%/2%, per-method + per-library tables). STATUS Established Results reconciled and all references to the deleted cards removed. **Next:** optionally reframe `7c446b85` (25-lib headline card, still carries cross-pipeline framing internally); confirm demo queries (rxn206 2-comp + mannich 3-comp) for the layer-1/2 diagnostic manuscript figures.
+- **2026-06-05 (cont. 2)**: Diagnosed the missing 1% ROCS points in `sensitivity_rocs.pdf` — NOT data loss. The 2026-06-04 `full_benchmark_1pct_splice_posthoc.sub` (under-provisioned at 96G/32w) OOM-killed the workers for all 21 large-parquet libs (20 ROCS + quinazoline_hybrid); `run_posthoc_analysis.py` swallowed the `BrokenProcessPool` errors, wrote a 7-docking-lib `recovery_summary` (10,500 rows vs 3,282,000), and exited 0 — silently overwriting the good 28-lib summary (Slurm reported COMPLETED). Blast radius is **1pct ONLY** (2/3/4/5pct + ema_cats intact; sibling subs already 248–370G/2–4w); search data + manifest fully intact; ROCS is gate-independent. Fixed `jobs/full_benchmark_1pct_splice_posthoc.sub` → 248G/4w (mirrors 3pct); splice is idempotent (`_pre_splice_ttts_backup` exists) so re-run only re-executes the posthoc. New [BUG] card `cd839e24` + Ruled-Out entry. **RESOLVED same session**: HPC3 resubmit cancelled (cluster backlog — 460+ jobs ahead) and the posthoc re-run **locally serial** (`--workers 1`, 21.5 min, no OOM possible) — `outputs/full_benchmark_1pct/analysis/recovery_summary.parquet` restored to 28 libs / 3,282,000 rows (all 20 ROCS present; rxn206 TT-TS top-100 = 0.810; aggregate top-100 unchanged TT-TS 81.0 / RWS 80.1 / L-RWS 75.5, matching the established 1pct numbers). `sensitivity_rocs.pdf` 1% column recoverable on a notebook data-cell reload. **Still open:** add an exit-on-partial guard to `run_posthoc_analysis.py` (raise/exit-nonzero before writing `recovery_summary` when `len(successful_libs) < available_libs`) so a future under-sized job can't silently overwrite a good summary.
+- **2026-06-05 (cont.)**: Quinazoline figure-interpretation clarification + diagnostics-notebook reorganization. (1) Resolved an apparent quinazoline contradiction (heatmap looks uniform but other plots show method gaps): the docking ranking heatmap is top-100, integer-rounded, on a global 0–100 viridis scale → hides the real per-method spread; TT-TS≈TACTICS RWS is tied (mean diff −0.55, p_adj 0.63, d −0.20), Legacy RWS is the lone per-method laggard (−2.3pp@100 → −4.1pp@250), and "all methods similar" is a best-of-class statement (best-T 85.5 ≈ best-Legacy=Greedy 85.4). New Established Results entry. (2) Reorganized `tutorials/diagnostic_benchmark_plots.py` (26→19 cells): removed min_GMIC-vs-advantage + GMIC-vs-Oracle + MECH-2 (2+3-comp) + MECH-5 cells and their now-unused imports (rationale: GMIC/min_GMIC alone is not a sufficient *standalone* explanation of library outperformance); kept MECH-4 + the docking Layers-1&2 dropdown; ADDED a "Layers 1 & 2 for Any ROCS Library" section — dropdown over all 20 ROCS libs, each at a representative query selected by `experiments/_debug/pick_rocs_demo_queries.py` (max late-search between-component GMIC separation on rep 0, cross-replicate consensus = 1.0; reproduces the prior MECH-4 manual picks rxn206→q047, groebke→q035). ROCS 2% diagnostics from `full_benchmark_nogate`. Verified: ast.parse clean, functional render passes for 2-comp + 3-comp + folder-remapped (amide-suzuki→`amide_suzuki`, groebke→`groebke`, Local Norm #11) libs. The `plot_min_gmic_vs_advantage`/`plot_gmic_vs_oracle`/`compute_oracle_flexibility` *functions* in `diagnostic_plots.py` are retained (only their notebook cells removed). Pre-edit backup `tutorials/diagnostic_benchmark_plots.py.bak` (notebook untracked in git). **Next**: user reviews the ROCS section in `marimo edit`; delete `.bak` once confirmed.
+- **2026-06-05**: Verified the gate-fix's recovery impact is **budget-dependent** while waiting on the HPC3 splice jobs (recomputed TT-TS top-100 gated→gate-fixed straight from search shards — `ttts_sweep_{1,3,4,5}pct` gate-fixed source + `full_benchmark_{3,4,5}pct`/splice-backup gated baseline are all already local; validated diff=0.000 vs `recovery_summary`, gated baselines match STATUS 60–61). **2%**: only adenine moves (+8.2, σ16→3). **1%**: dominant mover is adenine_hybrid (43.4→65.8, +22.4; best-T 50.7→65.8); adenine −4.8 is bimodal RNG not regression. **3/4/5%**: inert (saturated ≥96%). Fills the docking@1% gap in the 2026-06-04 same-seed "surgical" check. **Consequence (answers the open notebook question)**: all 2%-canonical figures already reflect the fix via the spliced ema_cats; the ONLY plot still needing a refresh when the splice lands is `sensitivity_dock.pdf` (and only its 1% 3-comp point, adenine_hybrid best-T +15pp — strengthens the ≥50%-floor-at-1% claim); `sensitivity_rocs.pdf` + all ROCS/per-library/SAR/diagnostic figures unaffected. Amended Heptabase `a5457730`. **Next**: when the 1/3/4/5% splice+posthoc land, regenerate `sensitivity_dock.pdf` only and confirm the 1% adenine_hybrid lift renders; no other figure re-run needed.
+- **2026-06-04**: Gate-fixed budget sweep submitted to HPC3; greedy-vs-RWS failure-mode characterization; single-component-exploration correction. (1) **Sweep submitted (CPU)**: `jobs/full_benchmark_{1,3,5}pct_splice_posthoc.sub` (splice gate-fixed TT-TS from `ttts_sweep_Fpct` via `examples/splice_gatefixed_ttts.py` + re-posthoc; added `--match-target-keys` flag) and a **full re-run of the OOM-incomplete 4pct** (`full_benchmark_4pct` was ~84% complete across ALL methods — 2/13 array shards lost to a silent OOM at `--mem=96G`; re-run uses current gate-fixed code so TT-TS is gate-fixed directly, no splice; mem matched to 5pct — array 192G/tmp150G, merge 96G, posthoc 370G; exit-guards added to array+merge). 2pct already gate-fixed (`ema_cats`). GPU declined (CPU-bound work; 248–370G posthocs exceed GPU-node RAM). Verified gate fix is **surgical**: same-seed docking deltas ≤0.6pp at 3/4/5pct, only adenine@2pct moves materially (σ 18.4→10.6 same-seed); ROCS ≤0.64pp at 1pct (max-gate-firing budget) ⇒ unaffected at all budgets; nogate TT-TS still beats Legacy every fraction (docking aggregate +5.6 to +12.3pp). (2) **Diagnostic notebook repointed**: `diagnostic_benchmark_plots.py` from stale gated `ema_cats` diagnostics → `full_benchmark_nogate` (gate-fixed, merged per-library; `load_diag_union` now prefers merged files; 2nd resolver prepends nogate). (3) **SAR probe refreshed**: re-ran `docking_landscape_probe.py` (reads gate-fixed `ema_cats/search`) → `manuscript_sar_plots.py` winner-fill flips adenine RWS→TT-TS (TT-TS 95.5 ≥ RWS 94.5). (4) **Greedy-vs-RWS = complementary failure modes** (canonical 2pct, Legacy gate-independent): greedy fails on mean-hidden hubs / dispersed-allocation libs exactly where Legacy RWS's blind exploration HELPS (L-RWS−L-Grdy: adenine +29.8, adenine_hybrid +54.2, amide_hybrid +27.6); Legacy RWS fails on mean-separable libs where greedy exploits (amide −8.4, quinazoline −2.3); the **imbalance axis (max/min count) is an RWS failure mode** (thrombin: Grdy 77.3 fine, RWS 67.7 broken), currently MISLABELED as a "greedy failure mode" in the reagent-score-landscape → re-attribute. Root: greedy = mean-only exploit (blind to uncertainty); Legacy RWS = signal-blind round-robin (blind to criticality); complementary; TACTICS wins by being signal-adaptive. (5) **CORRECTED Established Results entry (line ~42)**: single-component exploration is a SHARED limitation, NOT a TACTICS improvement — verified all three methods heat one component/cycle (`is_heated = component_idx == self.current_component_idx`; `roulette_wheel.py:261`, `top_two_selection.py:131`); TACTICS only redirects the single heated slot. Added Open Question (simultaneous multi-component exploration / candidate spin-off); the line-42 Heptabase card amended. **Next:** when HPC3 jobs land, re-run notebook data-load cells (docking + diagnostic); re-attribute the landscape's imbalance axis to RWS; consolidate the three RWS-failure manifestations (over-explore solved components / round-robin imbalance miscalibration / single-component bottleneck) into a manuscript limitations subsection.
+- **2026-06-03**: TT-TS gate fix validated at full-benchmark scale; canonical 2% updated; counterfactual retired. (1) `full_benchmark_nogate` (job 52894249, 218,800 trials, gate removed) confirmed the gate was an artifact: adenine TT-TS 87.3±16.2→95.5±3.3 (bimodal failures gone, beats RWS 94.5); 28-lib aggregate unchanged (TT-TS 87.0→87.3); TT-TS top-or-tied on all 8 docking libs; docking Welch 95/112 sig, ROCS Tukey TT-TS>Legacy 16–20/20, Friedman TT-TS rank 1. (2) **Cross-benchmark nogate-vs-ema_cats is confounded by run-to-run RNG** on bimodal hidden-hub libs — non-TT-TS methods (gate-independent: RWS softmax edit is shift-invariant, evaluator change is sEH-only) differ by failure-count between runs (adenine_hybrid RWS 86.4→77.5); clean gate attribution = the same-seed counterfactual. (3) **Decision: keep ema_cats canonical** (not adopt nogate wholesale, which imports nogate's unlucky bimodal RWS draw) — spliced gate-fixed TT-TS search from nogate into ema_cats via new `examples/splice_gatefixed_ttts.py` (search tier only; ema_cats diag/post/manifest are sharded; non-TT-TS byte-identical, 0/112 cells), re-ran posthoc locally (34.6 min serial). Backups: `_pre_splice_ttts_backup/`, `analysis_gated_backup/`. ema_cats stays the manuscript's canonical 2% (no notebook repoint). (4) **Counterfactual retired**: removed from `manuscript_sar_plots.py` + `manuscript_plots_docking.py` (§4 cells, renumbered §5→§4/§6→§5, stale adenine note stripped); deleted counterfactual parquets + debug outputs; **reagent-score-landscape kept**; the amidine_035-counterfactual card superseded (appended note); 2 new [RESULT] cards (`a5457730` gate-fix, `19f566ab` RWS-bimodality), tagged Project-TACTICS-Primary-Engine + Type-Manuscript + Section-Results/Discussion. (5) Workflow note: HPC3 standard ≈6 GB/core so a large `--mem` drives the reserved core count (370G forced ~62 cores; posthoc right-sized to 16 cpu/96 G). **Resume next session:** submit `./scripts/submit_benchmark.sh ttts_sweep` (gate-fixed TT-TS at 1/3/4/5%, prepared) → splice + posthoc per fraction; repoint diagnostics notebook resolver to `nogate`; sweep manuscript prose for remaining gate-fix-stale 'RWS wins adenine'/'TT-TS bimodal' annotations; optionally delete orphaned `examples/run_{amidine035_counterfactual,premerge_adenine_recovery}.py`.
+- **2026-06-02**: Layer-2 diagnostic, diagnostics budget correction, and the **TT-TS GMIC gate discovery + fix**. (1) Built **`plot_adaptive_intensity`** (layer-2 within-component intensity, mirroring the layer-1 grammar): INTENSITY = the per-component knob as a ×-multiplier centred on 1.0 (RWS `cats_multiplier`; TT-TS adaptive `heated_scale`) → DECISION = rolling heating probability → RESULT = new reagents/batch. In `src/TACTICS/library_analysis/diagnostic_plots.py` + a dropdown over all 8 docking libraries in `tutorials/diagnostic_benchmark_plots.py`. (2) **Diagnostics budget corrected 5%→2%**: the layer-1/2 cells were loading `full_benchmark_5pct` (saturated); switched to the **2% canonical** — `full_benchmark_ema_cats` for the 4 base docking libs, and a regenerated `outputs/hybrid_diag_2pct` for the 4 hybrids (ema_cats has no hybrid diagnostics). 5% over-resources the search (adenine_hybrid isocyanide ~430 obs/reagent → GMIC saturated → near-uniform heating); **obs-per-reagent (=budget/N), not reagent count, governs** — reagent imbalance only starves a component when the budget is tight relative to N (extends lines 39/54). (3) **TT-TS GMIC min-observation gate (KEY FINDING):** `TopTwoSelection._calculate_gmic` returns 0 whenever a component's least-observed active reagent has <`min_observations`(=5) samples; `RouletteWheelSelection._calculate_gmic` has **no such gate** (git blame: both added in the founding TT-TS commit `a990948`, 2026-03-15 — an initial design inconsistency, "RWS simplified to GMIC + divergence gate only", not a later bugfix). On large components a single sub-5-obs straggler pins whole-component GMIC=0 every cycle → that component is over-weighted in the `1/(1+GMIC)` rotation. Scan: gate fires (frac of cycles with a component GMIC==0) on **25/28 libraries** at 2%; only thrombin, groebke-blackburn-bienayme, niementowski clear. Paired counterfactual (`experiments/_debug/ttts_gmic_gate_counterfactual.py` + `jobs/ttts_gmic_gate_counterfactual.sub`, 20 reps, post-merge adenine, monkeypatch — shipped code untouched): ungating lifts TT-TS top-100 **87.1→93.2** (≈ RWS 92.2), sd **18.4→10.6**, rescues **3 of 4** bimodal failures; causal chain confirmed (isocyanide GMIC 0.02→1.06, frac_gmic0 0.99→0.00; heating isocyanide 0.56→0.38, amidine 0.19→0.27). **Gate is material** → revises the amidine_035-flip attribution (the amidine_035-counterfactual card): the gate caused 3 of the 4 adenine failures, amidine_035 the 4th. **FIX applied**: gate removed from `top_two_selection.py` (now RWS-consistent); 31/31 TT-TS tests pass; no test depended on it. (4) **Stale adenine diagnostics in `ema_cats`**: adenine FRED *search* is post-merge (2026-05-23, 28 amidines incl. amidine_035) but its *diagnostics+posteriors* are pre-merge (2026-04-07, 27 amidines) — never regenerated in the 2026-05-03 integration; amide/quinazoline verified consistent (search obs == posterior n_samples, diff 0). So the prior adenine layer-1/2 figures were pre-merge. (5) **Queued (cluster busy):** full gate-fixed re-benchmark **`full_benchmark_nogate`** (28 libs × 5 methods × 20 reps × 2%) — `jobs/full_benchmark_nogate_{array,merge,posthoc}.sub` are exact copies of the `ema_cats` 2% scripts with only the output folder changed; submit via `scripts/submit_benchmark.sh full_benchmark_nogate`. Becomes the new canonical; `ema_cats` kept as the **gated reference**. Also `jobs/adenine_rerun_amidine_035.sub` (job 52890927) = gated post-merge adenine to fix ema_cats's stale adenine. **Resume next session:** when `full_benchmark_nogate` lands — re-validate recovery (expect adenine TT-TS ≈93; the 3 gate-free libs unchanged; check the other 22 for movement), point the diagnostics notebook resolver at it, regenerate adenine figures, and **then** decide whether budget-sweep/top-N reruns are needed and write the Heptabase gate-change card. Workflow note recorded: for benchmark jobs, only edit-in-place or copy-and-edit existing `.sub`/scripts — never author new ones.
+- **2026-06-01 (cont.)**: Reagent score landscape figure + additivity-axis test. (1) **φ_int DEAD END** (card `d1468d13`): interaction-fraction-at-the-hits does not discriminate greedy-easy (quinazoline φ_int@100 0.551) from allocation-hard (amide_hybrid 0.549) — greedy failure is reagent-level mean-unreachability (select-then-evaluate), NOT score non-additivity; no additivity axis can positively explain quinazoline's easiness; extends 56dc09a3. `experiments/_debug/interaction_fraction_probe.py`. (2) **Coverage clamp clarified** (appended to framework card `66c29afe`): in `max(0, 1.5 + z_mean)` the max and the 1.5 are ONE mechanism — `max(0, z_mean)` is degenerate (all hit-hubs have z_mean < 0), so z* = −1.5 is the reachability reference (≈ top 7% by mean, Φ(−1.5)=0.067), justify descriptively + sensitivity, do not outcome-fit (n=8, reopens afa11eb2). De-clamping x to raw mean-quality was built + reverted (quinazoline lands mid-plot because its easiness is the ABSENCE of a failure mode, not an extreme mean); landscape stays the three greedy-failure-mode severities (origin = easy). (3) Renamed "SAR landscape" → "reagent score landscape" in `tutorials/manuscript_sar_plots.py` + `plots/reagent_score_landscape_map.pdf` + module title; broader SAR→reagent-score rename (ROCS section, other PDF filenames, marimo vars, `.py` filename) still PENDING. Combination of failure modes → method is Q1/Q2 of the framework (any failure → TACTICS; σ-character → TT-TS vs RWS). Resume: layer-2 within-component diagnostic (unchanged next step); optional explicit combination→method inset; warmup-reachability diagnostic → min_GMIC-from-warmup (line 185, Paper 3).
+- **2026-06-01**: Docking-narrative figure + mechanism-framework consolidation. (1) Established the **three-mechanism method-winner framework** — coverage `max(0,1.5+z_mean)` / allocation `z_std×(1−top1)` / imbalance `max/min count` for TACTICS-vs-Legacy (any one sufficient); hidden-hub σ-character for TT-TS-vs-RWS — explicitly DESCRIPTIVE/post-hoc (constructed thresholds fit to 8 libs, not a classifier; run-both-take-max stands). (2) Reworked `tutorials/manuscript_sar_plots.py`: docking SAR landscape → 3-mechanism axes (x=coverage, y=allocation, edge-width=imbalance, fill=outcome); counterfactual bars recolored z(std)→−z(mean) mean-quality; hub-mechanism plots REMOVED; containment curve added then removed (→ two new DEAD ENDS: two-fraction-on-quinazoline + containment-curve). (3) Built **GMIC signal→decision→result diagnostic** (`plot_gmic_directed_exploration` in `diagnostic_plots.py` + thrombin cell loading `full_benchmark_5pct`); thrombin shown mean-accessible-but-estimation-limited. (4) 5 Heptabase cards (3 manuscript Section-Discussion: `66c29afe`/`ccf22197`/`548d7583`; 2 dead-end: `afa11eb2`/`56dc09a3`); card `5a862478` amended with the quinazoline caveat. Resume next session: build **layer-2** within-component diagnostic (`plot_adaptive_intensity`: RWS ema_relative_gmic→cats_multiplier, TT-TS disagreement_ema→heated_scale, RESULT = within-component sampling breadth), prototyping on a BALANCED library first.
+- **2026-05-31**: Docking-benchmark expansion (opened 2026-05-30) evaluated and **dropped from Paper 1**; refocused on docking narrative + figures. (1) **sEH dead-ended** (overnight pilot, Heptabase card 24bd9c07): DEL read-count is mean-separable (greedy 61% > RWS 59% bimodal > TT-TS 26% at top-100/300K), exposing that TACTICS has no collapse-to-greedy off-switch; retained as the TACTICS-v2/Paper-1b diagnostic dataset — now in Ruled Out / Dead Ends. (2) **KinDEL (MAPK14+DDR1) declined** for Paper 1: no local files (fetch-from-S3, CC BY-NC-SA), redundant kinase family (JNK3 already covered), single-pipeline, mean-separability risk on its enrichment/Kd objective. (3) Docking story confirmed robust without expansion — 3/4 chemistries significant across 2 pipelines + 2 protein families with a clean negative control (quinazoline ≈0) and the amide FRED→HYBRID greedy-collapse (95→47, TT-TS +8.8 pp) rebutting the "adenine carried it" critique. (4) **Paper-1 scope** discussed and DECIDED: the current outline is ~2–3 papers' worth and a split (app note + ROCS paper + docking paper) would have been scientifically valid — each regime is independently novel and Zhao 2025 covered only ROCS — but **decision is to keep everything in one comprehensive paper**; the "mega-paper" characterization is accepted as deliberate. Resume next session: build the docking-section figure set + narrative (hidden-hub SAR landscape, cross-pipeline contrast, when-TS-beats-greedy-on-docking) from the docking manuscript notebook + Heptabase docking card set.
+- **2026-05-30**: Docking-figure polish + **docking-benchmark expansion scoped** (new active build work; full plan in `docs/handoffs/2026-05-30_docking-benchmark-expansion.md`). (1) **Figure work**: `tutorials/manuscript_plots_docking.py` §6 budget-sensitivity panel bottom row converted from single mean-bar to a **per-library stacked decomposition** (per-chemistry segments sum to the bin mean; answers "why is 2-comp ≥ 3-comp?" → adenine carries 3-comp, quinazolines ≈0); bar colors moved to unused Tableau-10 members (adenine green #59A14F, quinazoline pink #FF9DA7, amide brown #9C755F, thrombin gold #EDC949) to avoid colliding with the method-line palette; legend title dropped (the white-gap cause); standalone per-library-exemplars figure removed. Also restructured earlier this thread: heatmap §1 + per-library dropdown §2 (file renamed `manuscript_plots.py`→`manuscript_plots_docking.py`). (2) **Confirmed LookupEvaluator keys on reagent NAME, not structure** (`core/sampler.py:322`, `core/evaluators.py:159`): during search TACTICS skips molecule generation entirely and looks up `Product_Code = "_".join(reagent_names)` → SMARTS is irrelevant to scoring (need only parse). This unblocks "dock any enumerable library, score by product code." (3) **3 sub-agents run**: sEH-paper recon (✓ Zhang 2023 JCIM, our lab's DEL), SMARTS-discovery (rate-limited but report recovered to `research/blueprints/seh_caix_smarts.md` — sEH amide scheme reproduces 97.4% of products; CAIX too small at 216K → dropped), literature scan (KinDEL + sulfonamide/CA-II; sulfonamide DROPPED for catalytic-Zn docking noise). (4) **Decisions**: add **sEH @ ~60M** (centerpiece, our data, with experimental-binder overlay), **KinDEL MAPK14+DDR1** (license verified: Insitro custom CC-BY-NC-SA, benchmarking explicitly permitted, FETCH-not-vendor into separate repo), and (low priority) **ROCS-libs-on-JNK3-2ZDT**. sEH 60M recipe = all binder bb1×bb2 + top-~255 bb3 → 606×388×255 ≈ 60M, retains 95% of amide-compatible binders (109,663 of 116,666 binders survive the amide filter; full binder coverage would need ~261M). Receptor pick **4OD0** (TBD-confirm). **Open risk (do first): name reconciliation** — binder BB columns don't match master reagent lists 1:1 (665 vs 617 bb1; 388 vs 338 bb2 canonical), so library reagent sets must be built from the binder requirement ∪ fill, not by subsetting master lists. (5) **Standardized format locked** for all new datasets (reagent `.smi` `SMILES name` no-header; score parquet `Product_Code,Scores`; reactions.csv row; triple-config per Local Norms #11/#12) — see handoff doc §2. Resume: handoff doc §6 ordered steps, starting with sEH name reconciliation. No enumeration/docking launched.
+
+- **2026-05-05 (continued)**: Manuscript-card structuring continued from earlier in the day. (1) **Standardized TL;DR coverage**: confirmed all 31 then-tagged manuscript cards lead with an explicit `## TL;DR` H2 — 13 cards renamed (Finding/Summary/Overview → TL;DR), 16 cards got freshly written 4–6 sentence TL;DR paragraphs prepended after H1 title. Whiteboard previews now show the gist of each card without scrolling. (2) **4 Methods 2.1 bridge cards added**: explicit gap → method → mechanism transitions for the manuscript Methods overview. Cards: `[METHOD] TACTICS RWS — From Zhao's Signal-Blind Rotation to GMIC-Driven Criticality` (`7e34aa1e-1aa4-484d-abfb-ebe2a01c3c52`); `[METHOD] TACTICS TT-TS — From Klarich's TS-Regret-vs-Tail-Finding Mismatch to σ-Based Best-Arm Identification` (`0bc83dbd-9977-41ed-8c26-81969cb20115`); `[METHOD] Lean Enhanced Warmup — How TACTICS Exploits the Information Asymmetry` (`27cbde69-d7d0-4dc9-8011-e483876d6dce`); `[METHOD] TACTICS Library Diagnostics — From Riley's Qualitative TS-on-Docking Argument to Quantitative SAR-Landscape Characterization` (`a9f61a61-61fe-4cdd-b782-59defd48b98b`). Each bridges a [RESEARCH GAP] / [OBJECTIVE] card to one or more mechanism walkthrough cards. (3) **Lean Enhanced Warmup card rewrite (corrected attribution)**: original draft incorrectly attributed Enhanced Warmup (stochastic parallel pairing) to TACTICS as a new contribution. Corrected: Enhanced Warmup is Zhao 2025's published algorithm, used by TACTICS unchanged (`src/TACTICS/thompson_sampling/warmup/enhanced.py`). The TACTICS contribution is recognizing that a *lean* Enhanced warmup (default `num_warmup_trials = 5`) creates exploitable information asymmetry on imbalanced libraries (147 obs/acid vs 5 obs/amine on thrombin) and exploiting it through GMIC-aware selection. The same asymmetry is +5 pt accelerant for TACTICS but −9 pt poison for Legacy RWS's symmetric thermal cycling. (4) **3-way split of Benchmark Methods card** to support manuscript Methods 3.1 / 3.2 / 3.3 sub-subsection writing. Original combined card `739bb5d0-063c-419d-9403-acc82c1f301a` (80 blocks: methods + libraries + budget + scoring + statistics + key differences + implementation) replaced in-place with Card 1 (Methods + Libraries + Parameters scope only). Two new cards: Card 2 = `[METHOD] Evaluation Budget Equality and Warmup × Selection Pairings` (`84ae91a7-14c0-412b-b458-ad361d59976c`); Card 3 = `[METHOD] Statistical Analysis Framework` (`cdfa2e00-4a3f-408e-a1f2-e6d96b19c983`). All three tagged with `TACTICS-Manuscript`; tag now contains 37 cards total. Resume next session: begin drafting manuscript Methods + Discussion paragraphs from the tagged Heptabase set, especially using the 4 new bridge cards as the Methods 2.1 narrative spine.
+- **2026-05-04 to 2026-05-05**: HYBRID adenine narrow-top-N investigation + Heptabase manuscript-card cleanup. (1) **Narrow-top-N case study**: empirical confirmation that on adenine_hybrid top-25 (23/25 hits in ald137 + 2 tail-aldehyde rescues in ald111/ald121), Legacy RWS achieves 24.8 ± 0.5 vs TT-TS 23.9 ± 1.0 vs TACTICS RWS 22.4 ± 4.9 — Legacy RWS wins narrow-top-N because round-robin's blind component-symmetric exploration catches tail-aldehyde rescues that TACTICS' GMIC-driven reallocation marks as not worth exploring. The same reallocation produces the systematic wider-top-N gains. (2) **ROCS top-N decay quantified across 20 libraries**: Legacy RWS −6.49 pp from top-25 to top-250 vs TT-TS −3.63 pp (1.7× slower decay); 2-comp ROCS Legacy RWS −9.07 pp vs TT-TS −5.25 pp; TT-TS advantage over Legacy RWS grows monotonically from +2.2 (top-25) to +5.1 (top-250). (3) **Zhao 2025 Figure 5 read**: Both TS and RWS recovery decrease as top-100 diversity grows; RWS decline is steadier than TS. TACTICS extends one mechanistic step further: data-driven reallocation gentler than Zhao's symmetric thermal cycling. (4) **Two new Discussion cards in Heptabase**: `cfc20ceb-f4dd-4823-a916-dd9348b56bdb` (TACTICS as Efficient Learning) and `bf67bcd4-6136-45dd-a75f-2ca409da7e0d` (ROCS Top-N Decay). (5) **Manuscript notebook**: replaced single 5-method overlay Zhao-Fig-5 plot with cleaner 4 pairwise plots (TACTICS RWS / TT-TS × Legacy RWS / Legacy Greedy), each 2-panel (10 libraries 2-comp, 10 libraries 3-comp), with linear-fit slopes in legend. (6) **Total-eval audit**: confirmed budgets match across 23/25 libraries (within 3%); adenine_hybrid 8% gap and adenine FRED 68% gap are due to Legacy RWS's intentional `stop=6000` early-termination criterion (acceptable per user — part of Legacy method spec). The TACTICS modern methods compute `n_cycles = (total_budget − warmup) // batch_size` matching the Legacy TS budget formula. (7) **HYBRID top-100 SDF extracted**: `data/docking/adenine/top100_hybrid_canonical_docked.sdf` (621 KB, 100 poses rank-ordered by Chemgauss4 score) via new `extract_top100_canonical_hybrid.py`. (8) **Heptabase manuscript-card triage** (2026-05-05): created `TACTICS-Manuscript` tag, applied to 31 latest result/method/discussion/research-gap cards; trashed 20 outdated cards (older benchmark snapshots superseded by 25-library `7c446b85`, older mechanism walkthroughs superseded by canonical [METHOD] cards, duplicates, abandoned design proposals). The 25-library benchmark `7c446b85` is canonical headline; older 21-library, 20-library, 24-library, 8-library, top-N-sensitivity-20-library, and expanded-docking-24-library snapshots are now trashed. Resume next session: draft manuscript Methods + Discussion paragraphs from the tagged Heptabase set.
+- **2026-05-03**: Path A integration complete — post-merge adenine + adenine_hybrid integrated into `outputs/full_benchmark_ema_cats/`. (1) **Integration mechanics**: archived pre-merge adenine search shards to `_adenine_pre_merge/`; copied post-merge FRED adenine + new adenine_hybrid shards into the canonical search dir; updated all 16 manifest shards (dropped 100 pre-merge adenine entries) + added new manifest_shard_025 (200 new entries: 100 post-merge adenine + 100 adenine_hybrid). Total: 218,500 trials across 25 libraries. Added `"adenine_hybrid"` to `MINIMIZE_LIBRARIES` in `examples/run_posthoc_analysis.py`. (2) **Post-hoc re-run on integrated benchmark**: 20.2 min on 8 workers (I/O-bound on SSHFS mount; would be 2–5 min on HPC3 local SSD). Output: 2,185,000 recovery rows; 248/250 ANOVA tests significant; 1,743/2,480 Tukey HSD pairwise sig (326 large d); all 10 Friedman across-library tests p < 10⁻¹³. (3) **Headline empirical claim now validated at canonical-benchmark scale**: best-TACTICS beats best-Legacy on 24/25 libraries (only orru ns at −0.1). Adenine FRED post-merge: TACTICS RWS = 94.5 (best) vs Legacy RWS = 75.8 → +18.8 pp. Adenine HYBRID: TT-TS = 94.5 (best) vs Legacy RWS = 84.0 → +10.5 pp. Aggregate top-100 ranking: TACTICS TT-TS first (86.4 ± 15.7), TACTICS RWS second (85.9 ± 15.5), Legacy RWS third (82.9), Bal-Greedy fourth (82.0), Legacy TS last (81.1). (4) **Manuscript notebook updated** (`tutorials/manuscript_plots.py`): added adenine_hybrid to 4× `_DOCK_LIBS`, 2× `_DOCKING_LIBS`, 4× `_DOCK_NCOMP` definitions; added to SAR plot dropdown `_all_libs`; added to SAR plot `_CONFIGS` dict with parser shared with adenine (chemistry identical); added to multi-top-N `_libraries` list; added a parallel cell after the existing FRED-adenine "Top-N sensitivity (TT-TS vs Legacy) + Δ recovery bars" plot for adenine_hybrid (mirror of the existing cell, library swapped, generates `adenine_hybrid_topn_ttts_vs_legacy.pdf`). The dropdown menus that auto-populate from the loaded recovery DataFrame need the user to re-run the data-load cell at line 82 to pick up the 25-library aggregate (marimo doesn't auto-detect file-on-disk changes). (5) **Top-100 SMILES exports**: `outputs/adenine_fred_top100.csv` and `outputs/adenine_hybrid_top100.csv` written for chemistry inspection. Zero product-code overlap. Chemical contrast: FRED top-100 dominated by Boc-piperazine + cinnoline-amine + bulky scaffolds (amidine_003 + isocyanide_db_472); HYBRID top-100 dominated by **boron-containing chemotypes** — aldehyde_137 (3-formylphenyl pinacol boronate, 70/100 hits) + isocyanide_db_343 (3-isocyanophenylboronic acid, 11 hits) + small fluorinated 2-aminopyridines (amidine_009, 23 hits). Both top hubs in HYBRID contain boron; HYBRID-Lenselink-A2A binding pocket exploits boronate moieties that FRED-4EIY does not. (6) **Performance note**: post-hoc on SSHFS-mounted full benchmark is I/O-bound (30 GB total parquet read); CPU is not the bottleneck and adding workers beyond 8 won't help. Future runs should either run on HPC3 (local Lustre) or copy the search dir to local SSD first. Resume next session: draft manuscript Methods + Discussion paragraphs from the (now-six) framing cards; consider building the focused cross-docking-pipeline figure (Path B) for the Discussion centerpiece — special plots beyond the per-library bars (per-rep distributions, hub decomposition, axis-overlay annotations).
+
+- **2026-05-06**: Manuscript-plot polish + statistical framework extension (planned Welch's t-test) + new Heptabase Methods card. (1) **Zhao Fig 5 plots overhauled** in `tutorials/manuscript_plots.py`: added per-library vertical slopegraph arrows connecting baseline → challenger to make within-library Δ visible; added a 4-cell `zhao_delta_plot` companion (Δ recovery vs diversity per pairwise comparison, with linear fit + slope + R²); both helpers share `zhao_data`. Layout polish: figsize 15×6 → 18×8, legend moved below the panel via `bbox_to_anchor=(0.5, -0.14)` (Zhao 2025 attribution stripped from axis labels — kept only in the legacy-RWS legend label where it belongs). (2) **Top-N sensitivity plots (adenine FRED, adenine HYBRID, quinazoline) updated**: each cell now plots 4 lines (TT-TS, TACTICS RWS, Legacy Greedy, Legacy RWS) on the left panel using the canonical method palette; the right panel switched from TT-TS-only Δ to "best TACTICS" Δ — at each top-N, max(TT-TS, TACTICS RWS) is selected and 2 bars per top-N are drawn (orange = vs Greedy, steel blue = vs Legacy RWS) with the corresponding Tukey HSD significance star pulled for whichever method was best at that threshold. (3) **Statistical framework extended with planned Welch's t-test (Bonferroni-4)**: added `run_planned_welch_tests` + `_cohens_d_independent` to `examples/run_posthoc_analysis.py`. Tested directly on the four pre-specified TACTICS-vs-Legacy pairs without ANOVA gating (planned comparisons by construction). Bonferroni: bonf_p = welch_p × 4, α=0.05/4=0.0125. New output `outputs/full_benchmark_ema_cats/analysis/welch_planned_results.parquet` (1000 rows: 25 libs × 10 top-N × 4 pairs; 86.2% Bonferroni-significant). Computed via shortcut script reading existing `recovery_summary.parquet` (~1 s runtime); full pipeline can re-run end-to-end later for consistency. (4) **Adenine_hybrid Tukey-vs-Welch divergence empirically confirmed**: Tukey HSD flags TT-TS vs Legacy RWS as significant at only 1/10 top-N (top-100, p_adj=0.010 *) despite mean diffs of 6–10 pp at top-75 onwards; Bonferroni-Welch flags 9/10 thresholds as significant (Cohen's d_s 0.96–1.96, large). Root cause: Tukey's pooled MSE is inflated by TACTICS RWS bimodality (std ≈ 14) and TACTICS Balanced-Greedy variance (std ≈ 14), penalizing the TT-TS-vs-Legacy-RWS pair for variance that's in neither method (TT-TS std ≈ 5, Legacy RWS std ≈ 6). The narrow-top-N tail-aldehyde rescue (Legacy RWS wins at top-25 by 0.9 counts) is also now Bonferroni-Welch-significant (** at bonf_p=0.006), strengthening Discussion card `cfc20ceb`. (5) **Adenine_hybrid bar plot annotated dually**: Tukey star ("T:" prefix, black, closer to bar) + Bonferroni-Welch star ("W:" prefix, color `#4477AA`, further out) stacked above each Δ bar, with footnote caption explaining the convention. Adenine FRED + quinazoline cells stay Tukey-only by user scope (the two tests agree there). (6) **New Heptabase Methods card** (`005cb64e-cbfc-4aab-bcc9-35cc732330d4`): `[METHOD] Planned-Comparison Welch's t-test for TACTICS-vs-Legacy Pairs (Bonferroni-4)` — TL;DR + motivation (why Tukey HSD is conservative on heterogeneous-variance libraries) + method specification + empirical Tukey-vs-Welch divergence on adenine_hybrid + reporting convention. Tagged `TACTICS-Manuscript`. The existing `[METHOD] Statistical Analysis Framework` card (`cdfa2e00`) was appended with a cross-link addendum referencing the new card. (7) **Inconsistency flagged for follow-up**: lines 922 (Tukey HSD CI plot — docking) and 3124 (across-library Tukey HSD pairwise CI) of `tutorials/manuscript_plots.py` currently include `adenine_hybrid` in aggregate docking pools, which double-weights adenine chemistry in the pooled Tukey test (same statistical concern as the overall recovery vs top-N plot at line 2009 which already excludes it). User confirmed FRED-only aggregation is the right frame; not yet applied to lines 922/3124. Resume next session: drop adenine_hybrid from those two aggregate Tukey HSD cells; integrate planned-Welch results into manuscript Methods/Discussion text; consider whether to apply same dual annotation to other libraries (currently adenine_hybrid only).
+
+- **2026-05-14 to 2026-05-18**: Statistical framework finalization + manuscript Methods/Discussion drafting + four-paper pipeline formalization in Heptabase. (1) **Test-selection rule iterated to final form**: initial dual Tukey/Welch annotation on adenine_hybrid → variance-ratio threshold (σ_max/σ_min > 5×) → final **scoring-type rule**: docking libraries (1 query × 20 reps, lacks repeated-measures structure → reduces to one-way ANOVA where homoscedasticity is needed but empirically violated) get planned Welch + Bonferroni-4 on the 4 pre-specified TACTICS-vs-Legacy pairs (color #4477AA); ROCS libraries (109 queries × 20 reps, satisfies repeated-measures, approximately homoscedastic) get repeated-measures ANOVA + Tukey HSD per Ash et al. 2025 (color black). Codified in `tutorials/manuscript_plots.py` as `DOCKING_LIBS = {thrombin, adenine, adenine_hybrid, amide, quinazoline}` shared utility constant. Quinazoline cell converted from Tukey/black to Welch/blue as part of this change. (2) **Levene's test dropped from inferential pipeline**: error bars + per-method σ supplementary table is sufficient; pre-test bias (Zimmerman 2004, Rasch 2011) makes Levene-conditioned switching statistically incorrect. (3) **Cohen's d_s dropped from manuscript text**: all measurements share % recovery scale, raw mean diff + 95% CI is the appropriate effect-size statement; `_cohens_d_independent` retained in `examples/run_posthoc_analysis.py` and Welch parquets for reviewer audit only. (4) **CLT-based defense** for parametric tests at n=20 with bimodal-failure distributions documented in Methods. (5) **Adenine-without-amidine_035 counterfactual rewritten end-to-end**: initial post-hoc rescore (`run_amidine035_counterfactual.py`) gave misleading TACTICS-RWS-favoring result because post-merge search trajectories carry 28-amidine warmup-pool effects. Replaced with `examples/run_premerge_adenine_recovery.py` using the genuine pre-merge benchmark (27-amidine search shards from `outputs/.../search/_adenine_pre_merge/` against `data/scores/adenine.parquet.bak`). Pre-merge top-100 (mean ± std, 20 reps): TT-TS = 96.0 ± 4.0 (winner, no bimodal failures), TACTICS RWS = 78.5 ± 24.0 (bimodal: ~7/20 reps catastrophically fail), Legacy RWS = 45.1 ± 27.1, greedy ~31. New outputs: `outputs/.../premerge_adenine_recovery.parquet`, `_stats.parquet`. Old counterfactual script marked deprecated. (6) **Plot cosmetic iterations**: per-library bar plot dispatches by DOCKING_LIBS for star coloring; adenine FRED / HYBRID / quinazoline / pre-merge top-N cells use vertical (rotation=90) significance stars; pre-merge cell uses line-plot layout matching FRED/HYBRID/quinazoline pattern; star captions moved from inline `_ax.text` to underneath `mo.md` markdown cells; panel sizes returned to original `figsize=(16, 6)` after experimental enlargement was deemed too large. Two aggregate Tukey HSD docking-pool cells (formerly STATUS lines 922/3124 ref) dropped adenine_hybrid → now `{thrombin, adenine, amide, quinazoline}`. (7) **Supplementary 4-panel agreement matrix** added: shows both Tukey and Welch significance for every (library × top_n × planned pair) cell with 4-way classification (both-sig / both-ns / Welch-only / Tukey-only); output `supp_tukey_vs_welch_agreement.pdf`. (8) **Manuscript Methods §Statistical analysis written** in `~/Desktop/TACTICS-manuscript/manuscript.qmd` (repo moved off SSHFS mount mid-session due to repeated permission drops; canonical location is now Desktop). Discussion §Sensitivity analysis (pre-merge benchmark, TT-TS dominance on hidden-hub-axis-dominated landscape) + §Single-pose docking and ABFE/FEP integration future-work paragraph also written. (9) **Regime-taxonomy language eliminated** from manuscript and Heptabase per the 2026-05-01 three-axis framework decision; all "Regime Ia/Ib" references replaced with axis-language (hidden-hub size, μ-reachable share, hidden-hub σ-robustness). (10) **Heptabase corrections appended to 5 cards**: `005cb64e` (Welch), `cdfa2e00` (Statistical Framework), `d5e630c8` (new decision-rule card created this session), `a19b3878` (three-axis framework), `cfc20ceb` (efficient learning). An overstated TACTICS-RWS-favouring claim from earlier in the session was explicitly retracted on `a19b3878` + `cfc20ceb`. (11) **Cofolding-validator literature scan** (subagent): confirmed Paper 3 (TACTICS as cofolding-ensemble structure-quality validator for FEP triage) is novel. Closest prior art: Mizukoshi 2024 Sci Rep (AF2-MSM kinase ensemble ranked by VS enrichment factor — same conceptual inversion but not ML cofolding, not combinatorial library, not recovery-shape diagnostic, not AL) and Lupala 2023 (AF + FEP recapitulation — uses FEP itself as validator, not cheap TS-VS proxy). The four-piece combination (ML cofolding ensemble × combinatorial library × recovery-curve shape diagnostics × TS primitive) is the novel contribution. (12) **Zaballa & Hui 2024 MLSB NeurIPS** (arXiv 2410.10108) read: systems-biology-guided molecule generation via non-differentiable diffusion on the BMP pathway. Used as precedent for Paper 4 (evaluation-side dual: pathway-aware cell-line selection via TACTICS multi-target screen). (13) **Four-paper pipeline formalized in Heptabase** (2026-05-18): created `TACTICS-Papers` umbrella tag (`be5a2724`) + per-paper sub-tags `TACTICS-Paper-2-Generative` (`26559a33`), `TACTICS-Paper-3-Cofolding` (`2c994162`), `TACTICS-Paper-4-Pathway` (`9b8548d6`); Paper 1 retains existing `TACTICS-Manuscript` tag for backward compat. Spawned 3 parallel subagents that produced 6 OBJECTIVE + RESEARCH-GAP cards (2 per paper). New cards: Paper 2 OBJECTIVE `ed43a2f4` + GAP `2ba19f5c`; Paper 3 OBJECTIVE `af23611c` + GAP `8f92a732`; Paper 4 OBJECTIVE `1919aea8` + GAP `519eb72f`. All saved with LaTeX math, tagged with per-paper + umbrella tags. (14) **Memory feedback saved**: `feedback_equations_unicode.md` (Unicode math in terminal output, LaTeX in Heptabase + Quarto, ASCII in code) + `feedback_cohens_d_when_same_scale.md` (Cohen's d for cross-study/cross-metric work only; raw mean diff + 95% CI for same-scale benchmarks). Resume next session: draft manuscript.qmd Results + remaining Discussion paragraphs from the now-44-card Heptabase set (`TACTICS-Manuscript` 38 cards + `TACTICS-Papers` 6 new cards spanning Papers 2-4); decide whether to bring TACTICS-manuscript repo back to mount or keep on Desktop.
+
+- **2026-05-20**: HYBRID benchmark for 3 new docking libraries (thrombin_hybrid, amide_hybrid, quinazoline_hybrid) — built, integrated, and prospectively validated the three-axis framework 4-for-4. (1) **Brute-force HYBRID docking complete on all 3 libraries**: thrombin_hybrid (100/100 batches against 2ZFF, 492,982 unique products), amide_hybrid (1000/1000 batches against JNK3 2ZDT, 9,609,696 unique), quinazoline_hybrid (1997/1997 batches against JNK3 2ZDT, 19,566,280 unique). Slurm submission via new `data/docking/{lib}/dock_{lib}_hybrid.sub` + `submit_{lib}_hybrid.sh` + `resubmit_{lib}_hybrid.sh` (self-healing loop drained quinazoline's 646-batch node-failure tail). Merge scripts produced `data/scores/{lib}_hybrid.parquet`. Adenine merge script (`merge_full_hybrid.py`) had a 2-column-vs-4-column score-file parsing bug carried over from the 2026-05-01 ad-hoc fix; fixed in-place along with the three new merge scripts. (2) **TS benchmark on the 3 new HYBRID libraries** (5 methods × 20 reps × 3 libs = 300 trials, ~1.5h wall-clock): `outputs/full_benchmark_hybrid_3lib/`. Per-library top-100 (mean ± std): **thrombin_hybrid** TT-TS = 82.1 ± 3.3 (winner) / TACTICS RWS = 81.7 ± 2.2 / Legacy RWS = 70.7 ± 3.4 / Greedy methods ~61–67 / Δ(best-TACTICS − best-Legacy) = +11.45 pp ****; **amide_hybrid** TT-TS = 83.3 ± 2.5 (winner) / TACTICS RWS = 80.1 ± 2.8 / Legacy RWS = 74.5 ± 3.1 / Greedy methods ~47 (collapsed from FRED ~95) / Δ = +8.80 pp ****; **quinazoline_hybrid** all 5 methods at 97.0–97.1 / Δ = −0.10 pp ns. (3) **Triple-config drift bug — MINIMIZE_LIBRARIES** caught this session: the set lives in 3 files (`_benchmark_config.py`, `run_posthoc_analysis.py`, `stage_benchmark_data.sh`) and only the first two were updated when adding the new HYBRID libraries; posthoc treated them as MAXIMIZE → near-zero recovery. Fixed and codified as CLAUDE.md Local Norm #12. (4) **Independent recovery verification**: hand-recomputed `|observed_names ∩ GT_top100|` per (library, method, replicate) triple from raw search shards bypassing run_posthoc_analysis.py — 0/300 mismatches against the saved `recovery_summary.parquet`, confirming the fixed script is correct. (5) **Three-axis framework prospectively confirmed 4-for-4**: predictions were written into Heptabase card `a19b3878` 2026-05-19 addenda *before* this benchmark ran. Adenine HYBRID (n=1 confirmed earlier per STATUS line 58): TT-TS dominant, no bimodal failures ✓. Thrombin HYBRID: predicted TT-TS or tied with TACTICS RWS → TT-TS = 82.1 narrowly over TACTICS RWS = 81.7, both crush Legacy ✓. Amide HYBRID: predicted TT-TS gains a σ-niche → TT-TS = 83.3 wins by +3.2 over TACTICS RWS ✓. Quinazoline HYBRID: predicted null shift (axis profile structurally stable) → ALL 5 methods at 97.0–97.1, exactly flat — the documented null ✓. TT-TS marginal-advantage magnitude tracks axis-3 shift magnitude (amide z(σ) jumped +4.31 → TT-TS leads RWS by +3.2; thrombin z(σ) jumped +0.71 → TT-TS leads RWS by +0.4). Framework moved from post-hoc explanatory (n=1 adenine) to prospectively predictive (n=4 across 3 protein families). (6) **Greedy collapse on amide HYBRID**: TACTICS Balanced-Greedy = 47.25, Legacy Greedy = 46.90 — both essentially halved from FRED amide's ~95–97. This is the strongest direct empirical evidence in the manuscript that hidden-hub SAR makes mean-based selection insufficient. Cohen's d = 14–17 for TACTICS RWS / TT-TS vs Legacy Greedy on amide_hybrid. (7) **Integration into canonical benchmark**: copied search shards from `outputs/full_benchmark_hybrid_3lib/search/{lib}/` into `outputs/full_benchmark_ema_cats/search/`; added `manifest_shard_026_hybrid_3lib.parquet` (300 new entries). Total: 28 libraries, 218,800 trials. Posthoc re-run on the integrated 28-library benchmark in progress. (8) **Manuscript notebook updated** (`tutorials/manuscript_plots.py`): added the 3 new HYBRID libraries to `DOCKING_LIBS` (master), `_DOCKING_LIBS` (2 exclusion lists at lines ~1485, 3370), `_DOCK_LIBS` exclusion (line 1942) — kept FRED-only aggregates at lines 1352, 2443, 3562 unchanged (their FRED-only scope is by design per the user's "don't average across pipelines" rule); added to `_DOCK_NCOMP` (3 occurrences); added entries to SAR `_CONFIGS` dict (parsers shared with FRED counterparts since chemistry is identical); added to `_all_libs` SAR-dropdown list; added to multi-top-N `_libraries` list. Three new `@app.cell` Top-N + Δ-recovery cells inserted after the adenine_hybrid cell — verbatim copies of the adenine_hybrid pattern with library swapped, generating `thrombin_hybrid_topn_tactics_vs_legacy.pdf`, `amide_hybrid_topn_tactics_vs_legacy.pdf`, `quinazoline_hybrid_topn_tactics_vs_legacy.pdf`. Syntax-checked clean. (9) **Heptabase card maintenance**: trashed 3 regime-language DISCUSSION cards (`071cb95b`, `2833ab9b`, `2479ac91`) following the 2026-05-15 retraction of regime terminology; trashed `006cedd4` (adenine docking-failure-uniform card with stale 16% pre-merge framing) and replaced with a new Adenine Docking Coverage FRED-vs-HYBRID card (since deleted) covering post-merge FRED 2.4% + HYBRID 0.1% failure rates and the FRED-only amidine_003 outlier. Appended 2026-05-19 addenda to `a19b3878` (three-axis framework card) with thrombin, amide, and quinazoline predictions — these were the prospective predictions confirmed by today's benchmark. (10) **Memory cleanup**: deleted the previously-saved `feedback_stage_benchmark_data_dual_config.md` memory at user request — incident records belong in STATUS, not auto-memory. Resume next session: append the empirical 4-for-4 confirmation as a final addendum to `a19b3878`; integrate the per-library Welch+Bonferroni-4 results into manuscript Discussion §Cross-pipeline-framework-validation; verify the 28-library posthoc recovery numbers once that re-run completes (still in progress at time of writing).
+
+- **2026-05-18 (session 2)**: Spun out `investigate/cofolding-trust-meter` worktree (at `~/hpc3/TACTICS-cofolding-trust`) to investigate TACTICS recovery-curve diagnostics as a PoseBusters-analogue trust meter for cofolded receptor structures. Extended framing session converged on the trust-meter framing after discarding ensemble-TACTICS / FEP-triage / cryptic-pocket / discriminator-library framings (each had circular ground-truth or SBDD-incoherence problems). Parallel literature reviews (4 sub-agents) confirmed the contribution is novel — no prior work uses recovery-shape diagnostics for structural utility assessment, PoseBusters is the right precedent at the pose level, KinConfBench (Apr 2026) supplies kinase-specific failure cases, Mac1 `mac-x4091` is the showcase motivating example, A2A discarded as a pilot target (literature shows it is one of the best-predicted GPCRs). Heptabase OBJECTIVE (`af23611c`) and RESEARCH GAP (`8f92a732`) cards updated with revised framing; pilot specification card (`39129b4e`) created; superseded proposal card (`a7aacd84`) trashed. Pilot target: JNK3-quinazoline with 5 structures (crystal + 2 cofolds + 2 documented-bad cofolds). Compute budget ~1-2 days CPU for pilot, ~50-80K core-hours for full study if pilot passes. Worktree branch `investigate/cofolding-trust-meter` ready for first experiment.
+
+- **2026-05-21**: Manuscript figure-rendering convention decision (significance-star display rule) + Heptabase interpretation-guidance card. Continuation of 2026-05-20 manuscript-plots work. (1) **Quinazoline_hybrid bar-plot concern raised**: noticed *** significance stars on bars with absolute Δ-recovery of ~0.5–0.75 counts at top-125 and top-150, looked Type-I-error-inflated and potentially reviewer-pushback-prone. Diagnostic on the underlying recovery distributions confirmed the test is NOT Type-I-inflated — within-method variance is genuinely near-zero on this near-ceiling library (TACTICS methods deterministically recover the same count every replicate, Legacy Standard-Greedy varies by ~0.7 count across 20 reps), so Welch correctly says "these distributions are different." Actual issue is **statistical-significance-vs-practical-significance decoupling**, not Type I error. (2) **Option A (programmatic threshold) implemented then reverted by decision**: initial fix added `PRACTICAL_DELTA_FLOOR = 1.0` (absolute recovery count) to the rule cell of `tutorials/manuscript_plots.py`, gating all 8 per-library bar-plot star conditionals on `abs(_d) >= PRACTICAL_DELTA_FLOOR AND p < α`. Heptabase card `8a451c17` was created to document the threshold rule. Then reconsidered: at top-250 a 1 percentage point Δ is 2.5 compounds (meaningful, not trivial), and a fixed pp-floor over-suppresses at wider top-N where 1 pp represents multiple compounds. **Decision**: revert the programmatic suppression entirely; significance markers should reflect the underlying test result unconditionally; manuscript text (Methods + Discussion) handles the practical-significance interpretation. (3) **Heptabase card refactor**: trashed `8a451c17` (threshold rule, now misleading); created `50e1fe61` (`[METHOD] Interpreting Significance Stars on Per-Library Δ-Recovery Figures`, tagged `TACTICS-Manuscript`). The new card: documents the convention (stars = statistical detection, bar height = magnitude, the two are complementary), records why the programmatic threshold was rejected (pp-scaling with top-N, transparency of unconditional stars, bar height already carries magnitude), provides ready-to-paste Methods §Statistical analysis paragraph + Discussion §Quinazoline near-ceiling paragraph, and a four-step reading guide for figure interpretation (bar height → star → regime context → supplementary parquets). (4) **Cross-link addenda updated** on `cdfa2e00` (Statistical Analysis Framework) and `005cb64e` (Welch+Bonferroni-4) to replace the prior threshold-rule addenda with pointers to `50e1fe61`; addenda explicitly note that no programmatic threshold is applied at figure-render time. (5) **28-library posthoc complete** (was in progress at end of 2026-05-20 session): `outputs/full_benchmark_ema_cats/analysis/` (recovery_summary, anova, tukey, welch_planned, friedman, nemenyi) all written. **Three-axis cross-pipeline addenda on `a19b3878` are also appended** (the 4-for-4 prospective-prediction confirmation noted as 2026-05-20 resume task). (6) **Code revert in `tutorials/manuscript_plots.py` is STILL PENDING**: blocked on SSHFS mount in EPERM-on-content-read state (different from typical disconnect — `ls` succeeds but `cat` fails with EPERM); workarounds attempted (force unmount, kill sshfs/macfuse, kextunload/kextload, remount) did not resolve, restart appears required. The mechanical revert script (deterministic regex undo of the morning's edits) is recorded in the 2026-05-21 journal entry. **Verified 2026-05-22**: 18 `PRACTICAL_DELTA_FLOOR` references remain in `tutorials/manuscript_plots.py` (constant definition at line 164, 8 star conditionals at lines 1479/1988/2161/2353/2506/2681/2825/2969, 8 cell-signature usages, 1 docstring paragraph). (7) **SSHFS post-mortem notes (deferred)**: mount has dropped twice in the day after working briefly post-remount, EPERM-on-content-read pattern flagged as likely macFUSE kernel-extension state stuck after auto-mount-on-laptop-wake LaunchAgent, or TCC Full Disk Access lost for the LaunchAgent-context sshfs binary, or kernel-extension auto-unload-on-idle requiring re-approval. Long-term options under evaluation: UCI CRSP (NFSv4 via Mountain Duck/FUSE-T, avoids macFUSE), VS Code Remote-SSH (eliminates mount layer), or rsync-to-local-SSD for active development directories. None implemented. Resume next session: execute the journaled regex revert in `tutorials/manuscript_plots.py`, then continue drafting manuscript Results + remaining Discussion paragraphs.
+
+- **2026-05-22**: Manuscript-display revert + half-budget union experiment set up + library-naming convention codified. (1) **Ran the 2026-05-21-journaled regex revert** on `tutorials/manuscript_plots.py`: 8 star conditionals reverted to unconditional `if _s and _s != "ns":`, 8 cell-signature usages removed, 1 docstring paragraph stripped, 1 constant definition removed, 1 return-tuple updated. 0 remaining `PRACTICAL_DELTA_FLOOR` references. Interpretation-guidance card `50e1fe61` governs significance-star display going forward. (2) **Half-budget union benchmark scaffolded.** Parameterized `compute_budget(reagent_counts, num_products, fraction=0.02)` in `_benchmark_config.py`; added `--budget-fraction` to `run_benchmark.py` shared argparse; scaled CPU estimate by `max(0.6, fraction/0.02)`. Estimate at 1% × 28 libs × 5 methods × 20 reps: 218,800 trials, 10 nodes × 8h wall, 778 CPU-hours. Slurm files in `jobs/full_benchmark_1pct_*.sub` → `outputs/full_benchmark_1pct/`. Post-hoc union script `examples/run_union_recovery_analysis.py`: per (library, query, replicate) computes `union_recovery = |(TT-TS-top-N ∪ RWS-top-N) ∩ GT-top-N| / N` paired by replicate, compares against single-method-2pct from canonical recovery_summary. (3) **First submission (job 52641430) crashed on `amide-suzuki`** — `max(reagent_counts)` over empty list because the reagent-file glob used `folder_name` as the prefix but the actual filenames use `lib_id`. The 2026-05-01 4-site `lib_id → folder_name` rename had inverted the correct convention for non-hybrid hyphenated libraries (the bug was masked because no benchmark since 2026-05-01 had iterated `amide-suzuki` or `groebke-blackburn-bienayme`). Codified the canonical resolution rule as CLAUDE.md Local Norm #11 (`name_key = folder_name if lib_id.endswith("_hybrid") else lib_id`), applied at all 4 sites in `run_benchmark.py`; verified all 28 libraries resolve cleanly. Also codified the `MINIMIZE_LIBRARIES` triple-config-drift gotcha as Local Norm #12. Resume next session: re-push `run_benchmark.py` to HPC3, resubmit `./scripts/submit_benchmark.sh full_benchmark_1pct`, then run union analysis once complete.
+
+- **2026-05-23 (continued)**: Methods cards rewrite + 4pct benchmark staged + 3pct posthoc timeout diagnosed. (1) **6 new equation-level Methods cards** split per user request for one-mechanism-per-card whiteboard layout: `75953048` ([METHOD] Why GMIC — Information-Theoretic Criticality as the Engine for Adaptive Exploration, starts Methods after intro's research gap), `a9c895c1` ([METHOD] GMIC — Definition and Per-Component Criticality Measurement), `9e51b827` ([METHOD] TACTICS RWS — Boltzmann Selection on the Explored Component), `4e272cf7` ([METHOD] TACTICS RWS — EMA-Smoothed CATS Multiplier for Adaptive Temperature), `c0976c3a` ([METHOD] TT-TS — Posterior σ Scaling and Two-Sample Disagreement Selection), `78aef3bb` ([METHOD] TT-TS — EMA-Adaptive heated_scale Based on Disagreement Rate). All six tagged `TACTICS-Manuscript`. Each ~10–12 paragraphs with LaTeX equations; EMA explanation framed as low-pass filter on a noisy signal with method-specific terms (RWS EMA on relative_GMIC, decay 0.1, ~10-batch window; TT-TS EMA on binary disagreement rate, decay 0.02, ~50-cycle window). Two additional old mechanism cards trashed: `f62bf55c` (TT-TS Complete Mechanism Walkthrough) and `311c0bc1` (GMIC/CATS Detailed Mechanism). Two older non-[METHOD] cards (`f94f049d` RWS End-to-End Walkthrough, `92dd644e` original GMIC card) flagged for user review but not trashed. (2) **New result card** `32f0a5f5` ([RESULT] TACTICS Methods Are Never the Worst — Per-Library × Per-Top-N "Pareto Floor" Property) captures the universal-floor finding from the wider-top-N analysis: TT-TS and TACTICS RWS are never the worst-performing method on any of the 112 (library × top-N) cells across 28 libraries × 4 reported top-N. Includes the explicit retraction of the earlier "Pareto dominance" overreach. (3) **4pct benchmark staged**: `jobs/full_benchmark_4pct_{array,merge,posthoc,diagnostic}.sub` mirror the 5pct shape (13 shards × 24h × 48 cores × `--budget-fraction 0.04`) for evenly-spaced 1/2/3/4/5pct sensitivity sweep. (4) **3pct posthoc TIMED OUT at 8h wall** (job 52651982 cancelled, downstream diagnostic 52651983 dependency permanently unmet); root cause is I/O contention on staged data (32 workers reading large per-library shards from one TMPDIR), not CPU shortage. (5) **All posthoc + diagnostic .sub walls bumped to 24h** (1/3/5pct posthoc + 3/5pct diagnostic). Cores/memory kept at original defaults (32 cores / 96G for posthoc, 48 cores / 96G for diagnostic) because increasing them on a busy cluster lengthens queue time without speeding up I/O-bound work. Diagnostic script `examples/run_diagnostic_benchmark.py` patched to accept `--budget-fraction` (was hardcoded to 0.02). (6) **Mechanism correction via code inspection**: an earlier draft of the two-fraction mechanism story claimed TT-TS used reagent-level signal vs RWS's component-level GMIC; code inspection of `src/TACTICS/thompson_sampling/strategies/top_two_selection.py` showed TT-TS uses the *same* GMIC-weighted rotation as TACTICS RWS, so both make the same component-level decision. The corrected mechanism: they differ only in how they implement exploration on the GMIC-selected component (Boltzmann temperature for RWS, posterior σ inflation for TT-TS). Card `a492426f` content corrected before tagging. (7) **`heptabase-cli-skills` skill installed** at `.claude/skills/heptabase-cli/` from `heptameta/heptabase-cli-skills` GitHub repo (1 SKILL.md + 1 references/property-values.md, ~9KB). Heptabase desktop CLI is now reliably available for tag/trash/edit operations after the install settled. Resume next session: (a) tag manual UI action — none, all cards tagged via CLI; (b) when 3pct posthoc resubmission completes, append top-2500 numbers to ROCS-decay card `bf67bcd4` as an addendum; (c) start drafting Results §Headline empirical claim using the never-worst data + per-library winner counts from card `32f0a5f5`; (d) draft Discussion §Two-fraction SAR characterization prose from `5a862478` / `a492426f` / `04b96e77`.
+
+- **2026-05-23**: Framework rewrite — three-axis → two-fraction SAR characterization. (1) **Axis 3 (σ-robustness) dropped** because it is a downstream consequence of reagent-count balance and warmup stochasticity (system state), not a structural library property; the card `a19b3878`'s own May-15 addenda had already partially retracted it. (2) **Mean-driven vs σ-driven mechanism framing dropped** as imprecise — code inspection of `top_two_selection.py` confirmed both TACTICS methods make the *same* GMIC-weighted component-rotation decision; they differ only in *how* exploration is implemented on the selected component (TACTICS RWS = Boltzmann temperature softening over posterior samples; TT-TS = posterior σ scaling + two-sample disagreement). The σ-inflation mechanism is reagent-shape-aware (preferentially exposes under-explored high-σ reagents); the Boltzmann mechanism is reagent-shape-agnostic (uniformly softens across all low-mean reagents). This is the corrected mechanism for why TT-TS wins on the majority of benchmark libraries. (3) **Terminology standardized**: "explore/exploit" is the universal concept; "heating/cooling" applies only to RWS's literal Boltzmann temperature; TT-TS's analogous operation is "σ inflation/deflation." Cards rewritten with this terminology throughout. (4) **Three new Heptabase cards** replacing the old single long card: `5a862478` ([METHOD] Two-Fraction SAR Characterization — definition + ratio decision rule); `a492426f` ([METHOD] Two-Fraction Mechanism — Why TT-TS Wins More Often); `04b96e77` ([RESULT] Two-Fraction Framework Holds on ROCS — fractions shift with top-N, unifying docking and ROCS top-N decay stories). All ~10–12 paragraphs each per the new card-length feedback memory. Cards still need: (a) `TACTICS-Manuscript` tag applied in Heptabase UI (CLI flaky), (b) old `a19b3878` trashed, (c) old `f117b817` trashed (multi-cluster ROCS card, flagged "Not correct"). (5) **Wider-top-N posthoc data** generated this session by extending `TOP_N_THRESHOLDS` in `examples/run_posthoc_analysis.py` to `{500, 1000, 1500, 2000, 2500}` alongside existing 25–250; reran 1pct and 2pct posthoc serially (each ~15–20 min). Headline cross-budget finding: TT-TS-vs-Legacy-RWS gap on ROCS aggregate grows from +3.5 pp (top-100) to +10.4 pp (top-2500); on docking aggregate +8.5 pp → +11.4 pp. Per-library: TT-TS wins 21–23/28 libraries at every reported top-N; combined TACTICS wins 26–27/28; neither TACTICS method is ever the worst on any library × top-N. Legacy RWS crosses below Legacy Greedy on 2-comp ROCS at ~top-150 and on docking aggregate at ~top-1000 — symmetric thermal cycling becomes counterproductive at deployment-relevant top-N. (6) **`tutorials/manuscript_sar_plots.py` extended** with a new cell that computes hidden-hub fraction and μ-reachable fraction per (library, top_n) from ground-truth parquets, identifies empirical winner per cell from `outputs/full_benchmark_ema_cats/analysis/recovery_summary.parquet`, and plots prediction vs empirical winner with prominent post-hoc disclaimer. (7) **Cap on extended top-N grid set at 2500** (not 10000) because the 10K-budget ROCS libraries would have top-10000 = 50–67% of total evaluations, turning the metric into a coverage test rather than a prioritization test — the user's correct call. (8) **Manuscript prose updates**: `manuscript.qmd` Results §Cross-docking-pipeline + §Top-N sensitivity + Discussion §Two-fraction SAR characterization (renamed from §Three-axis) rewritten with explore/exploit terminology, post-hoc scope, and the unified docking-vs-ROCS framing. (9) **3pct + 5pct HPC3 jobs queued earlier** — recovery_summary updates from those will feed back into the SAR notebook plot automatically. Resume next session: tag the 3 new cards + trash 2 old cards in Heptabase UI; check on 3pct/5pct HPC3 completion; iterate on Results+Discussion drafting now that the framework cards are clean.
+
+- **2026-05-22 (session 3)**: 1pct benchmark posthoc completed, half-budget-union hypothesis resolved (does NOT generalize), sensitivity-analysis framing replaces it, 3pct benchmark queued. (1) **1pct array completed on HPC3** (218,800 trials, 28 libraries) while session opened. Local posthoc OOM-crashed the laptop again with default ProcessPoolExecutor settings; reverted `examples/run_posthoc_analysis.py` and `examples/run_union_recovery_analysis.py` to default `--workers 1` (serial); kept the flag so HPC3 .sub files (which pass `--workers 32` explicitly) still work. Memory `feedback_local_posthoc_8_workers.md` updated: no multiprocessing on the laptop, ever. (2) **1pct posthoc ran serially on laptop in 13 min** (much faster than feared). Aggregate top-100: TT-TS 81.0 (winner), RWS 80.1, L-RWS 75.5, Bal-Greedy 75.2, L-Greedy 74.0. **TACTICS-vs-Legacy gap grows at 1pct** (ROCS +5.5 vs +3.5 pp; docking +10.1 vs +8.5 pp). Best-TACTICS clears ≥50% top-100 on every library; best-Legacy fails to clear 50% on adenine_hybrid. (3) **Union recovery analysis ran serially in 9 min**; result **does not support the active-hypothesis prediction**. Win rate vs best-single-2pct ranges 31–40% across top-N (mean diff −0.6 to −1.0 pp). Catastrophic on hidden-hub libraries (adenine −20 pp, adenine_hybrid −23 pp): both methods fail on the same replicates at 1pct → union of failures is still a failure. Small wins on already-saturated libraries (thrombin, thrombin_hybrid) — artifact of saturation, not diversification. Union hypothesis retracted; better-framed sensitivity-analysis claim takes its place. (4) **Discussion with user clarified manuscript framing**: TACTICS still beats *all* Legacy methods on *every* library at 1pct, including hidden-hub (where the TACTICS-vs-L-RWS gap actually expands: +19.6 pp on adenine_hybrid 1pct vs +10.5 pp at 2pct). Refined claim for manuscript: "≥50% top-100 recovery on every benchmark library at 1pct budget; TACTICS-vs-Legacy advantage preserved and amplified on most libraries." User's directive: open Results with the 1pct constrained-budget benchmark (motivates TS choice), introduce 2pct as operating-point selection (justified by bimodal-failure threshold on hidden-hub libraries), use 3pct as saturation upper-bound confirmation. (5) **3pct benchmark queued**: generated four new .sub files (`full_benchmark_3pct_{array,merge,posthoc,diagnostic}.sub`) mirroring 1pct layout; bumped array shards 10 → 12 and wall 8h → 24h to handle the 3× compute. Patched `examples/run_diagnostic_benchmark.py` to accept `--budget-fraction` (was hardcoded to 0.02 — the previously-named `full_benchmark_1pct_diagnostic.sub` had silently been running at 2pct; never actually executed so no stale data). Wrote `scripts/submit_benchmark_full_chain.sh` (generic array→merge→posthoc→diagnostic dependency-chain submitter). Jobs submitted but not yet started at session end. (6) **Posthoc / union-analysis scripts modified**: both now default to serial; both retain `--workers` flag for HPC3 use. Resume next session: when 3pct chain completes, add the third data point to the budget-sensitivity figure, finalize Methods §Sensitivity-analysis + Discussion §Optimal-budget paragraphs, push the 3 new Heptabase cards ([RESULT] 1pct sensitivity, [METHOD] operating-point selection, [DEAD END] half-budget union).
+
+- **2026-05-22 (session 2)**: Reproducibility-code subdir built in the TACTICS-manuscript repo on branch `code-subdir` (off `main`, demo-branch work stashed). 1pct benchmark resubmitted on HPC3 and currently running at end of session. (1) **`~/Desktop/TACTICS-manuscript/code/`** scaffolded with 6 logical commits: `config.py` (single source of truth — consolidates `LIBRARIES`, `name_key` rule, `MINIMIZE` flag, method definitions, `compute_budget`, `estimate_resources` into one module that every runner imports — eliminates the lib_id/folder_name and triple-config-drift bugs by construction); `run_benchmark.py` (single-node multiprocess runner; strips Slurm sharding, TMPDIR staging, DFS paths; keeps `--workers`, `--skip-existing`, `--budget-fraction`; resumable across walltime kills via manifest comparison); `run_posthoc.py` (recovery + within-library ANOVA/Tukey, planned-Welch Bonferroni-4, across-library Friedman/Nemenyi — verified end-to-end on the canonical 25-library benchmark, 218,800 trials); `run_union_recovery.py` (paired-replicate half-budget union); `make_figures.py` (non-marimo headless figure regenerator — 14 named figures including the per-library Δ-bars, FRED/HYBRID docking, ROCS top-N, Zhao Fig 5 extended); `_worker.py` (copied verbatim from engine repo); `tests/test_library_resolution.py` (38 passing tests; data-dependent tests auto-skip when `data/scores/` is empty). (2) **`~/Desktop/TACTICS-manuscript/data/`** populated: `reactions.csv` + 24 reagent subdirs (5.8 MB total, bundled in-git); `download_scores.py` fetches the 28 score parquets (~3.2 GB) from a Zenodo deposit (ZENODO_DOI is a placeholder until deposit exists; `--source file:///...` works for local mirrors). (3) **Top-level README** rewritten to walk the reader end-to-end from `conda env create` through `quarto render manuscript.qmd`, naming every command. (4) **Three TODO figures** (SAR posterior-separation, Tukey CI, pre-merge adenine) flagged in `make_figures.py` with pointers to their marimo source cells. (5) **HPC3 1pct benchmark resubmitted** after the morning's `name_key` fix landed; jobs currently running at end of session — `union_recovery_analysis.py` runnable once `outputs/full_benchmark_1pct/` is complete. Resume next session: when HPC3 jobs complete, run posthoc + union analysis, then either iterate on remaining TODO figures or merge `code-subdir` → `main` in TACTICS-manuscript (then return to the demo branch via stash pop).
+
+- **2026-05-28**: ROCS manuscript card scaffold + analytical chained-decomposition + journal-strategy resolution. (1) **Manuscript-plots reorganization**: `tutorials/manuscript_plots.py` split into two notebooks for whiteboard organization. New `tutorials/manuscript_plots_ROCS.py` (1934 lines, 31 cells) holds all ROCS-related plots — sensitivity, mean recovery, per-library Δ, Zhao Fig 5 vector + Δ plots, Tukey HSD CIs, ROCS budget-sensitivity panel. `manuscript_plots.py` reduced to 2262 lines (36 cells), retains DOCK sections + DOCK budget-sensitivity panel. Setup cells (imports / UI / data load / scoring-type rule) and budget-sensitivity load+compute cells duplicated in both for self-containment. Both notebooks parse, load via marimo importer, complete cell-variable graph. (2) **Chained-decomposition stacked-bar visualization** added as new cell to ROCS notebook after the aggregate Δ-bars. Splits each TACTICS-vs-Klarich gain into Zhao-2025 lift (Δ Legacy RWS, Legacy Greedy) + TACTICS algorithmic lift (Δ TACTICS, Legacy RWS), stacked at the zero line so when Zhao goes negative (2-comp ROCS at wider top-N) the segment extends below zero and TACTICS recovers it visually. Pooled across 20 ROCS libs at top-100: total +5.25 pp, Zhao 33%, TACTICS 67%; at top-250 redistributes to 3% / 97% as Zhao approaches zero. On 2-comp specifically Zhao reaches −2.28 pp at top-250 — the strongest visual indictment of Legacy RWS's wider-top-N pathology in the manuscript. (3) **Per-library and aggregate Δ-bars converted to TT-TS-only** (from prior "best of TT-TS / TACTICS RWS"): empirical scan across 200 (ROCS lib × top-N) cells found TACTICS RWS beats TT-TS on only 21 cells with max gap 0.79 pp (poparov top-25), persistent only on `betti` (~0.5 pp), narrow-top-N flips on `orru`/`poparov` (top-N 25–100). All below the 2 pp practical-significance threshold; the practical equivalence justifies headlining TT-TS without method-selection caveat. Per-library plot's earlier T/R per-bar tagging removed. Tukey-HSD library dropdown also filtered to ROCS-only (was showing all 28 libs). (4) **10 new Heptabase cards** created via update-status + earlier in session: `[METHOD] Statistical Significance vs Practical Significance` (`ee819848`, reviewer-defense framework on why 20-rep power flags sub-1 pp effects as significant and how the manuscript reports both axes); `[RESULT] R1` budget sensitivity (`d35ab914`); `[RESULT] R2` mean recovery at 2% (`ac504c68`); `[RESULT] R3` per-library Δ (`8b668f51`); `[RESULT] R4` chained decomposition (`05cf848e`, headline numbers); `[RESULT] R5a` diversity-recovery vector (`fcbae16c`); `[RESULT] R5b` diversity-recovery Δ-slope (`75762f09`); `[RESULT] R6` Tukey HSD completeness (`a306fcfc`); `[DISCUSSION] D1` diversity-tradeoff centerpiece (`1e5691da`); `[DISCUSSION] D2` practical-vs-statistical commentary (`d7996148`). R5 split mid-session into R5a + R5b for whiteboard pairing with PDF figure cards; back-references in R2/R4/D1 rewired via ProseMirror JSON edit-and-save (heptabase note read → modify → save with contentMd5); old combined R5 trashed (user-approved). All cards tagged Project-TACTICS-Primary-Engine + Type-Manuscript + (Section-Results / Section-Discussion / Section-Methods). (5) **PDF embedding limitation discovered**: Heptabase CLI + MCP have no file-upload primitive; PDF cards must be created by drag-and-drop in the desktop app. User will manually drop figures onto the manuscript whiteboard adjacent to their explanation cards. (6) **Journal-fit strategy resolved**: Digital Discovery / JCIM / JCTC / J Cheminform are the realistic submission targets (similar IF 5.5–7.1, all are field-leading methods venues). Nature Computational Science is a stretch (~30–40% editor pass); Nature Communications is genuine stretch (~10–20%); both require substantial reframing beyond the current structure. Honest assessment: TACTICS' recovery magnitude is incremental on Zhao 2025 / Klarich 2024 and would not carry Nature on its own; the novel content is the chained-decomposition's demonstration of Zhao's previously-undocumented 2-comp wider-top-N failure mode plus TACTICS' resolution of it, plus the diversity-tradeoff finding. If Nature is attempted, presubmission inquiry first is recommended (~1 hr effort, cheap signal). (7) **Self-correction**: my early critique recommendation to add a computational-cost result card was withdrawn after the user pointed to existing Methods cards (`84ae91a7` Evaluation Budget Equality, `a7546df6` Budget Calculation) — wall-time overhead of GMIC is negligible vs the scoring-function call which dominates and is already controlled by the budget formula. Resume next session: build parallel docking-section card scaffold (R-DOCK series + DOCK discussion); begin first prose drafts of ROCS Results paragraphs in `~/Desktop/TACTICS-manuscript/manuscript.qmd`; user will drop PDFs onto the manuscript whiteboard.
+
+- **2026-05-29**: Docking landscape figures + diagnostics mechanism figures + amidine_035 counterfactual re-run + 7 docking Heptabase cards. (1) **Docking SAR landscape figure set** (standalone renderer `experiments/_debug/render_ms_plots.py`, reconciled into `tutorials/manuscript_sar_plots.py` cells): the M–S **landscape navigation map** (each docking library by its dominant enriched hub's average compound quality −z(mean) × partner-specificity z(std), colored by winning method, FRED→HYBRID landscape-shift arrows), the **ranking heatmap** (top-100 recovery, 5 methods × 8 docking libs; rank strip dropped per user), and the **hub-allocation mechanism** (Greedy at origin / TACTICS up-right on hidden-hub libs; x = × uniform allocation). Map axes **reframed to the interaction interpretation** (mean = a reagent's avg compound score over partners; σ = partner-specificity); hard navigation regions dropped (descriptive scatter) because no single-hub (mean, σ) marginal separates greedy-easy from greedy-hard. Hub = single most-enriched reagent; top-5 averaging tested+rejected (dilutes the genuine wide-σ hubs: adenine_hybrid 6.33→1.98, amide_hybrid 6.75→4.07). Probe `experiments/_debug/docking_landscape_probe.py` (M1–M6). (2) **Diagnostic mechanism figures (MECH-2/4/5)** (`render_diag_mech2.py`, `render_diag_mech45.py`) added to `tutorials/diagnostic_benchmark_plots.py`: MECH-2 per-component GMIC vs cycle (RWS|TT-TS, single representative query rxn206 q047 / groebke q035, quinazoline negative control, relabeled "probability of heating/exploration" + temperature/disagreement strips); MECH-4 per-query GMIC-direction heatmaps (rxn206 50/59, groebke 24/51/34; terminology corrected: high-GMIC=critical/exploited, low=flexible/explored); MECH-5 cross-library min_GMIC vs advantage. (3) **amidine_035 counterfactual diagnostics** (`run_premerge_adenine_diag.py`): re-ran pre-merge adenine (27-amidine reagents, `adenine.parquet.bak`, 20 reps, 2%, diagnostics; `data/` untouched) — sanity passed (TT-TS 95.5±4.4 0/20-fail, RWS 74.5±28.0 5/20-bimodal). **Key finding**: GMIC trajectories identical pre/post (TT-TS gap 2.25→2.22, RWS 1.21→1.23) → the method flip is reagent-level, not component-level (TT-TS σ-specialist blind to narrow-σ amidine_035; RWS Boltzmann generalist). Diagnostics at `outputs/debug/adenine_premerge_diag/`. (4) **min_GMIC verification**: cross-library Spearman −0.72, per-query within-ROCS Pearson −0.32 — corrected an earlier "per-query null" claim (3-lib spot-check that didn't hold on full 20-lib). (5) **7 Heptabase cards created + tagged** (Project-TACTICS-Primary-Engine + Type-Manuscript + Section-Results/Discussion): R1 docking-landscape `d868cd7e`, R2 amidine_035-counterfactual, R3 per-query-GMIC `708c0a08`, R4 min_GMIC-predictor, R5 GMIC-rises-exploited `7233805a`, D1 interaction-reframe `63e65a69`, D2 flip-mechanism `79065473`. Heptabase desktop CLI now **0.4.0** (heptabase-cli skill nominally targets 0.3.x; commands verified working). Cross-references are plain-text by title — **href backlinks pending a follow-up ProseMirror save pass**. Resume next session: add href backlinks across the 7 docking cards; begin docking Results/Discussion prose in `~/Desktop/TACTICS-manuscript/manuscript.qmd` from the card set; optional counterfactual-GMIC-overlay figure declined this session (MECH-2 + hub-spectrum cover it).
+
+- **2026-05-26 to 2026-05-27**: Manuscript-scope-tightening + budget-sensitivity figure implementation. (1) **Scope reduction**: dropped wider top-N (500-2500) from manuscript entirely; capped all `manuscript_plots.py` cells at top-N ≤ 250 (line plot filter + 8 per-library docking cell filters + removed Section 2b top-2500 Zhao plots, 382 lines). Cleaned `manuscript_sar_plots.py` (1040 → 461 lines): removed docking multi-top-N grid + two-fraction confusion-matrix cells; added ROCS individual SAR explorer with mean-per-query hit coloring (consistent with diversity averaging); fixed adenine/thrombin parsers (regex / dipeptide join) before deletion. (2) **Budget sensitivity Section 4 implemented in `manuscript_plots.py`** (5 cells, producing `sensitivity_rocs.pdf` and `sensitivity_dock.pdf`, 2×2 line+bar panels each: top row = recovery vs budget per method, bottom row = Δ(best-T − best-L) vs budget; columns = 2-comp / 3-comp). Aggregation methodology bug found and fixed (per-library best-T computation, not aggregate-then-max). (3) **Sub-1% benchmarks generated then scoped out**: 8 .sub files for 0.1% and 0.5% generated in `jobs/`, but pre-submission tabulation showed floor binds on 26/28 (0.1%) and 24/28 (0.5%) libraries — not worth running. (4) **Hidden-hub SAR framework tested on ROCS and ruled out**: tested both absolute z-threshold and relative Δz metric formulations; correlation r = -0.49 (p=0.15) at top-100, not significant; framework retained for docking only. (5) **Heptabase cleanup**: trashed `739bb5d0` (Benchmark Setup, superseded by §2.3 cards) and `cdfa2e00` (Statistical Analysis Framework, stacked addenda); created `f3690bb9` ([METHOD] Statistical Analysis Overview); removed lowercase `method` tag from Polars forkserver card. Saved `project_docking_dataset_provenance.md` memory note. (6) **New Heptabase cards created this session** via update-status: `ef20ea7e` (sensitivity result + aggregation fix), `84b632f5` (dead-end: hidden-hub on ROCS), `05b66809` (dead-end: sub-1% benchmarks). Resume next session: write manuscript Results §Budget Sensitivity prose from the corrected figure infrastructure; draft Discussion paragraph on DOCK library-dependent vs ROCS uniform finding; render the two PDFs and visually verify.
+- **2026-05-24 to 2026-05-25**: Long manuscript-and-infrastructure session. (1) **Heptabase taxonomy overhaul**: 3-axis tag scheme implemented (Project-* × Type-* × Section-*/Experiment-*/Subsection-*). Created `Type-Manuscript/Experiment/Conclusion`, `Section-Introduction/Methods/Results/Discussion`, `Subsection-Objective/Research-Gap`, `Experiment-Objective/Methods/Results`. Renamed `TACTICS-Manuscript → Project-TACTICS-Primary-Engine`, `TACTICS-Paper-N-* → Project-TACTICS-Paper-N-*`. Retired 6 granular `Type-*` tags + 3 legacy lowercase tags. **Heptabase internal-link format discovered and documented in auto-memory**: real backlinks require `link` mark with `href=https://app.heptabase.com/{workspace_id}/card/{card_id}`; `[[name]]` wikilink syntax is plain text. ~108 inline link marks added across 21 cards via bulk-relink script. (2) **Manuscript card structure built**: 16 new Type-Experiment cards for 1pct/2pct/3pct budget benchmarks (1 shared Objective + 3 Methods + 12 Results across 4 comparison axes each). 5 new Methods §2.3–§2.5 cards (Targets/Library Inventory/Reagents+Chemistry, Sensitivity Sweep Design, ROCS Diversity Framework). Rewrote SAR Posterior-Separation methodology card (`ebee37ac`), Lean Enhanced Warmup card (`27cbde69`) with new dual-argument structure (GMIC-signal on imbalanced + lean-efficiency on balanced), and both §2.1 transition cards (`7e34aa1e` RWS, `0bc83dbd` TT-TS). Trashed `91b0e70b` (1pct manuscript Results), `7c446b85` (canonical 2pct), `afc6774d` and `f94f049d` (end-to-end mechanism walkthroughs). (3) **3pct + 4pct posthoc completed**: 3pct (job 52670834) wrote 9 analysis parquets in 11m 33s; 4pct (job 52670979) in 30m. Aggregate top-100: TT-TS 90.7 (3pct) → 93.0 (4pct); see new Established Results. (4) **5pct array silent-OOM root cause diagnosed**: original 5pct array reported all 14 tasks COMPLETED exit 0 but 2 tasks were cgroup-OOM-killed at the 96 GB ceiling during the per-library merge step; .sub didn't check Python exit; manifest was short by 31,258 trials. Patched `examples/run_posthoc_analysis.py` and `run_union_recovery_analysis.py` to forkserver start method + per-library failure resilience wrapper. Bumped 5pct array .sub to `--mem=192G --tmp=150G -t 48:00:00 --skip-existing`. 5pct full re-run submitted (job 52673429) but `--skip-existing` discovered to NOT filter trials in array-job context (reads from per-task TMPDIR, not persistent DFS manifest). Re-run currently processing full 218,800 trials anyway, ~9 h estimated remaining. (5) **Memory-allocation framework**: empirical formula `mem ≈ workers × 15 × largest_compressed_GB × 2 + 10` derived from this session's posthoc measurements (peak per worker ≈ 30× compressed data due to polars decompression + concat working memory). Useful within polars-shaped workloads; pilot-then-measure is the universal protocol. Resume next session: verify 5pct re-run completed without silent OOM (Slurm exit code now meaningful); if successful, complete the budget-sensitivity sweep; if not, the patched posthoc resilience wrapper should still produce partial output for the libraries that did complete.
## Ruled Out / Dead Ends
@@ -109,7 +168,18 @@
NEVER delete entries from this section.
--\>
+- **Half-budget union strategy does not substitute for full budget on hidden-hub libraries**: Per-replicate union of `TT-TS-1pct ∪ TACTICS-RWS-1pct` top-N picks vs best-single-2pct, 28 libraries × top-N ∈ {25,…,250}. Win rate: 36.5% overall, dropping monotonically from 40.4% at top-25 to 31.4% at top-250. Catastrophic on hidden-hub docking (adenine −20.1 pp, adenine_hybrid −23.0 pp, amide_hybrid −8.4 pp). Small wins (+1.9 to +3.4 pp) on imbalanced mean-separable docking (thrombin, thrombin_hybrid) and easy saturated ROCS libs — but these are saturation artifacts, not diversification benefits. Root cause: at 1pct budget, per-reagent observation density falls below the criticality-signal floor (STATUS line 38) → both methods bimodal-fail on the *same* replicates on the libraries where union *should* help most. Rules out any "halve budget, take union of methods" strategy as a workaround for a priori method selection; the descriptive landscape framework remains a post-hoc explanatory tool and method selection still requires either pilot screening or running both methods at full budget.
+- **Batch size reduction (5–50) does not improve recovery or reliability over batch_size=100**: Full benchmarks at batch sizes 5, 10, 25, 50, and 100 (24 libraries, 5 methods, 20 replicates each). Aggregate TT-TS recovery: 86.1 / 86.1 / 86.2 / 86.3 / 86.4 — monotonically flat with a slight downward trend at smaller batches. Per-query analysis (2,184 queries): 0/2,184 queries show inter-batch-size variance exceeding replicate variance; replicate noise is 4.2× larger than batch-size variation. No individual library or query benefits from smaller batches. Root cause: GMIC is computed from cumulative posteriors and stabilizes by cycle 0–13 regardless of batch size; more frequent rotation decisions re-derive the same signal. Rules out batch_size reduction across the full tested range as a tuning lever. `outputs/full_benchmark_batch{5,10,25,50}/`.
- **Bayesian additive decomposition for exploration guidance**: Auxiliary regression model providing per-reagent uncertainty bonus and residual-informed rotation adds no measurable value on top of TT-TS (mean delta -0.06, symmetric help/hurt distribution, 65,430 trials). Root cause: TT-TS already allocates exploration via posterior disagreement, making the additive model's signal redundant. Rules out any auxiliary regression model for exploration guidance when the primary selection policy already targets posterior uncertainty.
+- **Benchmark .sub shells without explicit Python exit-code check are unsafe**: Standard pattern of `python examples/run_benchmark.py run ... [no guard] ... rsync ... echo Done` masks Python crashes (OOM, polars panic, unhandled exceptions). The 2026-05-23 5pct array job reported all 14 tasks `COMPLETED exit 0:0` but 2 tasks were silently cgroup-OOM-killed during the per-library merge step at exactly the 96 GB `--mem` ceiling; their manifests never wrote and 31,257 trials went missing. Required fix on every `jobs/full_benchmark_*.sub` file: append `|| { echo "ERROR: python failed (exit $?)" >&2; exit 1; }` to the Python invocation. Rules out the current .sub template as safe for any non-trivial Python workload; rules out trusting Slurm `COMPLETED` state without manifest-count verification when the .sub doesn't propagate Python's exit code.
+- **`run_benchmark.py --skip-existing` does not filter trials in the array-job context**: `_load_existing_manifest` reads from `--output-dir`, which the array .sub files set to a per-task TMPDIR location that is freshly created and empty at task start. The persistent DFS manifest is never seen and `existing_keys` is always empty, so every task runs its full assigned shard regardless of what already exists on DFS (verified empirically on 2026-05-25 5pct re-run: every task reports `Found 0 existing trials (will skip)` and processes 15,629/15,629 trials). Rules out the workflow "resubmit the same array with `--skip-existing` to top up missing trials" — the optimisation is defeated by the per-task TMPDIR staging pattern. Fix requires either an `--existing-manifest-dir` CLI flag decoupling read-location from write-location, or pre-staging the DFS manifest into TMPDIR before invoking Python in the .sub.
+- **Hidden-hub SAR framework does not generalize to ROCS libraries**: Tested two formulations (absolute `|z(mean)| ≤ 1.5` threshold + relative `Δz = z_topenriched − z_baseline_topN` at 5%/10%/25% cutoffs) across top-N ∈ {25, 100, 250, 1000, 2500} on 3-comp ROCS libraries (poparov, betti, mannich, orru). Absolute-threshold rule classifies ~87% of reagents as "hidden hubs" by default (smooth-unimodal reagent-mean distribution → 13% tails) → ratio always > 1 → "predicts TT-TS" for 100/100 ROCS cells (uninformative). Δz metric shows positive values everywhere (top-enriched ≡ high-mean by selection bias on ROCS). At top-100, 10% cutoff, n=10 3-comp libraries: Pearson r(mean_Δz, best-T − best-L) = −0.49, p = 0.15 (not significant); half-works (under-performers correctly flagged, over-performers missed). Rules out per-reagent-mean-based SAR classifiers for ROCS narrative.
+- **Sub-1% library-fraction benchmarks (0.1%, 0.5%) are floor-dominated**: Budget formula `max(fraction × library_size, max_reagent_count × 5 + 10K)` ensures the 10K-search floor binds on 26/28 libraries at 0.1% and 24/28 at 0.5%; only quinazoline + quinazoline_hybrid (19.6M products) escape the floor at 0.1%. Benchmarks would produce IDENTICAL effective budgets to 1% on 26/28 libraries — no genuinely new sensitivity information. .sub files generated as `jobs/full_benchmark_{0.1pct,0.5pct}_*.sub` but not submitted. Rules out sub-1% benchmarks under the current floor design without floor redesign (which would invalidate the existing 1-5% sweep comparison basis).
+- **sEH DEL read-count is mean-separable — greedy beats TACTICS, no graceful degradation**: Optimizing experimental DEL read counts (soluble epoxide hydrolase, Zhang 2023 JCIM) with TACTICS to showcase a third evaluator type and a "finds experimental binders in few iterations" claim. Pilot (300K budget = 0.03% of library, 5 seeds, top-100 recovery): Greedy 61% > RWS 59% (bimodal) > TT-TS 26%; total binder-recall Greedy 51% > RWS 33% > TT-TS 13% (all methods still enrich 250–1900× over random). Root cause: the read-count landscape is strongly mean-separable (Zhang 2023 per-building-block P(bind) predicts binding at AUC 0.961 — the high-min_GMIC regime), and TACTICS RWS/TT-TS do NOT gracefully degrade to greedy when exploration is unwarranted (GMIC-weighted rotation forces an explore+exploit selection every cycle; the within-component gates only modulate temperature, never disable exploration — there is no absolute-criticality off-switch). Rules out any mean-separable / high-min_GMIC / greedy-reachable objective (DEL read-count, additive experimental enrichment signals, any library the two-fraction framework predicts greedy-reachable) as a "TACTICS beats baselines" showcase; does NOT rule out hidden-hub / non-additive landscapes. Remedy is a method change (a criticality gate collapsing RWS/TT-TS to greedy when min_GMIC indicates no component needs exploration), not a reframing. Retained as the diagnostic dataset for that fix in TACTICS v2 / Paper 1b. Artifacts: `outputs/debug/seh_build/`, `experiments/_debug/seh_pilot.py`.
+- **Two-fraction metric misclassifies smooth-additive docking (quinazoline)**: The absolute z(mean)≤1.5 band classifier (AND-rolled across components) flags ~90% of quinazoline's top-100 as "hidden-hub-dependent" (≈ hard adenine) despite quinazoline being greedy-easy — it counts marginal-mean band membership, not greedy reachability, and saturates on any unimodal reagent-mean distribution. Rules out absolute-marginal-z-threshold classifiers for greedy-vs-TACTICS on smooth/unimodal-mean libraries (ROCS AND smooth-additive docking); the real boundary is concentrated-hidden-hub vs smooth-additive, not docking-vs-ROCS. Two-fraction scalar dropped from the manuscript (two-fraction framework cards since retired). Extends the ROCS dead-end (84b632f5).
+- **Idealized-greedy containment curve as a general accessibility axis**: Containment (fraction of true top-N whose reagents fall within the top-f of each component's mean ranking) does NOT order docking libraries by TACTICS advantage — the x-axis (fractional rank) is not budget-normalized, and coverage-depth ≠ greedy success (greedy is select-then-EVALUATE, needing additivity, not mean-rank position; the additive-model top-N recovery is ≈uniformly low even on easy amide, 4%). Only cleanly flags the extreme coverage-failure libraries (adenine pair, flat-early). Rules out any single fixed-depth "reachability/coverage" scalar as a winner-predictor; coverage is one of ≥3 mechanisms. Built + added to the SAR notebook then REMOVED.
+- **Interaction-fraction-at-the-hits (φ_int) does not predict greedy difficulty**: φ_int = Σ(ŝ−s)/Σ(ḡ−s) over the top-N (interaction residual share of hit advantage; ŝ = additive main-effects prediction) fails to separate greedy-easy from interaction-driven docking — quinazoline (easy) φ_int@100 = 0.551 ≈ amide_hybrid (hard) 0.549, and easy-quinazoline ranks 3rd-highest; φ_int only re-derives the coverage axis (isolates the adenine pair, 0.62/0.77). Root cause: interaction MAGNITUDE at the hits ≠ greedy difficulty — greedy is select-then-evaluate, so failure is reagent-level mean-UNreachability (adenine amidine_003 never sampled), not score non-additivity (quinazoline is 55% interaction yet easy). Rules out ANY score-additivity metric (global R², mean→hit ρ, hit-localized φ_int) as the "why greedy succeeds" axis; the governing quantity is reagent-level reachability (already the coverage + allocation axes). Extends 56dc09a3; a warmup-φ_int inherits the conflation → use reagent-level posterior separation (min_GMIC-from-warmup) instead. `experiments/_debug/interaction_fraction_probe.py`.
+- **Post-hoc silently writes a truncated recovery_summary when library workers OOM**: `run_posthoc_analysis.py` catches per-library `BrokenProcessPool` (OOM-killed worker), prints "continues with K/N libraries", writes a partial `recovery_summary`, and exits 0 — so Slurm reports COMPLETED while overwriting a good 28-lib summary with a partial one. On 2026-06-04 the under-provisioned `full_benchmark_1pct_splice_posthoc.sub` (96G/32w) OOM-killed the workers for all 21 large-parquet libraries (20 ROCS + quinazoline_hybrid), corrupting `outputs/full_benchmark_1pct/analysis` (10,500 rows / 7 docking libs vs 3,282,000 / 28) and dropping every ROCS 1% point from `sensitivity_rocs.pdf`. Blast radius is 1pct ONLY (2/3/4/5pct + ema_cats intact; sibling subs already 248–370G/2–4w). Rules out sizing the posthoc process pool by core count rather than per-item peak memory (peak RSS ≈ workers × largest-expanded-parquet → bound workers by `mem / max_parquet_footprint`, NOT `cpus-per-task`); rules out trusting any exit code (Slurm's or Python's) without asserting artifact completeness (`n_out == n_expected`) before overwriting. Fixed sub → 248G/4w (mirrors 3pct); 1pct `recovery_summary` restored 2026-06-05 via local serial re-run (`--workers 1`, 28 libs / 3,282,000 rows). The code-level exit-on-partial guard remains unimplemented. Application-layer sibling of c01bbfe2.
## Open Questions
@@ -118,6 +188,7 @@
These may become future hypotheses in new worktrees.
-->
+- **Layer-2 diagnostic — within-component adaptive exploration intensity (CATS multiplier / disagreement)**: extend the GMIC signal→decision→result grammar one level down. RWS: `ema_relative_gmic` (signal) → `cats_multiplier` (decision, Boltzmann temperature) → within-component sampling breadth (result); TT-TS: `disagreement_ema` → `heated_scale` (σ-inflation) → within-component sampling breadth. Open question: which library best demonstrates the breadth tracking the parameter — thrombin's 3844-dipeptide component may sit near-maximal breadth throughout, so prototype on a balanced library (rxn206 / orru, ~100 reagents/component) first. Planned function `plot_adaptive_intensity`.
- How does the observation gate behave in the high-N regime?
- Optimal transition point from James-Stein shrinkage to empirical variance?
- MOTS/TSEMO extension: what is the right surrogate for multi-objective TS over combinatorial libraries?
@@ -127,7 +198,8 @@
- **Mechanism visualization plots — DONE (2026-04-02)**: Marimo notebook `tutorials/diagnostic_benchmark_plots.py` renders all 21 diagnostic figures (10 RWS + 10 TT-TS + 1 min_GMIC scatter). Three plot functions added to `diagnostic_plots.py`: `plot_rws_diagnostic` (dual-axis GMIC + temperature with divergence gate shading, 95% CI, rolling average), `plot_ttts_diagnostic` (dual-axis disagreement EMA + heated_scale with threshold bands, 95% CI), `plot_min_gmic_vs_advantage` (scatter with SAR difficulty zones). Remaining: per-reagent rank bump charts and anytime recovery curves require additional per-iteration tracking not yet implemented.
- **min_GMIC as a prescriptive diagnostic**: min_GMIC < 1.0 predicts where GMIC-directed exploration provides value. Could be computed from warmup posteriors alone (before search begins) to estimate expected TACTICS advantage and guide method selection. Connects to Paper 3 (library diagnostics).
- **Continuous GMIC-based temperature gradient (follow-up study)**: Replace binary heated/cooled assignment with per-component temperature interpolated from GMIC: T_c = beta + (alpha - beta) * (flexibility_c / max_flexibility). Most flexible component gets alpha, least flexible gets beta, middle components get proportional temperatures instead of being forced into beta. Preserves explore/exploit contrast while giving intermediate components appropriate exploration levels. Key constraint from diagnostic analysis: the explore/exploit contrast itself drives gains (2-comp libraries with both GMIC < 1.0 still benefit from single-component heating), so any multi-component scheme must maintain at least one fully-cooled component. Diagnostic data shows 2+ components below GMIC 1.0 is rare on 3-comp (3% of query-phases), so the practical impact may be limited to specific SAR landscapes.
-- **GMIC noise denominator inflated by unexplored reagents (Paper 3)**: GMIC's `mean(σᵢ²)` is dominated by the ~85-90% of reagents that remain at warmup-level observation counts after directed search. Noise is 2-12× higher than expected with uniform observations (measured across 3-comp ROCS libraries). This makes GMIC a conservative, blunted signal that underestimates the true SNR of the exploited reagent subpopulation. Conservatism is why GMIC self-corrects and never underperforms round-robin, but recovery is likely lower than achievable with a sharper metric. **Proposed solutions**: (1) observation-weighted GMIC variant (weight σᵢ² by observation count or filter to reagents above a minimum obs threshold), (2) diagnostic plot showing unique reagent usage fraction per component per iteration (reveals how quickly the search narrows and whether narrowing differs across components — first-pass assessment of whether observation-weighted GMIC would differ), (3) filtered GMIC computed on top-K reagents by observation count. **Manuscript context**: discussion point for Paper 1 (TACTICS outperforms legacy even with conservative estimates; recovering 2-3 additional compounds per 100 is meaningful), primary investigation for Paper 3 (does sharper GMIC translate to better recovery, or does conservatism serve as beneficial regularization?).
+- **GMIC noise denominator inflated by unexplored reagents (Paper 3)**: GMIC's `mean(σᵢ²)` is dominated by the ~85-90% of reagents that remain at warmup-level observation counts after directed search. Noise is 2-12× higher than expected with uniform observations (measured across 3-comp ROCS libraries). This makes GMIC a conservative, blunted signal that underestimates the true SNR of the exploited reagent subpopulation. Conservatism is why GMIC self-corrects and never underperforms round-robin, but recovery is likely lower than achievable with a sharper metric. **Manuscript context**: discussion point for Paper 1 (TACTICS outperforms legacy even with conservative estimates; recovering 2-3 additional compounds per 100 is meaningful); proposed solutions (observation-weighted GMIC variant, filtered GMIC, reagent usage diagnostics) scoped to Paper 1b (TACTICS v2) — see Future Manuscript Pipeline.
+- **Simultaneous multi-component exploration (future method improvement / candidate spin-off)**: All current methods (Legacy RWS, TACTICS RWS, TACTICS TT-TS) heat exactly ONE component per cycle (`is_heated = component_idx == self.current_component_idx` in `roulette_wheel.py:261` and `top_two_selection.py:131`), so a single combination never explores two components at once. This is a SHARED limitation — NOT something TACTICS improves over Legacy RWS (GMIC only redirects the single heated slot; it never heats more than one). A true multi-component scheme (≥2 components heated simultaneously, retaining ≥1 fully-cooled component to preserve the explore/exploit contrast) could help hidden-hub libraries whose hits require the conjunction of two mean-hidden reagents (where one-at-a-time heating must discover them sequentially). Closely tied to the continuous GMIC temperature-gradient question above. Candidate for a `/spinout` investigation or a Paper 1b (TACTICS v2) method refinement. (Surfaced 2026-06-04 while correcting the Established Result that wrongly claimed TACTICS does coordinated multi-component exploration.)
## Future Manuscript Pipeline
@@ -136,11 +208,21 @@
Each entry: paper number, title concept, core idea, dependencies.
-->
-- **Paper 1 — Docking benchmark complete**: 4 docking libraries (thrombin 500K, adenine 6M, amide 10M, quinazoline 20M) targeting 3 protein families (protease, GPCR, kinase). JNK3 receptor = PDB 2ZDT (same as Klarich 2024). Full benchmark run (200 trials for new libraries, 20 replicates, 5 methods). Post-hoc analysis complete with rigorous statistical framework. Next: manuscript figures via marimo notebook, then manuscript writing.
-- **Paper 2a — Reagent Expansion via Vendor Mapping**: TACTICS screen on 10-20M member subset → enriched reagents from posteriors → MCS-based vendor mapping against commercial catalogues (Enamine) → focused library construction → re-screen → experimental validation. Short paper. Requires experimental collaborator for assay validation on multiple targets. Worktree: `reagent-expansion-DrugEx`.
-- **Paper 2b — DrugEx Integration for Novel Reagent Generation**: Modify DrugEx v3 graph transformer for fragment-constrained generation (reactive group fixed, scaffold varied). Three-tier architecture: (1) vendor mapping retrieval, (2) constrained fragment generation with MCS synthesizability filter, (3) unconstrained generation. GMIC-based stopping criterion from TACTICS posteriors signals when chemical space expansion saturates. Collaboration with Leiden University (Remco van den Broek, Willem Jespers — DrugEx expertise). Depends on Paper 2a infrastructure.
-- **Paper 3 — Library Diagnostics and Design**: Prescriptive diagnostics that help users design libraries amenable to TACTICS screening. Candidate axes: Free-Wilson additivity, gap hardness, effective reagent rank, NK ruggedness, GMIC profiles. Extends the diagnostic framework from Paper 1. Spun out to `library-diagnostics` worktree (2026-03-17).
-- **Paper 4 — Multi-Objective TACTICS**: MOTS/TSEMO extension for multi-objective TS over combinatorial libraries. Requires defining the right surrogate for Pareto-optimal reagent selection. Furthest out — depends on Papers 1-3 infrastructure.
+**Heptabase organization (formalized 2026-05-18):**
+- Umbrella tag: `TACTICS-Papers` (`be5a2724`)
+- Per-paper tags: `TACTICS-Manuscript` (Paper 1, existing, 37+ cards); `TACTICS-Paper-2-Generative` (`26559a33`); `TACTICS-Paper-3-Cofolding` (`2c994162`); `TACTICS-Paper-4-Pathway` (`9b8548d6`).
+
+- **Paper 1 — TACTICS algorithm + 25-library benchmark** (in preparation): the current manuscript. Two algorithmic enhancements over Klarich 2024 / Zhao 2025 (GMIC-weighted criticality rotation in RWS, σ-based best-arm identification in TT-TS); best-TACTICS beats best-Legacy on 24/25 libraries at top-100. Manuscript repo at `~/Desktop/TACTICS-manuscript/` (moved off SSHFS mount 2026-05-17). Methods §Statistical analysis + Discussion §Sensitivity analysis + Discussion §Single-pose docking and ABFE/FEP integration future-work paragraph written; Results + remaining Discussion sections still TODO.
+- **Paper 2 — TACTICS + Generative Chemistry (DrugEx integration)** (concept formalized 2026-05-18): Closes the loop between TACTICS combinatorial-library screening and generative chemistry. TACTICS posteriors drive a three-tier reagent-expansion engine — Tier 1 vendor-mapping retrieval against Enamine REAL (subsumes former Paper 2a), Tier 2 fragment-constrained DrugEx v3 generation with reactive-group fixity + MCS synthesizability filter (subsumes former Paper 2b), Tier 3 unconstrained generation for slots whose GMIC remains low. GMIC-based saturation criterion (ΔGMIC < ε) provides principled stopping. Prospective experimental validation with Leiden Centre for Drug Research (van den Broek, Jespers). Worktree: `reagent-expansion-DrugEx`. Cards: `ed43a2f4` (OBJECTIVE), `2ba19f5c` (RESEARCH GAP). Closest precedents from lit-scan: SALSA (Exscientia ICLR 2025) does synthon-factorized AL without generative reagent expansion; REINVENT + ESMACS AL (Bhati JCTC 2024) is generative + AL but uses expensive ABFE not combinatorial TS; Jespers 2025 Nat Commun (the Leiden A2A nanomolar result) sets the experimental-validation bar.
+- **Paper 3 — TACTICS as Cofolding-Ensemble Structure-Quality Validator for FEP Triage** (concept formalized 2026-05-18): Inverts the standard VS axis — fixed library, variable receptor structures from ML cofolded ensembles (AF3, Boltz-1, Chai-1, RoseTTAFold-AA, ESMFold). Uses TACTICS recovery-curve shape diagnostics (bimodality, GMIC trajectory smoothness, σ-asymmetry, replicate variance) to rank structures for downstream ABFE/FEP triage. Outperforms cofolding built-in metrics (pLDDT, ipTM, min-iPAE, ipSAE) on FEP reproducibility. Closest prior art: Mizukoshi 2024 Sci Rep (AF2-MSM kinase ensemble ranked by enrichment factor — same inversion, but AF2-MSM not ML cofolding, DUD-E libraries not combinatorial, no AL, enrichment not recovery-shape) and Lupala 2023 J Phys Chem B (AF + FEP recapitulation on congeneric series — uses expensive FEP itself as validator). Four-piece novel combination: ML cofolding × combinatorial library × recovery-shape diagnostics × TS primitive. Lands now because Skreta 2025 + Nat Commun 2025 + Mac1 eLife reviewed preprint 110475 destroyed the assumption that pLDDT/ipTM are sufficient, and Boltz-ABFE JCTC 2025 made cofolding→FEP a real pipeline demanding triage. Cards: `af23611c` (OBJECTIVE), `8f92a732` (RESEARCH GAP).
+- **Paper 4 — TACTICS + Pathway-Aware Multi-Target Validation for Cell-Line Selection** (concept formalized 2026-05-18): Extends Paper 3 from single-target structure-quality triage to multi-target pathway-aware assay-design optimization. For each protein in a signaling pathway (BMP, MAPK/ERK, JAK-STAT, EGFR), generate cofolded ensembles and run TACTICS; combine with systems-biology priors (CCLE/DepMap expression, KEGG/Reactome topology, ChEMBL/DGIdb off-target maps) to rank (cell line × target × structure) tuples by expected assay informativeness. Evaluation-side dual to Zaballa & Hui 2024 MLSB NeurIPS (arXiv 2410.10108), which operates on the generation side on the same BMP pathway. Pilot: BMP (matches Zaballa & Hui for back-to-back comparison), MAPK/ERK (tests scaling). Validation: held-out PRISM cell-line panels; one wet-lab BMP-node test before submission. Cards: `1919aea8` (OBJECTIVE), `519eb72f` (RESEARCH GAP).
+
+### Deprecated / renumbered (historical, superseded by the four-paper hierarchy above)
+
+- "Paper 1b — TACTICS v2 Method Refinements" — reclassified 2026-05-18 as TACTICS-engine improvements deployable across Papers 2/3/4, not a standalone paper.
+- "Paper 2a / 2b" — consolidated 2026-05-18 into the unified Paper 2 (Tiers 1 + 2/3 of the three-tier hierarchy).
+- "Paper 3 (old) — Library Diagnostics and Design" — pre-existing line of work tracked in `library-diagnostics` worktree; not the new Paper 3 (cofolding validator). Could be renumbered Paper 5+ or merged into a separate diagnostics-paper series.
+- "Paper 4 (old) — Multi-Objective TACTICS" — stays on the longer-term roadmap; renumbered (likely Paper 5+).
## Key References
@@ -174,7 +256,29 @@
- `outputs/ttts_validation/` — TT-TS validation benchmark results (pending)
- `examples/run_benchmark.py` — Unified benchmark runner (run/estimate/merge). Replaces all specialized benchmark scripts. Use `/benchmark` skill to generate Slurm jobs.
- `examples/_benchmark_config.py` — Shared benchmark config: method definitions, library discovery, resource estimation (historical CPU-sec per library from manifest data)
-- `tutorials/diagnostic_benchmark_plots.py` — Marimo notebook for diagnostic benchmark visualization (RWS GMIC + temperature, TT-TS disagreement + heated_scale, min_GMIC scatter). 21 figures with 95% CI from 5 replicates.
+- `tutorials/diagnostic_benchmark_plots.py` — Marimo notebook for diagnostic visualization (REORGANIZED 2026-06-05, 19 cells). Sections: RWS GMIC+temperature, TT-TS disagreement+heated_scale, Action Panel, MECH-4 (per-query GMIC-direction heatmaps), and **Layers 1 & 2 dropdown sections for BOTH docking (8 libs) and ROCS (all 20 libs)** via `plot_gmic_directed_exploration` + `plot_adaptive_intensity`. Removed (no longer viable as standalone outperformance explanations): min_GMIC-vs-advantage scatter, GMIC-vs-Oracle, MECH-2, MECH-5. ROCS per-library representative queries are selected by `experiments/_debug/pick_rocs_demo_queries.py`; 2% diagnostics source = `full_benchmark_nogate`.
- Heptabase: "GMIC-Weighted Rotation and CATS Multiplier — Detailed Mechanism" — step-by-step mechanism walkthrough (companion to RWS end-to-end card)
- Heptabase: "Diagnostic Benchmark — Per-Query GMIC Analysis Confirms Adaptive Mechanism" — per-query analysis, min_GMIC predictor, 3-comp SAR difficulty explanation
+- Heptabase: `[RESEARCH GAP] Klarich 2024 — Open Questions from the Original TS Paper` (`402f94ee-cd57-41ef-977d-9e7d8c0fa27b`) — manuscript Introduction source: Klarich-explicit + implicit limitations
+- Heptabase: `[RESEARCH GAP] Zhao 2025 — Open Questions and Extension Opportunities` (`3cc58e43-6710-4902-9b85-792202da9615`) — manuscript Introduction source: Zhao within-scope follow-up questions
+- Heptabase: `[RESEARCH GAP] Systematic Gaps in the Combinatorial-TS Literature` (`e8723de0-72c5-4607-89c3-cff89d1fdb29`) — manuscript Introduction source: cross-paper gaps TACTICS extends
+- Heptabase: `[OBJECTIVE] TACTICS Manuscript — Aims and Contribution Statement` (`9471bb1b-b04c-4f2c-bcc8-42f53c294e81`) — manuscript framing: 4-line contribution statement (method advances, benchmark, mechanistic characterization, open resources) + TT-TS lineage as TACTICS-conceived σ-primitive enhancement parallel to Zhao's RWS extension of Klarich's TS
+- Heptabase: `[METHOD] Docking Reagent-Score Landscape — Three Greedy-Failure Modes for Method Selection` (`66c29afe-fe6b-470e-8596-8b4e767bb231`) — the current docking method-selection framework (supersedes the retired two-fraction and three-regime frameworks). Q1 (does TACTICS beat Legacy) = any of three greedy-failure modes: coverage (greedy can't SELECT the mean-blind hub), allocation (SELECTS but mis-PAIRS partner-dependent dispersed hits — the amide→amide_hybrid σ-control), imbalance (can't ESTIMATE true tops from sparse obs). Q2 (TT-TS vs RWS) = hidden-hub σ-character. Descriptive/post-hoc (n=8), not a classifier; run both, take max. Greedy-reach calibration: `experiments/_debug/greedy_reach_calibration.py`.
+- `data/scores/adenine_hybrid.parquet` — HYBRID (OpenEye) docking scores against the Lenselink-prepared A2A receptor for the adenine library; 5,203,413 unique products across 28 valid amidines (amidines 022, 029 absent due to SMARTS-coverage gap, same as FRED parquet). Score range -19.46 to 24.52, mean -8.16. Top hits dominated by aldehyde_137 (70/100) and aldehyde_430 (20/100).
+- `outputs/adenine_rerun_amidine_035/` — FRED post-merge benchmark results (5 methods × 20 reps × 1 query, 100 trials, 0 errors). TACTICS RWS = 94.5 ± 2.1 winner; TT-TS = 87.3 ± 16.3 (3 bimodal failures via amidine_003).
+- `outputs/adenine_hybrid_benchmark/` — HYBRID benchmark results (5 methods × 20 reps × 1 query, 100 trials, 0 errors). TT-TS = 94.5 ± 5.1 winner (no bimodal failures); TACTICS RWS = 86.4 ± 14.3 (1 catastrophic + 3 partial bimodal failures via aldehyde_137 GMIC under-rotation).
+- `jobs/adenine_hybrid_benchmark.sub` — Slurm submission script for the HYBRID benchmark (8h walltime, 48 cores, 96G memory; runtime ~7 min — over-provisioned, can be reduced).
+- `data/docking/adenine/submit_missing_hybrid.sh` — One-shot submission of missing hybrid batches; not needed for the canonical campaign (the 37 missing batches are pure amidine_022/029, structurally invalid and not recoverable).
+- `outputs/full_benchmark_ema_cats/analysis/recovery_summary.parquet` — **Canonical 25-library recovery summary** (2,185,000 rows; integrated 2026-05-03 with post-merge adenine + adenine_hybrid). Drives all per-library bar plots and Top-N sensitivity plots in `tutorials/manuscript_plots.py`.
+- `outputs/full_benchmark_ema_cats/analysis/{anova,tukey_hsd,friedman,nemenyi}_results.parquet` — Statistical test results across 25 libraries × 10 top-N thresholds.
+- `outputs/full_benchmark_ema_cats/search/_adenine_pre_merge/` — Archived pre-merge adenine search shards (29 amidines, 4.9M-row ground truth). Available if needed for sensitivity analyses; not part of the canonical benchmark.
+- `outputs/adenine_fred_top100.csv`, `outputs/adenine_hybrid_top100.csv` — Top-100 SMILES with docking scores, ranked. For chemistry inspection of cross-docking-pipeline differences (zero product-code overlap; FRED prefers Boc-piperazine scaffolds, HYBRID prefers boronate-containing chemotypes).
+- Heptabase: `[RESULT] Integrated 25-Library Benchmark — TACTICS Beats Legacy on 24/25 Libraries` (`7c446b85-4905-4162-8f52-9456a45310f8`) — canonical headline result for the manuscript primary claim. Per-library Δ table, aggregate ranking, statistical framework summary, cross-docking-pipeline framework validation.
+- **Heptabase tag `TACTICS-Manuscript`** (37 cards as of 2026-05-05) — every latest result/method/discussion/research-gap card relevant to the Paper 1 manuscript, with a standardized `## TL;DR` lead in every card for whiteboard readability. Initial triage (2026-05-05 morning) tagged 31 cards and trashed 20 outdated/duplicate cards (older benchmark snapshots superseded by `7c446b85`; older mechanism walkthroughs superseded by canonical [METHOD] cards; abandoned design proposals). Continuation (2026-05-05 afternoon) added 6 cards: 4 Methods 2.1 bridge cards (`7e34aa1e`, `0bc83dbd`, `27cbde69`, `a9f61a61`) + 2 cards from splitting the original combined Benchmark Methods card (`84ae91a7`, `cdfa2e00`). Use `heptabase tag cards $(heptabase tag list --name-filter TACTICS-Manuscript --json | jq -r '.tags[0].id')` or browse via the desktop app to retrieve the full set.
+- Heptabase: `[DISCUSSION] TACTICS as Efficient Learning — Within-Component Exploitation Enables Across-Component Reallocation` (`cfc20ceb-f4dd-4823-a916-dd9348b56bdb`) — manuscript Discussion mechanism card. HYBRID adenine narrow-top-N case study: Legacy RWS top-25 = 24.8 vs TT-TS 23.9 because TACTICS' GMIC-driven reallocation of exploration *out* of resolved aldehyde component misses two tail-aldehyde rescues (ald111, ald121). Frames TACTICS as data-driven exploration reallocation; the narrow-top-N cost is the expected boundary of the same mechanism that produces wider-top-N gains.
+- Heptabase: `[DISCUSSION] ROCS Top-N Decay — Why Recovery Erodes With Wider Top-N, and Why TACTICS Erodes Less` (`bf67bcd4-6136-45dd-a75f-2ca409da7e0d`) — manuscript Discussion mechanism card. Quantifies ROCS top-N decay across 20 libraries: Legacy RWS −6.49 pp from top-25 to top-250 vs TT-TS −3.63 pp (1.7× slower decay). On 2-comp ROCS Legacy RWS decays −9.07 pp vs TT-TS −5.25 pp. TT-TS advantage over Legacy RWS grows monotonically with top-N (+2.2 → +5.1 pp). Explicitly clarifies method-naming: "Legacy Enhanced-RWS" = Zhao 2025 RWS; "TACTICS RWS" = Zhao + GMIC + CATS. Extends Zhao 2025 Figure 5 (top-100 diversity sensitivity) to within-library top-N expansion. Companion plot in `tutorials/manuscript_plots.py` (cell after `overall_recovery_vs_topn.pdf`): `zhao_fig5_extended_tactics.pdf`.
+- `~/Desktop/TACTICS-manuscript/code/` (branch `code-subdir`) — Paper 1 reproducibility-code subdir co-located with the Quarto manuscript. `config.py` is the single source of truth for the 28-library set, MINIMIZE flag, `SCORE_TO_REAGENT_FOLDER` map, and the `name_key` resolution rule (Local Norm #11). Runners: `run_benchmark.py` (single-node multiprocess, `--workers N`, `--skip-existing`, `--budget-fraction`), `run_posthoc.py` (recovery + ANOVA/Tukey/Welch/Friedman/Nemenyi), `run_union_recovery.py` (half-budget union analysis), `make_figures.py` (14 named figures regenerated from `recovery_summary.parquet`), `_worker.py` (TS execution worker copied from engine repo). `tests/test_library_resolution.py` — 38 tests catching lib_id/folder_name drift before it ships. Pinned dep: `chem-tactics==1.1.0`. Six logical commits on `code-subdir` branch, ready to merge to `main` in TACTICS-manuscript when the user is ready.
+- `~/Desktop/TACTICS-manuscript/data/` — bundled reagents (5.8 MB across 24 reagent subdirs + `reactions.csv`) + `download_scores.py` (Zenodo fetcher for the ~3.2 GB score parquets; ZENODO_DOI is placeholder until deposit exists; `--source file:///...` works for local mirrors during development).
+- `tutorials/manuscript_plots_ROCS.py` — ROCS-only manuscript notebook (1934 lines, 31 cells; created 2026-05-28 by splitting `manuscript_plots.py`). Self-contained: imports, UI controls, data load, scoring-type rule, ROCS Sections 1 (mean recovery + Δ bars + chained-decomposition stack), 2 (Zhao Fig 5 vector + Δ plots, 8 cells), 3 (Tukey HSD CIs + per-library ROCS bar+line), 4 (ROCS budget-sensitivity 2×2 panel). Pairs cleanly with explanation cards R1–R6 + D1–D2 on the manuscript whiteboard.
+- Heptabase: **ROCS Results + Discussion card set (10 cards, 2026-05-28)** — complete manuscript scaffold for the ROCS section of Paper 1. `[METHOD] Statistical Significance vs Practical Significance — Why High-Power Tests Detect Small Effects, and How TACTICS Reports Both` (`ee819848-a37e-4313-aea0-4505a80ff9f3`) — reviewer-defensible framework on high-power statistical tests with ready-to-paste Methods + Discussion paragraphs and four anticipated-reviewer-concern rebuttals. Result cards: `[RESULT] R1` ROCS Budget Sensitivity (`d35ab914-de11-4597-8657-b7d460d03390`), `[RESULT] R2` ROCS Mean Recovery at 2% Budget (`ac504c68-6363-4965-bce5-ed0787a3aacc`), `[RESULT] R3` ROCS Per-Library Δ (`8b668f51-30d3-42f2-be08-f5a67157d04a`), `[RESULT] R4` Chained Decomposition (`05cf848e-f329-4ed3-baa1-31bcccb006a8` — Zhao 33% / TACTICS 67% at top-100 redistributing to 3% / 97% at top-250; 2-comp Zhao negative from top-100), `[RESULT] R5a` Diversity-Recovery Vector (`fcbae16c-d625-4a04-a197-7e7ddc065918` — universality claim), `[RESULT] R5b` Diversity-Recovery Δ-Slope (`75762f09-b705-462d-9143-18cd6e131134` — non-erosion claim), `[RESULT] R6` ROCS Pairwise Tukey HSD Completeness (`a306fcfc-69af-400f-9736-b016acfb85c6`). Discussion cards: `[DISCUSSION] D1` TACTICS Addresses Zhao 2025's Open Question (`1e5691da-b63f-4aec-9a3f-4742c557b63f` — centerpiece synthesizing R4 + R5a + R5b), `[DISCUSSION] D2` Practical vs Statistical Significance on ROCS Findings (`d7996148-96ac-49de-8542-7de8a62a16a3`).
diff --git a/research/blueprints/seh_caix_smarts.md b/research/blueprints/seh_caix_smarts.md
new file mode 100644
index 0000000..2ad9571
--- /dev/null
+++ b/research/blueprints/seh_caix_smarts.md
@@ -0,0 +1,250 @@
+# SMARTS discovery for sEH and CAIX DEL benchmark libraries
+
+Empirically derived reaction SMARTS for two DEL combinatorial libraries that have reagents
+but no SMARTS entry in `data/reactions.csv`. Validated against ground-truth products where
+available (sEH). All scratch artifacts in `outputs/debug/seh_smarts/` (gitignored).
+
+Method: read `smarts_toolkit/{config,_validator}.py` + `synthesis_pipeline.py` for the real
+API, censused functional groups in every BB set, deduced bond-forming chemistry by diffing
+reagents against product substructure, then validated by reproducing known canonical
+products (stereo-stripped on BOTH sides) and by per-position coverage.
+
+Sampling caveat (important): `total_compounds.csv` is sorted, so any head slice contains a
+tiny number of distinct bb3. Always sample with `.gather_every(N)` over a large `n_rows`,
+never a head slice.
+
+---
+
+## sEH — VERDICT: enumerable in TACTICS = YES
+
+**97.4% of ground-truth products reproduced exactly** with an alternative-SMARTS amide +
+N-methyl-amide + methylamine-cap scheme (3898/4000, broad sample, stereo-insensitive;
+`outputs/debug/seh_smarts/unified.json`). The 2.6% residual is almost entirely
+amino-acid-bb1 cyclization/lactam edge cases (`(bb1=both, bb2=acid, bb3=amine)` = 91 of the
+102 misses); the 270 non-amine/non-acid bb3 (sulfonyl chlorides, isocyanates, aryl/heteroaryl
+halides) are a separate, real chemistry gap that an amide-only library simply excludes.
+
+### Library chemistry (deduced + ground-truth-confirmed)
+
+sEH is a **3-cycle amide-coupling DEL**: a central di-acid scaffold (bb2) bearing two
+building blocks (bb1, bb3) as amides, with leftover acids capped by methylamine.
+
+Full distinct-BB functional-group census, streamed over all 4,433,801 rows
+(`outputs/debug/seh_smarts/real_bb_census.txt`):
+
+- **bb2 (338) = central scaffold, ALL carboxylic acid**: 124 acid-only, 214 amino-acid, **110 di-carboxylic acids**. The di-acids form the bis-amide core.
+- **bb1 (617) = ALL have a primary/secondary amine**: 291 are amino acids (amine + acid), 326 amine-only, 0 acid-only. bb1 always contributes its amine.
+- **bb3 (4529)**: 2363 amine, 1897 acid, 1 both, **270 "neither"**. The 270 "neither" are **sulfonyl chlorides, isocyanates, and aryl/heteroaryl halides** (163 carry a C–halide; the rest are mostly ArSO2Cl / R–N=C=O), confirmed in `outputs/debug/seh_smarts/bb3_neither.json` — NOT Boc-protected amines. These react by sulfonamide formation / urea / SNAr, i.e. chemistry outside the amide scheme, and are the main coverage gap.
+
+Bonds and the N-methyl: bb1's amine and bb3's amine each form an amide to a bb2 acid (when
+bb3 is itself an acid, its acid couples to a free amine instead). The N-methyl seen in ~73%
+of products has **two distinct origins**, both required to reach 95.8%:
+1. **N-methylation of a coupling amide nitrogen** → tertiary `CN(R)C(=O)` (the common case when bb1/bb3 are simple amines).
+2. **Methylamine capping of a leftover free carboxylic acid** → `C(=O)NHCH3` (e.g. an amino-acid bb1's own acid, or an unconsumed bb2 acid).
+
+A unified assembler with amide + N-methyl-amide + methylamine-cap (both coupling polarities,
+0–2 caps) reproduces **97.4% exactly** (3898/4000). Failure taxonomy of the 2.6% residual
+(`outputs/debug/seh_smarts/unified.json`): 91 `(bb1=both, bb2=acid, bb3=amine)` — amino-acid
+bb1 cyclization/lactam edge cases — and 11 `(bb1=amine, bb2=acid, bb3=amine)` di-acid stereo
+edge cases. The 270 non-amine/non-acid bb3 (ArSO2Cl / R-N=C=O / Ar-halide) are excluded by an
+amide-only scheme and are NOT counted here (they appear as `bb3=neither`, a separate gap).
+
+### Best reaction SMARTS
+
+Amine filter used throughout (excludes amides, hydrazines, N–heteroatom amines):
+`#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O)`
+
+| role | SMARTS |
+|------|--------|
+| amide coupling | `[CX3:1](=[OX1])[OX2H1].[#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O):2]>>[CX3:1](=O)[#7:2]` |
+| N-methyl amide coupling | `[CX3:1](=[OX1])[OX2H1].[#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O):2]>>[CX3:1](=O)[#7:2][CH3]` |
+| methylamine cap of free acid | `[CX3:1](=[OX1])[OX2H1]>>[CX3:1](=O)NC` |
+
+(All four SMARTS in this report — these three plus the CAIX reductive-amination and Fmoc —
+parse cleanly under `AllChem.ReactionFromSmarts`; verified.)
+
+### Per-position coverage (ValidationResult.coverage_stats), single amide config
+
+- bb2 acid template: **100.0%** (all 338 scaffolds have a couplable acid)
+- bb1 amine template: **100.0%** (all 617 bb1 have an amine)
+- bb3 amine template: **55.0%** — EXPECTED, not a defect: only 2363/4529 bb3 are amines; 1897 are acids (couple via the acid template) and 270 are sulfonyl-Cl/halide (need different chemistry).
+
+Full bb3 coverage needs **alternative SMARTS** per step (amine-coupling OR acid-coupling),
+via `step_modes={step: "alternative"}`. The 270 non-amine/non-acid bb3 are out of scope for
+an amide-only library (drop them, or add sulfonamide/urea/SNAr patterns to include them).
+
+### Valid vs dropped reagents/products
+
+- Valid reagents: bb2 100% (338/338, all acid); bb1 100% (617/617, all amine); bb3 ~94% directly couplable (2363 amine + 1897 acid); 270 (~6%) bb3 are non-amide chemistry (sulfonyl-Cl / halide). 0 invalid SMILES in any list.
+- Products reproduced exactly: 97.4% with the unified amide/N-methyl-amide/cap scheme; the 2.6% miss is amino-acid-bb1 lactam/cyclization + di-acid stereo edge cases. (The 270 non-amide bb3 are a separate, deliberately-excluded class, not part of this 2.6%.)
+- A naive SINGLE fixed-polarity 3-step config (`run_single_reaction` emits only `products[0][0]`) reproduces ~62% exact while still producing a *valid* member for ~95% of combinations (`outputs/debug/seh_smarts/{modelB_clean,definitive}.json`); the gap is di-acid regiochemistry that the alternative-SMARTS config closes.
+
+### Copy-pasteable ReactionConfig (practical 3-step; ~95% valid products)
+
+```python
+from TACTICS.library_enumeration import SynthesisPipeline
+from TACTICS.library_enumeration.smarts_toolkit import (
+ ReactionConfig, ReactionDef, StepInput, InputSource,
+)
+
+AMINE = "#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O)"
+AMIDE = f"[CX3:1](=[OX1])[OX2H1].[{AMINE}:2]>>[CX3:1](=O)[#7:2]"
+CAP = "[CX3:1](=[OX1])[OX2H1]>>[CX3:1](=O)NC" # methylamine cap
+
+# Reagent order: [bb2 scaffold, bb1, bb3]
+config = ReactionConfig(
+ reactions=[
+ ReactionDef(reaction_smarts=AMIDE, step_index=0), # bb2 acid + bb1 amine (acid-template-first; inputs are [bb2 acid, bb1 amine] — OK)
+ # step 1 inputs are [PREVIOUS_STEP product (carries the free acid), bb3 amine] -> acid-first matches:
+ ReactionDef(reaction_smarts=AMIDE, step_index=1), # remaining acid + bb3 amine
+ ReactionDef(reaction_smarts=CAP, step_index=2), # cap any leftover free acid
+ ],
+ reagent_file_list=[
+ "data/reagents/seh/DEL_seH/bb2_list.csv",
+ "data/reagents/seh/DEL_seH/bb1_list.csv",
+ "data/reagents/seh/DEL_seH/bb3_list.csv",
+ ],
+ step_inputs={
+ 0: [StepInput(source=InputSource.REAGENT_FILE, file_index=0),
+ StepInput(source=InputSource.REAGENT_FILE, file_index=1)],
+ 1: [StepInput(source=InputSource.PREVIOUS_STEP, step_index=0),
+ StepInput(source=InputSource.REAGENT_FILE, file_index=2)],
+ 2: [StepInput(source=InputSource.PREVIOUS_STEP, step_index=1)],
+ },
+)
+pipeline = SynthesisPipeline(config)
+```
+
+To approach the 97.4% exact ceiling, add alternative SMARTS at steps 0/1 so each step tries
+amide AND N-methyl-amide in both polarities (`step_modes={0:"alternative", 1:"alternative"}`).
+The N-methyl-amide pattern is the same AMIDE with `>>[CX3:1](=O)[#7:2][CH3]`.
+
+### `reactions.csv` entry
+
+The library is multi-step + cap, so it does not collapse to one row. If a single
+representative SMARTS is required for the lookup table, use the amide:
+```
+seh,"[CX3:1](=[OX1])[OX2H1].[#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O):2]>>[CX3:1](=O)[#7:2]"
+```
+Otherwise register the multi-step config above directly in the benchmark runner.
+
+### Reagent-file note
+
+Use **`bb1_list.csv` / `bb2_list.csv` / `bb3_list.csv`** (column `SMILES`, auto-detected by
+the validator). Their counts (617 / 338 / 4529) and full functional-group census match the
+distinct `bb1/bb2/bb3` columns of `total_compounds.csv` exactly — they are the true reagent
+sets.
+
+### Expected enumerated product count
+
+Full combinatorial size = 617 × 338 × 4529 ≈ **9.44e8** (~944M). Ground-truth
+`total_compounds.csv` has 4,433,801 measured members. Enumerate on demand — Thompson
+Sampling never materializes the full space.
+
+---
+
+## CAIX — VERDICT: enumerable in TACTICS = YES (2-step enum 100% = 400/400 verified; Fmoc removal post-enumeration)
+
+No ground-truth product file, so confidence rests on coverage + chemical reasoning, but the
+BB roles are unambiguous from the functional-group census.
+
+### Library chemistry (deduced)
+
+3-component reductive amination + amide, with Fmoc removal:
+
+- **AL (67) = aromatic aldehydes** (100% have `[CX3H1]=O`; no acids/amines)
+- **AM (52) = primary amines** (100% primary NH2)
+- **AA (62) = Fmoc-protected amino acids** (100% carboxylic acid, 100% Fmoc carbamate)
+
+Sequence: (1) reductive amination AL + AM → secondary amine; (2) amide coupling of AA's acid
+onto that secondary amine; (3) Fmoc removal to expose the warhead amine (the CAIX
+zinc-binding handle).
+
+### Best SMARTS
+
+| role | SMARTS |
+|------|--------|
+| reductive amination (aldehyde + amine → CH2–N) | `[#6:4][CX3H1:1]=[OX1].[#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O):3]>>[#6:4][CH2:1][#7:3]` |
+| amide coupling (acid + amine → amide) | `[CX3:1](=[OX1])[OX2H1].[#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O):2]>>[CX3:1](=O)[#7:2]` |
+| Fmoc removal (carbamate → free amine) | `[N:1]C(=O)OCC1c2ccccc2-c2ccccc21>>[N:1]` |
+
+### Coverage + enumeration
+
+- reductive amination: aldehyde template 100.0% (AL), amine template 100.0% (AM)
+- amide: AA acid template 100.0%
+- **2-step pipeline enum success (reductive amination + amide, no Fmoc step): 100% = 400/400** (`outputs/debug/seh_smarts/caix_fixed.json`)
+- Fmoc removal SMARTS verified standalone on a real product: cleanly converts the carbamate → free amine (`outputs/debug/seh_smarts/{caix_defmoc_test,caix_fixed}.json`).
+
+### CRITICAL config gotcha — reactant-template order MUST match step_inputs order
+
+`enumerate_single` feeds the inputs of a step positionally to the reaction's reactant
+templates (input[0] → template[0], input[1] → template[1]). It does NOT reorder by
+functional group. So if a step's `step_inputs` are `[PREVIOUS_STEP (an amine), REAGENT_FILE
+(an acid)]`, the step-1 SMARTS must be written **amine-template-first**:
+`[{AMINE}:2].[CX3:1](=[OX1])[OX2H1]>>[CX3:1](=O)[#7:2]`. Writing it acid-first silently
+yields zero products (verified: acid-first = 0/300, amine-first = 400/400;
+`outputs/debug/seh_smarts/{caix_debug2,caix_fixed}.json`). The same rule applies to the sEH
+config above — keep each step's template order aligned with its `step_inputs`.
+
+### Fmoc removal — do it as a POST-enumeration step
+
+In-pipeline Fmoc removal via `DeprotectionSpec(group="Fmoc", target="product")` with a custom
+`ProtectingGroupInfo` did not fire in testing. **Working path:** run the 2-step config
+(reductive amination + amide → 100% products), then apply the Fmoc-removal reaction SMARTS
+`[N:1]C(=O)OCC1c2ccccc2-c2ccccc21>>[N:1]` to each product as a one-line post-processing pass
+(verified: deprotects a real product to the free amine, `caix_fixed.json`). If leaving Fmoc
+on is acceptable for benchmark scoring, the post-step is optional. Either way CAIX enumerates
+fully.
+
+### Reagent-file note
+
+CAIX `.smi` files have a 3-column `BB,cas,SMILES` header. The validator's CSV loader
+auto-detects the `SMILES` column, but the **`.smi` loader splits on whitespace and would
+take `BB` as the SMILES**. Either pass them as `.csv` (rename/symlink), or pre-write plain
+1-column `.smi` (as done: `outputs/debug/seh_smarts/caix_{AL,AM,AA}.smi`).
+
+### Expected enumerated product count
+
+67 × 52 × 62 = **216,008** products — small enough to enumerate exhaustively if desired.
+
+### Copy-pasteable ReactionConfig (CAIX, 2-step; apply Fmoc removal post-hoc)
+
+```python
+AMINE = "#7X3;H2,H1;!$(N=*);!$(N-[!#6]);!$(NC=O)"
+config = ReactionConfig(
+ reactions=[
+ ReactionDef(reaction_smarts=f"[#6:4][CX3H1:1]=[OX1].[{AMINE}:3]>>[#6:4][CH2:1][#7:3]", step_index=0),
+ # step-1 amide written AMINE-FIRST to match step_inputs order [PREVIOUS_STEP amine, AA acid]:
+ ReactionDef(reaction_smarts=f"[{AMINE}:2].[CX3:1](=[OX1])[OX2H1]>>[CX3:1](=O)[#7:2]", step_index=1),
+ ],
+ reagent_file_list=[
+ "data/reagents/CAIX/AL_SEL2.smi", # aldehyde (pass as .csv OR clean .smi)
+ "data/reagents/CAIX/AM_SEL2.smi", # amine
+ "data/reagents/CAIX/AA_SEL2.smi", # Fmoc amino acid
+ ],
+ step_inputs={
+ 0: [StepInput(source=InputSource.REAGENT_FILE, file_index=0),
+ StepInput(source=InputSource.REAGENT_FILE, file_index=1)],
+ 1: [StepInput(source=InputSource.PREVIOUS_STEP, step_index=0),
+ StepInput(source=InputSource.REAGENT_FILE, file_index=2)],
+ },
+)
+# Post-enumeration Fmoc removal (per product mol):
+# from rdkit.Chem import AllChem
+# defmoc = AllChem.ReactionFromSmarts("[N:1]C(=O)OCC1c2ccccc2-c2ccccc21>>[N:1]")
+# ps = defmoc.RunReactants((product_mol,)); free = ps[0][0] if ps else product_mol
+```
+
+---
+
+## DEL objective note (read_count vs docking)
+
+Both libraries carry an experimental **DEL read_count** enrichment signal (sEH:
+`total_compounds.csv` `read_count`, plus `del_binders.txt` / `del_nonbinders.csv`; 4.43M
+measured members). For a TACTICS benchmark the natural objective is the **DEL read_count
+itself** — a real, noisy, experimentally-measured target that exercises Thompson Sampling on
+authentic assay signal and is the cheapest LookupEvaluator path. Docking the enumerated
+products is a viable alternative (and lets sEH/CAIX join the existing docking-benchmark set),
+but read_count is the more faithful DEL task and needs no new computation. Recommendation:
+read_count as primary objective; docking as an optional second objective for cross-signal
+studies.
diff --git a/src/TACTICS/library_analysis/diagnostic_plots.py b/src/TACTICS/library_analysis/diagnostic_plots.py
index e3e245c..3d75bea 100644
--- a/src/TACTICS/library_analysis/diagnostic_plots.py
+++ b/src/TACTICS/library_analysis/diagnostic_plots.py
@@ -5,6 +5,7 @@
objects so callers can save, show, or compose them freely.
"""
+from pathlib import Path
from typing import Optional
import numpy as np
import polars as pl
@@ -133,11 +134,11 @@ def plot_temperature_decomposition(
for ax, (col, title) in zip(axes, fields):
vals = comp[col].to_numpy()
ax.plot(cycles, vals, linewidth=1.5)
- ax.set_ylabel(title, fontsize=9)
+ ax.set_ylabel(title, fontsize=13)
ax.grid(alpha=0.3)
axes[-1].set_xlabel("Cycle")
- fig.suptitle(f"Temperature Decomposition — Component {component_idx}", fontsize=12)
+ fig.suptitle(f"Temperature Decomposition — Component {component_idx}", fontsize=15)
fig.tight_layout()
return fig
@@ -278,20 +279,20 @@ def plot_ttts_mechanism_decomposition(
vals = comp[col].to_numpy()
valid = np.isfinite(vals)
ax.plot(cycles[valid], vals[valid], linewidth=1.5)
- ax.set_ylabel(title, fontsize=9)
+ ax.set_ylabel(title, fontsize=13)
ax.grid(alpha=0.3)
# Add reference lines
axes[0].axhline(0.3, color="red", linestyle="--", alpha=0.4)
axes[1].axhline(0.8, color="red", linestyle="--", alpha=0.4, label="Saturated")
axes[1].axhline(0.3, color="green", linestyle="--", alpha=0.4, label="Resolved")
- axes[1].legend(fontsize=7)
+ axes[1].legend(fontsize=12)
axes[2].axhline(1.0, color="gray", linestyle="--", alpha=0.4)
axes[-1].set_xlabel("Cycle")
fig.suptitle(
f"TT-TS Mechanism Decomposition — Component {component_idx}",
- fontsize=12,
+ fontsize=15,
)
fig.tight_layout()
return fig
@@ -333,7 +334,7 @@ def plot_posterior_heatmap(
y_pos = np.arange(len(names))
ax.barh(y_pos, means, xerr=stds, align="center", alpha=0.8, capsize=3)
ax.set_yticks(y_pos)
- ax.set_yticklabels(names, fontsize=7)
+ ax.set_yticklabels(names, fontsize=15)
ax.set_xlabel("Posterior Mean")
ax.set_title(f"Top-{top_n} Reagents — Component {component_idx}")
fig.tight_layout()
@@ -403,7 +404,7 @@ def plot_convergence_comparison(
ax2.set_ylim(-0.05, 1.05)
ax2.legend()
- fig.suptitle("Convergence Comparison: Trajectory vs Snapshot", fontsize=13)
+ fig.suptitle("Convergence Comparison: Trajectory vs Snapshot", fontsize=16)
fig.tight_layout()
return fig
@@ -426,6 +427,28 @@ def _rolling_mean(arr: np.ndarray, window: int) -> np.ndarray:
return np.convolve(padded, kernel, mode="valid")[: len(arr)]
+def _smooth_trim(arr: np.ndarray, window: int) -> np.ndarray:
+ """Centred rolling mean with an edge-shrinking window (no padding).
+
+ Unlike :func:`_rolling_mean` (which edge-pads by repeating the boundary
+ value), this averages only the points that actually exist within the window,
+ shrinking the window at the array ends. This avoids the boundary bias that a
+ repeated heated/cooled extreme would otherwise inject into the last few
+ cycles of an oscillating per-cycle signal. NaNs are ignored.
+ """
+ n = len(arr)
+ if window < 2 or n == 0:
+ return arr.copy()
+ half = window // 2
+ out = np.full(n, np.nan)
+ for i in range(n):
+ seg = arr[max(0, i - half): min(n, i + half + 1)]
+ seg = seg[~np.isnan(seg)]
+ if seg.size:
+ out[i] = seg.mean()
+ return out
+
+
def _ci95(std: np.ndarray, n: int) -> np.ndarray:
"""Convert std to 95 % confidence interval half-width."""
if n <= 1:
@@ -549,7 +572,7 @@ def plot_rws_diagnostic(
facecolor="#ffe0e0", edgecolor="red", linewidth=0.8, alpha=0.7,
label="Divergence gate active",
))
- ax_gmic.legend(handles=_handles, loc="upper left", fontsize=7)
+ ax_gmic.legend(handles=_handles, loc="upper left", fontsize=12)
fig.tight_layout()
return fig
@@ -643,12 +666,12 @@ def plot_ttts_diagnostic(
ax_ema.set_title(title or "TT-TS Diagnostic")
ax_scale.axhline(1.0, color="gray", linestyle=":", alpha=0.4, linewidth=0.8)
- ax_ema.legend(loc="upper right", fontsize=8)
+ ax_ema.legend(loc="upper right", fontsize=13)
ax_ema.annotate("saturated (>0.8)", xy=(0.02, 0.88),
- xycoords="axes fraction", fontsize=7, color="red", alpha=0.5)
+ xycoords="axes fraction", fontsize=12, color="red", alpha=0.5)
ax_ema.annotate("resolved (<0.3)", xy=(0.02, 0.22),
- xycoords="axes fraction", fontsize=7, color="green", alpha=0.5)
+ xycoords="axes fraction", fontsize=12, color="green", alpha=0.5)
fig.tight_layout()
return fig
@@ -691,7 +714,7 @@ def plot_min_gmic_vs_advantage(
ax.annotate(
f"{d['library']}\n{d['query']}",
(d["min_gmic"], d["advantage"]),
- fontsize=6, ha="center", va="bottom",
+ fontsize=15, ha="center", va="bottom",
xytext=(0, 6), textcoords="offset points",
)
@@ -704,7 +727,7 @@ def plot_min_gmic_vs_advantage(
ax.set_xlabel("Minimum Component GMIC (late search)")
ax.set_ylabel("TACTICS Advantage (recovery pts)")
ax.set_title("SAR Difficulty Predicts TACTICS Advantage")
- ax.legend(fontsize=7, loc="upper right")
+ ax.legend(fontsize=12, loc="upper right")
# Marker legend
unique_n = sorted(set(d["n_components"] for d in query_data))
@@ -722,8 +745,1124 @@ def plot_min_gmic_vs_advantage(
))
ax2 = ax.twinx()
ax2.set_yticks([])
- ax2.legend(handles=elems, loc="center right", fontsize=7,
- title="Library type", title_fontsize=8)
+ ax2.legend(handles=elems, loc="center right", fontsize=12,
+ title="Library type", title_fontsize=13)
+
+ fig.tight_layout()
+ return fig
+
+
+# ── Oracle GMIC comparison plots ─────────────────────────────────────
+
+
+def compute_oracle_flexibility(
+ scores_df: pl.DataFrame,
+ query_col: str,
+ reagent_files: list[Path],
+ top_n: int = 100,
+ mode: str = "maximize",
+) -> dict:
+ """Compute ground-truth per-component reagent mean variance and hit
+ concentration for a single query.
+
+ For each component, computes:
+ - ``reagent_mean_var``: Var(per-reagent mean scores). High = clear
+ winners exist (solved/critical). Low = flat landscape (flexible).
+ - ``hit_reagent_fraction``: fraction of the component's reagents that
+ appear in at least one top-*top_n* compound.
+ - ``hit_concentration``: what fraction of top-N hits are accounted for
+ by the top 10% of reagents (by hit participation count).
+
+ The **oracle GMIC** uses the same ``0.5 * ln(1 + signal/noise)``
+ formula as TACTICS, but computed from all products (exhaustive data)
+ rather than the ~2% posterior sample. It represents the best possible
+ GMIC estimate and serves as the reference for validating that GMIC
+ from limited observations converges to the true component ranking.
+
+ Args:
+ scores_df: Full score DataFrame with ``Name`` column encoding
+ underscore-joined reagent IDs and a score column for the query.
+ query_col: Column name for the query scores (e.g. ``"query_008"``
+ or ``"Scores"``).
+ reagent_files: Ordered list of ``.smi`` file paths, one per
+ component. Reagent IDs are read from the second column.
+ top_n: Number of top compounds to define "hits" (default 100).
+ mode: ``"maximize"`` or ``"minimize"``.
+
+ Returns:
+ Dict with keys:
+ - ``per_component``: list of dicts (one per component) each with
+ ``reagent_mean_var``, ``noise_var``, ``oracle_gmic``,
+ ``n_hit_reagents``, ``n_total_reagents``, ``hit_reagent_fraction``,
+ ``hit_concentration``.
+ - ``oracle_flexibility``: array of flexibility weights
+ ``1 / (1 + oracle_gmic)`` per component, normalized to sum to 1.
+ - ``oracle_most_flexible``: index of the most flexible component.
+ """
+ # ── Load reagent ID → index mappings ──────────────────────────
+ reagent_id_to_idx: list[dict[str, int]] = []
+ for rf in reagent_files:
+ mapping = {}
+ with open(rf, encoding="latin-1") as fh:
+ for idx, line in enumerate(fh):
+ parts = line.strip().split()
+ if len(parts) >= 2:
+ mapping[parts[1]] = idx
+ reagent_id_to_idx.append(mapping)
+ n_components = len(reagent_files)
+
+ # ── Parse Name → per-component reagent IDs ────────────────────
+ working = scores_df.select(["Name", query_col]).drop_nulls(subset=[query_col])
+
+ names = working["Name"].to_list()
+ scores_arr = working[query_col].to_numpy()
+
+ # Build per-compound reagent index arrays
+ comp_indices = [[] for _ in range(n_components)]
+ valid_mask = np.ones(len(names), dtype=bool)
+ for row_idx, name in enumerate(names):
+ parts = name.split("_")
+ if len(parts) != n_components:
+ valid_mask[row_idx] = False
+ continue
+ for c in range(n_components):
+ rid = parts[c]
+ comp_indices[c].append(reagent_id_to_idx[c].get(rid, -1))
+
+ scores_valid = scores_arr[valid_mask]
+ for c in range(n_components):
+ comp_indices[c] = np.array(comp_indices[c])
+
+ # ── Identify top-N hits ───────────────────────────────────────
+ if mode == "minimize":
+ top_idx = np.argsort(scores_valid)[:top_n]
+ else:
+ top_idx = np.argsort(scores_valid)[-top_n:][::-1]
+
+ # ── Per-component analysis ────────────────────────────────────
+ per_component = []
+ oracle_gmics = []
+
+ for c in range(n_components):
+ indices = comp_indices[c]
+ n_reagents = len(reagent_id_to_idx[c])
+
+ # Per-reagent mean score (ground truth)
+ reagent_scores: dict[int, list[float]] = {}
+ for i, score in zip(indices, scores_valid):
+ if i < 0:
+ continue
+ reagent_scores.setdefault(i, []).append(score)
+ reagent_means = np.array([
+ np.mean(reagent_scores[r])
+ for r in sorted(reagent_scores.keys())
+ ])
+
+ # Ground-truth signal variance
+ signal_var = float(np.var(reagent_means)) if len(reagent_means) > 1 else 0.0
+
+ # Ground-truth noise: mean of per-reagent score variance
+ reagent_vars = np.array([
+ np.var(reagent_scores[r])
+ for r in sorted(reagent_scores.keys())
+ if len(reagent_scores[r]) > 1
+ ])
+ noise_var = float(np.mean(reagent_vars)) if len(reagent_vars) > 0 else 1e-10
+
+ # Oracle GMIC (same formula, but from exhaustive ground truth)
+ oracle_gmic = 0.5 * np.log1p(signal_var / max(noise_var, 1e-10))
+
+ # Hit reagent analysis
+ hit_reagent_ids = set(indices[top_idx])
+ hit_reagent_ids.discard(-1)
+
+ # Hit concentration: how many top-N hits come from top 10%
+ # of reagents (by hit participation count)
+ hit_counts: dict[int, int] = {}
+ for i in indices[top_idx]:
+ if i >= 0:
+ hit_counts[i] = hit_counts.get(i, 0) + 1
+ if hit_counts:
+ sorted_counts = sorted(hit_counts.values(), reverse=True)
+ top_10pct = max(1, len(sorted_counts) // 10)
+ hit_concentration = sum(sorted_counts[:top_10pct]) / top_n
+ else:
+ hit_concentration = 0.0
+
+ per_component.append({
+ "reagent_mean_var": signal_var,
+ "noise_var": noise_var,
+ "oracle_gmic": float(oracle_gmic),
+ "n_hit_reagents": len(hit_reagent_ids),
+ "n_total_reagents": n_reagents,
+ "hit_reagent_fraction": len(hit_reagent_ids) / max(n_reagents, 1),
+ "hit_concentration": hit_concentration,
+ })
+ oracle_gmics.append(float(oracle_gmic))
+
+ oracle_gmics_arr = np.array(oracle_gmics)
+ oracle_flex = 1.0 / (1.0 + oracle_gmics_arr)
+ oracle_flex_norm = oracle_flex / oracle_flex.sum()
+
+ return {
+ "per_component": per_component,
+ "oracle_flexibility": oracle_flex_norm,
+ "oracle_most_flexible": int(np.argmax(oracle_flex_norm)),
+ }
+
+
+def plot_gmic_vs_oracle(
+ diagnostics_df: pl.DataFrame,
+ oracle: dict,
+ title: str = "",
+ figsize: tuple = (12, 6),
+) -> Figure:
+ """Compare GMIC flexibility ranking against ground-truth oracle.
+
+ **Top panel**: Per-component GMIC trajectories (mean across replicates)
+ with 95% CI, plus dashed horizontal lines showing oracle GMIC values.
+ Demonstrates that GMIC from ~2% sample converges toward the
+ exhaustive ground-truth GMIC ordering.
+
+ **Bottom panel**: Per-cycle heating probability under GMIC rotation
+ (mean across replicates) vs oracle heating probability (horizontal
+ dashed lines) vs round-robin (dotted grey). Shows that GMIC
+ consistently directs more heating to the component the oracle
+ identifies as most flexible.
+
+ Args:
+ diagnostics_df: RWS or TT-TS diagnostics for a single query,
+ all replicates.
+ oracle: Output of :func:`compute_oracle_flexibility`.
+ title: Plot title.
+ figsize: Figure size.
+
+ Returns:
+ Matplotlib Figure.
+ """
+ cycle_col = "current_cycle"
+ comps = sorted(diagnostics_df["component_idx"].unique().to_list())
+ n_comps = len(comps)
+ n_reps = diagnostics_df["replicate"].n_unique()
+ colors = _component_palette(n_comps)
+ cycles = sorted(diagnostics_df[cycle_col].unique().to_list())
+
+ fig, (ax_gmic, ax_prob) = plt.subplots(
+ 2, 1, figsize=figsize, sharex=True, height_ratios=[1, 1]
+ )
+ for ax in (ax_gmic, ax_prob):
+ ax.set_facecolor("white")
+ fig.patch.set_facecolor("white")
+
+ # ── Top panel: GMIC trajectories + oracle reference ───────────
+ for i, comp in enumerate(comps):
+ color = colors[i]
+ comp_data = diagnostics_df.filter(pl.col("component_idx") == comp)
+ agg = (
+ comp_data.group_by(cycle_col)
+ .agg([
+ pl.col("gmic").mean().alias("gmic_mean"),
+ pl.col("gmic").std().alias("gmic_std"),
+ ])
+ .sort(cycle_col)
+ )
+ c = agg[cycle_col].to_numpy()
+ gmic_mean = agg["gmic_mean"].to_numpy()
+ gmic_ci = _ci95(agg["gmic_std"].fill_null(0).to_numpy(), n_reps)
+
+ ax_gmic.plot(c, gmic_mean, color=color, linewidth=1.8,
+ label=f"Comp {comp} (GMIC)")
+ ax_gmic.fill_between(c, gmic_mean - gmic_ci, gmic_mean + gmic_ci,
+ color=color, alpha=0.15)
+ # Oracle GMIC as horizontal dashed line
+ oracle_gmic_val = oracle["per_component"][i]["oracle_gmic"]
+ ax_gmic.axhline(oracle_gmic_val, color=color, linestyle="--",
+ linewidth=1.2, alpha=0.7,
+ label=f"Comp {comp} (oracle = {oracle_gmic_val:.2f})")
+
+ ax_gmic.set_ylabel("GMIC")
+ ax_gmic.set_title(title or "GMIC vs Oracle Ground Truth")
+ ax_gmic.legend(fontsize=12, loc="upper right", ncol=2)
+
+ # ── Bottom panel: heating probabilities ───────────────────────
+ for i, comp in enumerate(comps):
+ color = colors[i]
+ prob_means = []
+ prob_stds = []
+ for cycle in cycles:
+ cycle_data = diagnostics_df.filter(pl.col(cycle_col) == cycle)
+ reps = cycle_data["replicate"].unique().to_list()
+ rep_probs = []
+ for rep in reps:
+ rep_data = cycle_data.filter(pl.col("replicate") == rep)
+ gmics = []
+ for c2 in comps:
+ g = rep_data.filter(
+ pl.col("component_idx") == c2
+ )["gmic"].to_list()
+ gmics.append(g[0] if g else 0.0)
+ gmics_arr = np.array(gmics)
+ flex = 1.0 / (1.0 + gmics_arr)
+ probs = flex / flex.sum()
+ rep_probs.append(probs[i])
+ prob_means.append(np.mean(rep_probs))
+ prob_stds.append(np.std(rep_probs))
+
+ prob_means = np.array(prob_means)
+ prob_ci = _ci95(np.array(prob_stds), n_reps)
+ cycles_arr = np.array(cycles)
+
+ ax_prob.plot(cycles_arr, prob_means, color=color, linewidth=1.8,
+ label=f"Comp {comp} (GMIC)")
+ ax_prob.fill_between(cycles_arr, prob_means - prob_ci,
+ prob_means + prob_ci,
+ color=color, alpha=0.15)
+
+ # Oracle probability as horizontal dashed
+ oracle_prob = oracle["oracle_flexibility"][i]
+ ax_prob.axhline(oracle_prob, color=color, linestyle="--",
+ linewidth=1.2, alpha=0.7,
+ label=f"Comp {comp} (oracle = {oracle_prob:.2f})")
+
+ # Round-robin reference
+ ax_prob.axhline(1.0 / n_comps, color="gray", linestyle=":",
+ linewidth=1.0, alpha=0.6, label="Round-robin")
+
+ ax_prob.set_xlabel("Cycle")
+ ax_prob.set_ylabel("P(heat component)")
+ ax_prob.set_ylim(0, min(1.0, max(oracle["oracle_flexibility"]) * 2))
+ ax_prob.legend(fontsize=12, loc="upper right", ncol=2)
fig.tight_layout()
return fig
+
+
+# ── Reagent-usage action panel ───────────────────────────────────────
+
+
+def _parse_name(
+ name: str,
+ reagent_id_sets: list[set[str]],
+) -> Optional[tuple[str, ...]]:
+ """Parse a product ``Name`` into per-component reagent IDs.
+
+ Reagent IDs may themselves contain underscores (e.g. adenine's
+ ``amidine_034`` or ``isocyanide_db_511``), so a simple split by
+ ``_`` does not recover the components. Instead we greedily consume
+ the leftmost tokens until they form a known ID for the next
+ component.
+
+ Returns ``None`` if the name cannot be parsed.
+ """
+ tokens = name.split("_")
+ n_comps = len(reagent_id_sets)
+ out: list[str] = []
+ i = 0
+ for c in range(n_comps):
+ remaining_comps = n_comps - c - 1
+ found = False
+ # Try shortest prefix first (greedy). Leave at least one token
+ # per remaining component.
+ max_take = len(tokens) - i - remaining_comps
+ for take in range(1, max_take + 1):
+ candidate = "_".join(tokens[i : i + take])
+ if candidate in reagent_id_sets[c]:
+ out.append(candidate)
+ i += take
+ found = True
+ break
+ if not found:
+ return None
+ if i != len(tokens):
+ return None
+ return tuple(out)
+
+
+def _load_reagent_ids(reagent_files: list[Path]) -> list[set[str]]:
+ """Load reagent IDs from .smi files — second column of each row."""
+ out: list[set[str]] = []
+ for rf in reagent_files:
+ ids: set[str] = set()
+ with open(rf, encoding="latin-1") as fh:
+ for line in fh:
+ parts = line.strip().split()
+ if len(parts) >= 2:
+ ids.add(parts[1])
+ out.append(ids)
+ return out
+
+
+def _infer_component_name(reagent_ids: set[str], idx: int) -> str:
+ """Infer a human-readable name from the common prefix of reagent IDs.
+
+ Examples:
+ {"amidine_001", "amidine_034"} → "Amidine"
+ {"isocyanide_db_511", "isocyanide_db_042"} → "Isocyanide"
+ {"aldehyde_305", "aldehyde_112"} → "Aldehyde"
+ {"3043690", "1526634"} → "Component {idx}"
+
+ Returns the first common non-numeric token, capitalised. Falls
+ back to ``"Component {idx}"`` when no chemistry-suggesting prefix
+ can be found (e.g. purely numeric IDs).
+ """
+ if not reagent_ids:
+ return f"Component {idx}"
+ tokens_per_id = [rid.split("_") for rid in reagent_ids]
+ first_tokens = {t[0] for t in tokens_per_id if t}
+ if len(first_tokens) == 1:
+ token = next(iter(first_tokens))
+ if not token.isdigit() and token.isalpha():
+ return token.capitalize()
+ return f"Component {idx}"
+
+
+def _per_batch_reagent_usage(
+ names: list[str],
+ reagent_id_sets: list[set[str]],
+ batch_size: int,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Return (unique_count, new_count) arrays of shape (n_batches, n_components).
+
+ ``unique_count[b, c]`` = number of distinct reagent IDs for component
+ *c* seen in batch *b*. ``new_count[b, c]`` = subset of those that had
+ never appeared in batches 0..b-1.
+ """
+ n_components = len(reagent_id_sets)
+ n_batches = len(names) // batch_size
+ unique_count = np.zeros((n_batches, n_components), dtype=int)
+ new_count = np.zeros((n_batches, n_components), dtype=int)
+ seen: list[set[str]] = [set() for _ in range(n_components)]
+
+ for b in range(n_batches):
+ batch_names = names[b * batch_size : (b + 1) * batch_size]
+ per_comp_ids: list[set[str]] = [set() for _ in range(n_components)]
+ for name in batch_names:
+ parsed = _parse_name(name, reagent_id_sets)
+ if parsed is None:
+ continue
+ for c in range(n_components):
+ per_comp_ids[c].add(parsed[c])
+ for c in range(n_components):
+ unique_count[b, c] = len(per_comp_ids[c])
+ new_count[b, c] = len(per_comp_ids[c] - seen[c])
+ seen[c] |= per_comp_ids[c]
+ return unique_count, new_count
+
+
+def _find_activity_cutoff(
+ new_count: np.ndarray,
+ heated_by_cycle: dict[int, int],
+ window: int = 20,
+ peak_fraction: float = 0.15,
+ buffer: int = 10,
+) -> int:
+ """Find the last batch where the mechanism is still 'active'.
+
+ Active = rolling-window new-reagent discoveries are at least
+ ``peak_fraction`` of the peak rolling total over the full search.
+ Once discoveries drop permanently below the threshold the search
+ has entered an exploitation regime where later batches look
+ essentially identical to each other.
+
+ The peak is taken over batches from index ``window`` onward so the
+ initial warmup-adjacent burst of novel reagents does not inflate
+ the threshold. ``heated_by_cycle`` is reserved for future tuning
+ and is currently unused.
+ """
+ del heated_by_cycle # reserved for future use
+ n_batches = new_count.shape[0]
+ if n_batches == 0:
+ return 0
+ total_new_per_batch = new_count.sum(axis=1)
+ rolling_new = np.zeros(n_batches)
+ for b in range(n_batches):
+ lo = max(0, b - window + 1)
+ rolling_new[b] = total_new_per_batch[lo : b + 1].sum()
+ # Peak over post-startup portion so initial all-new burst does not
+ # dominate the threshold.
+ peak = float(rolling_new[window:].max() if n_batches > window else rolling_new.max())
+ threshold_new = peak * peak_fraction if peak > 0 else 0.0
+ if threshold_new == 0:
+ return n_batches
+
+ last_active = -1
+ for b in range(n_batches):
+ if rolling_new[b] >= threshold_new:
+ last_active = b
+ if last_active < 0:
+ return n_batches
+ return min(n_batches, last_active + 1 + buffer)
+
+
+def plot_reagent_usage_action_panel(
+ search_df: pl.DataFrame,
+ diag_df: pl.DataFrame,
+ reagent_files: list[Path],
+ batch_size: int = 100,
+ replicate: Optional[int] = None,
+ aggregate_window: int = 1,
+ normalize: str = "library",
+ auto_trim: bool = True,
+ max_batches: Optional[int] = None,
+ component_names: Optional[list[str]] = None,
+ title: str = "",
+ figsize: tuple = (14, 5.5),
+ heated_label: str = "Heated",
+) -> Figure:
+ """Two-panel reagent usage diagnostic: heating timeline + new-reagent lines.
+
+ **Top panel (action strip).** A thin categorical timeline showing
+ which component was marked ``is_heated`` at each batch — one
+ coloured block per batch, coloured by the component's palette
+ entry. Reads left-to-right like a sequence diagram. Batches with
+ no heated record appear white. The strip's y-label is set by
+ ``heated_label`` so the figure can use method-appropriate
+ terminology (e.g. ``"Heated"`` for RWS, ``"Explore"`` for TT-TS).
+
+ **Bottom panel (lines).** Per-component lines showing the
+ fraction of that component's reagents first-seen in this batch
+ (``new_count / n_reagents_per_component``) — always
+ library-normalized so the scree-plot decay reads the same across
+ components. Uses the same component palette as the heating strip,
+ so the reader can match each line to the heated-block colours in
+ the strip above. Interpretation: the decay rate summarizes when
+ the algorithm has exhausted the informative reagents per
+ component.
+
+ Note: the ``normalize`` parameter is retained for API
+ compatibility but no longer affects the plot (the bar panel it
+ controlled has been replaced by the heating strip). The value is
+ still validated so stale callers passing unknown strings still
+ raise.
+
+ Args:
+ search_df: Search-phase rows for a single trial (library, query,
+ method, replicate), in evaluation order. Must have ``Name``
+ (underscore-joined reagent IDs) and ``phase`` columns. If
+ ``replicate`` is None the DataFrame must already be filtered
+ to a single replicate.
+ diag_df: Per-cycle diagnostics for the same trial. Must have
+ ``component_idx``, ``current_cycle``, ``is_heated``.
+ reagent_files: Ordered list of ``.smi`` file paths, one per
+ component. Used to count total reagents (for ``library``
+ normalization) and to correctly parse product names where
+ reagent IDs themselves may contain underscores
+ (e.g. adenine's ``isocyanide_db_511``).
+ batch_size: Evaluations per batch (default 100).
+ replicate: If provided, filter both DataFrames to this replicate.
+ aggregate_window: If >1, bin every ``aggregate_window`` batches
+ into a single bar. Useful for long searches (e.g., 1000+
+ batches on docking libraries). Diversity is averaged
+ across batches in the bin; *new* is recomputed against all
+ prior bins; heated component is the majority within the bin.
+ auto_trim: If True (default), automatically crop to the active
+ region of the search — batches where the mechanism is still
+ switching heated components or discovering new reagents.
+ Trailing exploitation-only batches (same component heated
+ for many batches with no new reagents) are dropped and a
+ note is annotated on the figure.
+ max_batches: Hard upper limit on the number of batches shown.
+ Takes precedence over ``auto_trim`` if smaller.
+ component_names: Optional list of display names (one per
+ component) to use in the legend instead of "Component N".
+ If ``None``, names are inferred from the common prefix of
+ each component's reagent IDs (e.g., adenine automatically
+ becomes ["Amidine", "Isocyanide", "Aldehyde"]). Components
+ with purely numeric reagent IDs fall back to "Component N".
+ normalize: ``"library"`` (default) normalizes by total reagents
+ in each component — interpretation: fraction of the library
+ sampled this batch. ``"batch"`` normalizes by
+ ``min(batch_size, n_reagents)`` — interpretation: fraction
+ of the maximum achievable diversity. ``"share"`` normalizes
+ each batch's per-component count by the batch's *total*
+ unique-reagent count across components, producing a
+ stacked-to-1 view. Use ``"share"`` on imbalanced libraries
+ where components differ in size by many-fold, so small
+ components don't disappear visually when heated.
+ title: Plot title.
+ figsize: Figure size.
+ heated_label: Y-axis label for the action strip. Use
+ ``"Heated"`` for RWS-family methods (default) and
+ ``"Explore"`` for TT-TS, where the strip marks the
+ exploring component each batch (cooled / exploit
+ components are the unfilled cells).
+
+ Returns:
+ Matplotlib Figure.
+ """
+ search = search_df
+ diag = diag_df
+ if replicate is not None:
+ search = search.filter(pl.col("replicate") == replicate)
+ diag = diag.filter(pl.col("replicate") == replicate)
+
+ search = search.filter(pl.col("phase") == "search")
+ reagent_id_sets = _load_reagent_ids(reagent_files)
+ n_comps = len(reagent_id_sets)
+ n_reagents_per_component = [len(s) for s in reagent_id_sets]
+ if component_names is None:
+ component_names = [
+ _infer_component_name(reagent_id_sets[c], c) for c in range(n_comps)
+ ]
+ elif len(component_names) != n_comps:
+ raise ValueError(
+ f"component_names has {len(component_names)} entries but "
+ f"there are {n_comps} components"
+ )
+
+ names = search["Name"].to_list()
+ unique_count, new_count = _per_batch_reagent_usage(
+ names, reagent_id_sets, batch_size
+ )
+ n_batches_raw = unique_count.shape[0]
+ old_count = unique_count - new_count
+
+ heated_by_cycle: dict[int, int] = {}
+ for row in diag.iter_rows(named=True):
+ if row["is_heated"]:
+ heated_by_cycle[row["current_cycle"]] = row["component_idx"]
+
+ if aggregate_window > 1:
+ w = aggregate_window
+ n_bins = n_batches_raw // w
+ unique_b = np.zeros((n_bins, n_comps))
+ new_b = np.zeros((n_bins, n_comps))
+ old_b = np.zeros((n_bins, n_comps))
+ heated_b: dict[int, int] = {}
+ for i in range(n_bins):
+ slc = slice(i * w, (i + 1) * w)
+ unique_b[i] = unique_count[slc].mean(axis=0)
+ new_b[i] = new_count[slc].sum(axis=0)
+ old_b[i] = unique_b[i] - new_b[i] / w
+ batch_votes: list[int] = []
+ for j in range(i * w, (i + 1) * w):
+ if j in heated_by_cycle:
+ batch_votes.append(heated_by_cycle[j])
+ if batch_votes:
+ heated_b[i] = max(set(batch_votes), key=batch_votes.count)
+ unique_count = unique_b
+ new_count = new_b / w
+ old_count = old_b
+ heated_by_cycle = heated_b
+ n_batches = n_bins
+ else:
+ n_batches = n_batches_raw
+
+ # Determine plotting cutoff — auto-trim trailing exploitation-only
+ # batches, respecting any user-supplied max_batches hard cap.
+ n_total = n_batches
+ cutoff = n_total
+ if auto_trim:
+ cutoff = _find_activity_cutoff(new_count, heated_by_cycle)
+ if max_batches is not None:
+ cutoff = min(cutoff, max_batches)
+ cutoff = max(1, min(cutoff, n_total))
+ n_trimmed = n_total - cutoff
+
+ if cutoff < n_total:
+ unique_count = unique_count[:cutoff]
+ new_count = new_count[:cutoff]
+ old_count = old_count[:cutoff]
+ heated_by_cycle = {
+ b: c for b, c in heated_by_cycle.items() if b < cutoff
+ }
+ n_batches = cutoff
+
+ # normalize is retained for API compatibility but no longer affects
+ # the plotted data — the bar panel was replaced with a heating
+ # timeline strip. Validate the value so stale callers still error.
+ if normalize not in {"library", "batch", "share"}:
+ raise ValueError(
+ f"normalize must be 'library', 'batch', or 'share', got {normalize!r}"
+ )
+ del unique_count, old_count # no longer used after bar panel removal
+
+ # Line panel: fraction of a component's reagents first-seen in this
+ # batch — a scree-plot decay, comparable across components.
+ n_reagents_arr = np.asarray(n_reagents_per_component, dtype=float)
+ new_frac_lib = new_count / n_reagents_arr[None, :]
+
+ fig, (ax_heat, ax_line) = plt.subplots(
+ 2, 1, figsize=figsize, sharex=True,
+ gridspec_kw={"height_ratios": [0.35, 3], "hspace": 0.08},
+ )
+ fig.patch.set_facecolor("white")
+ ax_heat.set_facecolor("white")
+ ax_line.set_facecolor("white")
+ colors = _component_palette(n_comps)
+
+ # Top panel: heating timeline — one colored block per batch, colored
+ # by the heated component. Batches with no heated record (e.g. all
+ # cooled or missing diagnostics) are left white.
+ for b in range(n_batches):
+ hc = heated_by_cycle.get(b)
+ if hc is None:
+ continue
+ ax_heat.axvspan(b - 0.5, b + 0.5, color=colors[hc], linewidth=0)
+ ax_heat.set_ylim(0, 1)
+ ax_heat.set_yticks([])
+ ax_heat.set_ylabel(
+ heated_label, fontsize=11, rotation=0, ha="right", va="center", labelpad=10,
+ )
+ for spine in ("top", "right", "left"):
+ ax_heat.spines[spine].set_visible(False)
+ ax_heat.tick_params(axis="x", length=0)
+ ax_heat.set_title(title or "Reagent Usage per Batch", fontsize=15, loc="left")
+
+ if n_trimmed > 0:
+ _unit = "batches" if aggregate_window == 1 else "batch bins"
+ ax_heat.annotate(
+ f"algorithm converged after {cutoff} {_unit}\n"
+ f"(remaining {n_trimmed} {_unit} were exploitation-only "
+ f"— no heated-component switching or new reagents)",
+ xy=(0.995, 2.2),
+ xycoords="axes fraction",
+ ha="right", va="top",
+ fontsize=11, color="#333333",
+ bbox=dict(boxstyle="round,pad=0.35",
+ facecolor="white", edgecolor="#bbbbbb", linewidth=0.6),
+ )
+
+ # Bottom panel: per-component lines, fraction of reagents first-seen
+ # this batch. Shares x-axis with the heating strip.
+ x = np.arange(n_batches)
+ for c in range(n_comps):
+ ax_line.plot(
+ x, new_frac_lib[:, c],
+ color=colors[c], linewidth=1.8,
+ marker="o", markersize=3.0,
+ label=component_names[c], zorder=3,
+ )
+ ax_line.set_xlabel(
+ f"Batch bin (window={aggregate_window})" if aggregate_window > 1 else "Batch",
+ fontsize=13,
+ )
+ ax_line.set_ylabel(
+ "Fraction of new reagents\nper reaction component", fontsize=13,
+ )
+ ax_line.tick_params(axis="both", labelsize=12)
+ _line_ymax = (
+ max(1e-3, float(new_frac_lib.max()) * 1.12) if new_frac_lib.size else 0.05
+ )
+ ax_line.set_ylim(0, _line_ymax)
+ ax_line.set_xlim(-0.5, n_batches - 0.5)
+ ax_line.grid(axis="y", color="#dddddd", linewidth=0.5, zorder=0)
+
+ # Legend: component colors (shared between strip and lines).
+ from matplotlib.patches import Patch
+
+ handles = [
+ Patch(facecolor=colors[c], edgecolor="none", label=component_names[c])
+ for c in range(n_comps)
+ ]
+ ax_line.legend(
+ handles=handles,
+ loc="upper center",
+ bbox_to_anchor=(0.5, -0.22),
+ ncol=len(handles),
+ fontsize=12,
+ frameon=False,
+ handlelength=1.6,
+ columnspacing=1.6,
+ )
+ fig.tight_layout()
+ return fig
+
+
+def plot_gmic_directed_exploration(
+ search_df: pl.DataFrame,
+ diag_df: pl.DataFrame,
+ reagent_files: list[Path],
+ methods: list[str],
+ *,
+ method_labels=None,
+ replicate=None,
+ batch_size: int = 100,
+ component_names=None,
+ component_colors=None,
+ gmic_ref: float = 1.0,
+ title=None,
+ figsize=None,
+):
+ """Layer 1: how GMIC directs WHICH component to explore (signal -> decision -> result).
+
+ One column per method; three stacked rows sharing the search-cycle x-axis:
+
+ SIGNAL -- per-component GMIC (criticality). High = the component's reagent
+ means are well separated relative to noise (resolved -> exploit);
+ low = unresolved (explore).
+ DECISION -- the per-component "explored" strip (``is_heated``): which component
+ the GMIC-weighted rotation sends exploration to each cycle.
+ RESULT -- cumulative per-component reagent coverage (fraction of *that*
+ component's reagents discovered), each normalised to its own size.
+
+ On an imbalanced library (thrombin: 130 acids x 3844 dipeptides) the small
+ component resolves immediately (high GMIC -> exploited -> coverage saturates to
+ 100%) while the large one stays unresolved (low GMIC -> explored -> coverage
+ climbs), making the GMIC-directed explore/exploit reallocation visible end to end.
+
+ Args:
+ search_df: Per-evaluation search log (``Name``, ``method``, ``replicate``,
+ ``phase``); only ``phase == "search"`` rows are used.
+ diag_df: Per-cycle diagnostics (``method``, ``replicate``, ``current_cycle``,
+ ``component_idx``, ``gmic``, ``is_heated``).
+ reagent_files: One ``.smi`` per component (defines each component's reagent IDs).
+ methods: Method names to show as columns (e.g. the RWS and TT-TS labels).
+ method_labels: Optional ``{method: display label}`` for the column titles.
+ replicate: Replicate to plot; defaults to the smallest in ``diag_df``.
+ batch_size: Evaluations per cycle/batch (default 100).
+ component_names: Display names per component; inferred from reagent-ID
+ prefixes when ``None``.
+ component_colors: Line colours per component.
+ gmic_ref: Reference line on the GMIC row (default 1.0 = critical/flexible boundary).
+ title: Figure suptitle.
+ figsize: Defaults to ``(7.5 * n_methods, 9.2)``.
+
+ Returns:
+ Matplotlib Figure.
+ """
+ rid = _load_reagent_ids(reagent_files)
+ ncomp = len(rid)
+ nreag = [len(s) for s in rid]
+ if component_names is None:
+ component_names = [_infer_component_name(rid[c], c) for c in range(ncomp)]
+ if component_colors is None:
+ _palette = ["#4477AA", "#EE6677", "#228833", "#CCBB44", "#AA3377"]
+ component_colors = [_palette[c % len(_palette)] for c in range(ncomp)]
+ if method_labels is None:
+ method_labels = {}
+ if replicate is None:
+ replicate = int(diag_df["replicate"].min())
+
+ ncols = len(methods)
+ if figsize is None:
+ figsize = (7.5 * ncols, 9.2)
+ fig, axes = plt.subplots(
+ 3, ncols, figsize=figsize, squeeze=False,
+ gridspec_kw={"height_ratios": [1.1, 0.35, 1.1]},
+ )
+
+ for col, m in enumerate(methods):
+ d = diag_df.filter(
+ (pl.col("method") == m) & (pl.col("replicate") == replicate)
+ )
+ ncyc = int(d["current_cycle"].max()) + 1
+ gmic = np.full((ncyc, ncomp), np.nan)
+ heat = np.zeros((ncomp, ncyc))
+ for r in d.iter_rows(named=True):
+ gmic[r["current_cycle"], r["component_idx"]] = r["gmic"]
+ if r["is_heated"]:
+ heat[r["component_idx"], r["current_cycle"]] = 1.0
+
+ s = search_df.filter(
+ (pl.col("method") == m)
+ & (pl.col("replicate") == replicate)
+ & (pl.col("phase") == "search")
+ )
+ _, new = _per_batch_reagent_usage(s["Name"].to_list(), rid, batch_size)
+ cov = np.cumsum(new, axis=0) / np.array(nreag) * 100.0
+ nb = cov.shape[0]
+
+ a0, a1, a2 = axes[0, col], axes[1, col], axes[2, col]
+ a0.set_facecolor("white")
+ for c in range(ncomp):
+ a0.plot(np.arange(ncyc), gmic[:, c], color=component_colors[c],
+ lw=2, label=f"{component_names[c]} ({nreag[c]})")
+ a0.axhline(gmic_ref, color="grey", ls=":", lw=0.8)
+ a0.set_title(method_labels.get(m, m), fontsize=13, fontweight="bold",
+ color="black")
+ a0.legend(fontsize=9, loc="center right", framealpha=0.9)
+ a0.grid(alpha=0.3, color="grey")
+ a0.tick_params(colors="black")
+
+ _im = a1.imshow(heat, aspect="auto", cmap="Oranges", vmin=0, vmax=1,
+ extent=[0, ncyc, ncomp - 0.5, -0.5], interpolation="nearest")
+ a1.set_yticks(range(ncomp))
+ a1.set_yticklabels([cn[:9] for cn in component_names], fontsize=9,
+ color="black")
+ a1.tick_params(colors="black")
+
+ a2.set_facecolor("white")
+ for c in range(ncomp):
+ a2.plot(np.arange(nb), cov[:, c], color=component_colors[c], lw=2)
+ a2.set_xlabel("search cycle (batch)", fontsize=11, color="black")
+ a2.set_ylim(-3, 103)
+ a2.grid(alpha=0.3, color="grey")
+ a2.tick_params(colors="black")
+
+ if col == 0:
+ a0.set_ylabel("SIGNAL\nGMIC (criticality)", fontsize=11, color="black")
+ a1.set_ylabel("DECISION\nexplored", fontsize=11, color="black")
+ a2.set_ylabel("RESULT\ncomponent coverage (%)", fontsize=11,
+ color="black")
+
+ if title:
+ fig.suptitle(title, fontsize=13, fontweight="bold", color="black", y=1.0)
+ fig.tight_layout(rect=(0, 0, 0.93, 1))
+
+ # Colorbar key for the DECISION (explored) strip -- placed in the reserved
+ # right margin, vertically aligned with the (middle) decision row, so it
+ # does not disturb the shared-x alignment of the three rows.
+ _pos = axes[1, -1].get_position()
+ _cax = fig.add_axes([0.945, _pos.y0 - 0.02, 0.012, _pos.height + 0.04])
+ _cb = fig.colorbar(_im, cax=_cax)
+ _cb.set_ticks([0.05, 0.95])
+ _cb.ax.set_yticklabels(
+ ["exploit\n(cooled)", "explore\n(heated)"], fontsize=8, color="black"
+ )
+ _cb.ax.tick_params(length=0)
+ _cb.set_label("exploration\ndecision", fontsize=8.5, color="black")
+ return fig
+
+
+# Per-method within-component intensity knob. RWS modulates the Boltzmann
+# temperature; TT-TS modulates the posterior-σ inflation scale. Both are the
+# "how hard do I explore *within* the chosen component" lever, one level below
+# the GMIC "which component" decision shown by plot_gmic_directed_exploration.
+#
+# Both knobs are reported as dimensionless multipliers centred on 1.0, so the
+# two methods are directly comparable on a shared interpretation (1.0 = no
+# adjustment; >1 = amplify exploration):
+# RWS -> ``cats_multiplier``: the GMIC-driven factor that scales the base
+# thermal-cycling temperature (final_temperature = base_temp x mult).
+# TT-TS -> ``heated_scale``: the disagreement-driven factor that scales the
+# posterior σ. This is the smooth per-component state the controller
+# sets, NOT the per-cycle ``effective_scale`` (= heated_scale on heated
+# cycles, cooled_scale otherwise), which just re-encodes the heating
+# rotation already shown in the DECISION row.
+# ``effective_scale`` / ``final_temperature`` are kept only as last-resort
+# fallbacks for older diagnostics that lack the multiplier columns.
+_INTENSITY_SPECS = (
+ # (column, display label, reference value)
+ ("cats_multiplier", "CATS multiplier (×T)", 1.0),
+ ("heated_scale", "σ-inflation multiplier (×σ)", 1.0),
+ ("effective_scale", "σ-inflation scale (effective)", 1.0),
+ ("final_temperature", "Boltzmann temperature", None),
+)
+
+
+def _detect_intensity_col(d: pl.DataFrame) -> tuple[str, str, Optional[float]]:
+ """Pick the within-component intensity column present for these rows.
+
+ RWS diagnostics carry ``final_temperature`` (TT-TS columns null) and TT-TS
+ diagnostics carry ``effective_scale`` (RWS columns null); the canonical
+ benchmark stores both methods in one union-schema parquet. Returns the
+ ``(column, label, reference)`` of the first spec whose column exists and has
+ at least one non-null value.
+ """
+ for col, label, ref in _INTENSITY_SPECS:
+ if col in d.columns and d[col].drop_nulls().len() > 0:
+ return col, label, ref
+ raise ValueError(
+ "No within-component intensity column "
+ f"({', '.join(s[0] for s in _INTENSITY_SPECS)}) found in diagnostics."
+ )
+
+
+def plot_adaptive_intensity(
+ search_df: pl.DataFrame,
+ diag_df: pl.DataFrame,
+ reagent_files: list[Path],
+ methods: list[str],
+ *,
+ method_labels=None,
+ replicate=None,
+ batch_size: int = 100,
+ component_names=None,
+ component_colors=None,
+ smooth_window: int = 9,
+ show_decision: bool = True,
+ result: str = "new",
+ title=None,
+ figsize=None,
+):
+ """Layer 2: how each method tunes exploration *within* a component.
+
+ Where :func:`plot_gmic_directed_exploration` (layer 1) shows WHICH component
+ GMIC sends exploration to, this shows HOW HARD each method explores the
+ chosen component, and what that yields in reagent discovery. One column per
+ method; rows share the search-cycle x-axis:
+
+ INTENSITY -- the per-component within-component knob, smoothed. Both knobs
+ are reported as **dimensionless multipliers centred on 1.0**
+ (1.0 = no adjustment; >1 = amplify exploration), so the two
+ methods read on a common interpretation: RWS modulates the
+ **CATS temperature multiplier** (``cats_multiplier``, ×base
+ temperature -- the GMIC-driven factor); TT-TS modulates the
+ **σ-inflation multiplier** (``heated_scale``, ×posterior σ --
+ the disagreement-driven factor). Note the asymmetry the knobs
+ reveal: RWS's multiplier is GMIC-driven, so it *amplifies above
+ 1.0* on the unresolved (low-GMIC) component; TT-TS's heated_scale
+ is disagreement-driven, and because two-sample disagreement
+ saturates (>0.8) on both components of a large library it
+ *decays toward 1.0* -- most on the highest-disagreement
+ (unresolved) component -- i.e. the TT-TS adaptation is largely
+ one-directional and cannot manufacture exploration beyond what
+ the raw posteriors already provide.
+ DECISION -- per-component **heating probability** (rolling fraction of
+ cycles in which the component was the explored/heated one).
+ Omitted when ``show_decision=False`` (leaving INTENSITY +
+ RESULT, i.e. "just the temperature").
+ RESULT -- per-component within-component discovery. ``result="new"``
+ (default) plots **new reagents discovered per batch** (the
+ per-batch *rate* -- the discrete increment of coverage);
+ ``result="coverage"`` plots **cumulative coverage** (% of the
+ component's reagents seen at least once -- the running total,
+ identical to the layer-1 RESULT row). New-per-batch is the
+ derivative of coverage: it falls toward zero as the reagent
+ pool is exhausted even while coverage is still rising.
+
+ On thrombin (130 acids × 3844 dipeptides) both methods keep discovering new
+ dipeptides long after the small acid component is exhausted -- the same
+ reallocation, reached through two different within-component mechanisms.
+
+ Args:
+ search_df: Per-evaluation search log (``Name``, ``method``,
+ ``replicate``, ``phase``); only ``phase == "search"`` rows are used.
+ diag_df: Per-cycle diagnostics (``method``, ``replicate``,
+ ``current_cycle``, ``component_idx``, ``is_heated`` plus the
+ method-specific intensity column ``cats_multiplier`` for RWS or
+ ``heated_scale`` for TT-TS).
+ reagent_files: One ``.smi`` per component (defines each component's
+ reagent IDs).
+ methods: Method names to show as columns.
+ method_labels: Optional ``{method: display label}`` for column titles.
+ replicate: Replicate to plot; defaults to the smallest in ``diag_df``.
+ batch_size: Evaluations per cycle/batch (default 100).
+ component_names: Display names per component; inferred from reagent-ID
+ prefixes when ``None``.
+ component_colors: Line colours per component.
+ smooth_window: Centred (edge-shrinking) rolling-mean window (cycles)
+ applied to every row so the per-cycle heated/cooled square-wave reads
+ as an effective average (default 9).
+ show_decision: Include the heating-probability row (default True).
+ result: ``"new"`` (new reagents per batch, default) or ``"coverage"``
+ (cumulative % coverage, as in layer 1).
+ title: Figure suptitle.
+ figsize: Defaults to ``(7.5 * n_methods, 9.2 or 7.0)``.
+
+ Returns:
+ Matplotlib Figure.
+ """
+ if result not in ("new", "coverage"):
+ raise ValueError(f"result must be 'new' or 'coverage', got {result!r}")
+ rid = _load_reagent_ids(reagent_files)
+ ncomp = len(rid)
+ nreag = [len(s) for s in rid]
+ if component_names is None:
+ component_names = [_infer_component_name(rid[c], c) for c in range(ncomp)]
+ if component_colors is None:
+ _palette = ["#4477AA", "#EE6677", "#228833", "#CCBB44", "#AA3377"]
+ component_colors = [_palette[c % len(_palette)] for c in range(ncomp)]
+ if method_labels is None:
+ method_labels = {}
+ if replicate is None:
+ replicate = int(diag_df["replicate"].min())
+
+ ncols = len(methods)
+ nrows = 3 if show_decision else 2
+ if figsize is None:
+ figsize = (7.5 * ncols, 9.2 if show_decision else 7.0)
+ height_ratios = [1.1, 0.35, 1.1] if show_decision else [1.1, 1.1]
+ fig, axes = plt.subplots(
+ nrows, ncols, figsize=figsize, squeeze=False,
+ gridspec_kw={"height_ratios": height_ratios},
+ )
+ decision_im = None
+
+ for col, m in enumerate(methods):
+ d = diag_df.filter(
+ (pl.col("method") == m) & (pl.col("replicate") == replicate)
+ )
+ icol, ilabel, iref = _detect_intensity_col(d)
+ ncyc = int(d["current_cycle"].max()) + 1
+ intensity = np.full((ncyc, ncomp), np.nan)
+ heat = np.zeros((ncyc, ncomp))
+ for r in d.iter_rows(named=True):
+ intensity[r["current_cycle"], r["component_idx"]] = r[icol]
+ if r["is_heated"]:
+ heat[r["current_cycle"], r["component_idx"]] = 1.0
+
+ s = search_df.filter(
+ (pl.col("method") == m)
+ & (pl.col("replicate") == replicate)
+ & (pl.col("phase") == "search")
+ )
+ _, new = _per_batch_reagent_usage(s["Name"].to_list(), rid, batch_size)
+ nb = new.shape[0]
+ if result == "coverage":
+ res = np.cumsum(new, axis=0) / np.array(nreag) * 100.0
+ res_label = "RESULT\ncomponent coverage (%)"
+ else:
+ res = new.astype(float)
+ res_label = "RESULT\nnew reagents / batch"
+
+ a_int = axes[0, col]
+ a_res = axes[nrows - 1, col]
+
+ # --- INTENSITY row -------------------------------------------------
+ a_int.set_facecolor("white")
+ for c in range(ncomp):
+ a_int.plot(
+ np.arange(ncyc), _smooth_trim(intensity[:, c], smooth_window),
+ color=component_colors[c], lw=2, label=component_names[c],
+ )
+ if iref is not None:
+ a_int.axhline(iref, color="grey", ls=":", lw=0.8)
+ a_int.set_title(method_labels.get(m, m), fontsize=13, fontweight="bold",
+ color="black")
+ a_int.legend(fontsize=9, loc="best", framealpha=0.9)
+ a_int.grid(alpha=0.3, color="grey")
+ a_int.tick_params(colors="black")
+ a_int.set_ylabel(f"INTENSITY\n{ilabel}", fontsize=11, color="black")
+
+ # --- DECISION row (heating probability) ----------------------------
+ if show_decision:
+ a_dec = axes[1, col]
+ heat_prob = np.column_stack(
+ [_smooth_trim(heat[:, c], smooth_window) for c in range(ncomp)]
+ ).T # shape (ncomp, ncyc)
+ decision_im = a_dec.imshow(
+ heat_prob, aspect="auto", cmap="Oranges", vmin=0, vmax=1,
+ extent=[0, ncyc, ncomp - 0.5, -0.5], interpolation="nearest",
+ )
+ a_dec.set_yticks(range(ncomp))
+ a_dec.set_yticklabels([cn[:9] for cn in component_names], fontsize=9,
+ color="black")
+ a_dec.tick_params(colors="black")
+ if col == 0:
+ a_dec.set_ylabel("DECISION\nP(explore)", fontsize=11,
+ color="black")
+
+ # --- RESULT row (within-component discovery) -----------------------
+ a_res.set_facecolor("white")
+ for c in range(ncomp):
+ a_res.plot(
+ np.arange(nb), _smooth_trim(res[:, c], smooth_window),
+ color=component_colors[c], lw=2,
+ )
+ a_res.set_xlabel("search cycle (batch)", fontsize=11, color="black")
+ if result == "coverage":
+ a_res.set_ylim(-3, 103)
+ else:
+ a_res.set_ylim(bottom=-0.5)
+ a_res.grid(alpha=0.3, color="grey")
+ a_res.tick_params(colors="black")
+ if col == 0:
+ a_res.set_ylabel(res_label, fontsize=11, color="black")
+
+ if title:
+ fig.suptitle(title, fontsize=13, fontweight="bold", color="black", y=1.0)
+ fig.tight_layout(rect=(0, 0, 0.93, 1) if show_decision else None)
+
+ if show_decision and decision_im is not None:
+ _pos = axes[1, -1].get_position()
+ _cax = fig.add_axes([0.945, _pos.y0 - 0.02, 0.012, _pos.height + 0.04])
+ _cb = fig.colorbar(decision_im, cax=_cax)
+ _cb.set_ticks([0.05, 0.95])
+ _cb.ax.set_yticklabels(["never\nexplore", "always\nexplore"], fontsize=8,
+ color="black")
+ _cb.ax.tick_params(length=0)
+ _cb.set_label("heating\nprobability", fontsize=8.5, color="black")
+ return fig
+
diff --git a/src/TACTICS/thompson_sampling/config.py b/src/TACTICS/thompson_sampling/config.py
index e34c40d..5e7109f 100644
--- a/src/TACTICS/thompson_sampling/config.py
+++ b/src/TACTICS/thompson_sampling/config.py
@@ -138,8 +138,30 @@ class ThompsonSamplingConfig(BaseModel):
)
log_filename: Optional[str] = Field(default=None, description="Log file path")
- # Performance (Note: multiprocessing typically not needed for compound generation)
+ # Performance
hide_progress: bool = Field(default=False, description="Hide progress bars")
+ processes: int = Field(
+ default=1,
+ ge=1,
+ description=(
+ "Number of worker processes used for batch evaluation. Leave at 1 "
+ "for fast evaluators (Lookup, DB, FP, MW), where process overhead "
+ "exceeds the cost of scoring and parallelism makes runs slower. "
+ "Set it to the number of allocated cores for slow evaluators "
+ "(Fred docking, ROCS, ML models), where evaluation dominates "
+ "runtime. Compound generation itself is fast; this parameter "
+ "exists for the evaluation step."
+ ),
+ )
+ min_cpds_per_core: int = Field(
+ default=10,
+ ge=1,
+ description=(
+ "Minimum compounds per worker before a batch is evaluated in "
+ "parallel. Batches smaller than processes * min_cpds_per_core are "
+ "evaluated sequentially to avoid process overhead on small batches."
+ ),
+ )
# Diagnostics
track_diagnostics: bool = Field(
diff --git a/src/TACTICS/thompson_sampling/core/evaluator_config.py b/src/TACTICS/thompson_sampling/core/evaluator_config.py
index 2b3be57..c38a835 100644
--- a/src/TACTICS/thompson_sampling/core/evaluator_config.py
+++ b/src/TACTICS/thompson_sampling/core/evaluator_config.py
@@ -16,6 +16,15 @@ class LookupEvaluatorConfig(BaseModel):
ref_filename: str = Field(..., description="Path to CSV file with pre-computed scores")
compound_col: str = Field(default="Product_Code", description="Column name for compound identifiers")
score_col: str = Field(default="Scores", description="Column name for scores in CSV")
+ default_score: Optional[float] = Field(
+ default=None,
+ description=(
+ "Score returned for product codes absent from the lookup table. "
+ "None → np.nan (the sampler skips NaN evaluations). Set to 0.0 for "
+ "sparse lookup libraries such as DEL read counts, where a combination "
+ "absent from the measured set is a true non-binder (score 0)."
+ ),
+ )
class DBEvaluatorConfig(BaseModel):
diff --git a/src/TACTICS/thompson_sampling/core/evaluators.py b/src/TACTICS/thompson_sampling/core/evaluators.py
index e4113f8..970565d 100644
--- a/src/TACTICS/thompson_sampling/core/evaluators.py
+++ b/src/TACTICS/thompson_sampling/core/evaluators.py
@@ -5,16 +5,59 @@
import useful_rdkit_utils as uru
-try:
- from openeye import oechem
- from openeye import oeomega
- from openeye import oeshape
- from openeye import oedocking
- import joblib
-except ImportError:
- # Since openeye is a commercial software package, just pass with a warning if not available
- warnings.warn(f"Openeye packages not available in this environment; do not attempt to use ROCSEvaluator or "
- f"FredEvaluator")
+# OpenEye toolkit modules are loaded lazily by _ensure_openeye() rather than
+# at module import time. They stay None until an OpenEye-backed evaluator is
+# actually constructed.
+oechem = None
+oeomega = None
+oeshape = None
+oedocking = None
+
+
+def _ensure_openeye():
+ """Import the OpenEye toolkits on first use and cache them as module globals.
+
+ Deferred rather than imported at module level for two reasons:
+
+ 1. Segfault avoidance. OpenEye's shared libraries load libexpat.so.1 and
+ initialize its global C-level XML parser state. tqdm pulls in
+ prompt_toolkit, whose progress-bar formatter calls
+ xml.dom.minidom.parseString() at *module* level, re-entering libexpat
+ through Python's pyexpat extension. When OpenEye initialized expat
+ first, that second entry conflicts with the existing global state and
+ segfaults the interpreter (exit 139) during `import TACTICS`, before
+ any user code runs. Deferring the OpenEye load until an evaluator is
+ constructed lets Python own and initialize expat first.
+
+ 2. Import cost. Most TACTICS runs use Lookup, FP, MW or DB evaluators and
+ never need the OpenEye toolkits at all.
+
+ Safe to call repeatedly; the import happens once.
+
+ Raises:
+ ImportError: if the OpenEye toolkits are not installed.
+ """
+ global oechem, oeomega, oeshape, oedocking
+
+ if oechem is not None:
+ return
+
+ try:
+ from openeye import oechem as _oechem
+ from openeye import oeomega as _oeomega
+ from openeye import oeshape as _oeshape
+ from openeye import oedocking as _oedocking
+ except ImportError as exc:
+ raise ImportError(
+ "The OpenEye toolkits are required for ROCSEvaluator and "
+ "FredEvaluator but are not installed in this environment. "
+ "Install them with `pip install openeye-toolkits` (a valid "
+ "OpenEye license is required)."
+ ) from exc
+
+ oechem, oeomega, oeshape, oedocking = _oechem, _oeomega, _oeshape, _oedocking
+
+
from rdkit import Chem, DataStructs
import pandas as pd
from sqlitedict import SqliteDict
@@ -70,6 +113,7 @@ class ROCSEvaluator(Evaluator):
"""
def __init__(self, input_dict):
+ _ensure_openeye()
ref_filename = input_dict['query_molfile']
ref_fs = oechem.oemolistream(ref_filename)
self.ref_mol = oechem.OEMol()
@@ -141,6 +185,10 @@ def __init__(self, input_dictionary):
ref_filename = input_dictionary['ref_filename']
compound_col = input_dictionary.get('compound_col', 'Product_Code')
score_col = input_dictionary.get('score_col', 'Scores')
+ # Score for product codes absent from the table. None preserves the
+ # historical np.nan behavior; 0.0 is used for sparse DEL read-count
+ # libraries where an unmeasured combination is a true non-binder.
+ self.default_score = input_dictionary.get('default_score', None)
# Determine file type and read accordingly
if ref_filename.endswith('.parquet'):
@@ -158,8 +206,11 @@ def counter(self):
def evaluate(self, product_name):
self.num_evaluations += 1
- # Return score from lookup, or np.nan if product not in reference
- return self.ref_dict.get(product_name, np.nan)
+ # Return score from lookup. If the product is absent, return the
+ # configured default (e.g. 0.0 for sparse read-count libraries) or
+ # np.nan when no default is set (sampler skips NaN evaluations).
+ missing = self.default_score if self.default_score is not None else np.nan
+ return self.ref_dict.get(product_name, missing)
class DBEvaluator(Evaluator):
"""A simple evaluator class that looks up values from a database.
@@ -197,6 +248,7 @@ class FredEvaluator(Evaluator):
"""
def __init__(self, input_dict):
+ _ensure_openeye()
du_file = input_dict["design_unit_file"]
if not os.path.isfile(du_file):
raise FileNotFoundError(f"{du_file} was not found or is a directory")
@@ -277,6 +329,7 @@ def generate_confs(mol, max_confs):
:param mol: input OEMolecule
:return: Boolean Omega return code indicating success of conformer generation
"""
+ _ensure_openeye()
rms = 0.5
strict_stereo = False
omega = oeomega.OEOmega()
@@ -297,6 +350,7 @@ def read_design_unit(filename):
:param filename: design unit filename (.oedu)
:return: a docking grid
"""
+ _ensure_openeye()
du = oechem.OEDesignUnit()
rfs = oechem.oeifstream()
if not rfs.open(filename):
@@ -342,6 +396,11 @@ class MLClassifierEvaluator(Evaluator):
"""
def __init__(self, input_dict):
+ # joblib was previously imported inside the OpenEye try/except, so it
+ # was undefined whenever OpenEye was absent even though it has nothing
+ # to do with OpenEye. Import it where it is actually used.
+ import joblib
+
self.cls = joblib.load(input_dict["model_filename"])
self.num_evaluations = 0
diff --git a/src/TACTICS/thompson_sampling/core/parallel_evaluator.py b/src/TACTICS/thompson_sampling/core/parallel_evaluator.py
index 5d9f03b..51377b0 100644
--- a/src/TACTICS/thompson_sampling/core/parallel_evaluator.py
+++ b/src/TACTICS/thompson_sampling/core/parallel_evaluator.py
@@ -2,9 +2,67 @@
Parallel evaluation wrapper for Thompson Sampling.
Handles multi-process evaluation of compound batches using multiprocessing.Pool.
+
+Why this module is not a plain ``pool.map(sampler.evaluate, ...)``
+------------------------------------------------------------------
+``pool.map`` serialises the task function and its arguments with ``pickle`` to
+send them down a pipe to each worker. ``sampler.evaluate`` is a bound method,
+so pickling it means pickling the sampler, which owns the evaluator. For
+OpenEye-backed evaluators the evaluator holds a SWIG-wrapped C++ object
+(``OEDock`` for Fred, the shape engine for ROCS), and SWIG objects raise
+``TypeError: cannot pickle 'SwigPyObject' object`` on any pickle attempt.
+The result was that ``processes > 1`` crashed on the very first batch for
+exactly the slow evaluators that need parallelism most.
+
+The fix is to never send an evaluator across the pipe. Each worker builds its
+*own* evaluator instead, once, from the evaluator's picklable Pydantic config
+(a receptor file path plus options). See :func:`_init_worker`.
"""
-from multiprocessing import Pool
-from typing import List, Callable, Any, Optional
+import multiprocessing
+from typing import Any, Callable, List, Optional, Tuple
+
+# Worker-process globals. Populated once per worker by _init_worker() and
+# reused for every task that worker handles. They only ever exist inside a
+# worker process; in the parent they stay None.
+_worker_sampler = None
+
+
+def _init_worker(sampler, evaluator_config):
+ """Pool initializer: rebuild this worker's own evaluator.
+
+ Runs exactly once per worker process, at pool creation. ``sampler`` arrives
+ without its evaluator (see ``ThompsonSampler.__getstate__``), and this
+ function constructs a fresh evaluator here, inside the worker, from the
+ picklable ``evaluator_config``. The evaluator -- and any SWIG object it
+ owns, such as ``OEDock`` -- is therefore created in the worker and never
+ crosses the pickle boundary.
+
+ Building it in the initializer rather than per task also means the design
+ unit is read and the docking engine constructed once per worker, not once
+ per molecule.
+
+ Args:
+ sampler: A ThompsonSampler pickled without its evaluator, pool, or logger.
+ evaluator_config: Picklable evaluator config (a Pydantic model).
+ """
+ global _worker_sampler
+
+ # Imported here rather than at module scope to avoid a circular import
+ # (factories -> evaluators -> ... -> sampler -> parallel_evaluator).
+ from ..factories import create_evaluator
+
+ sampler.set_evaluator(create_evaluator(evaluator_config))
+ _worker_sampler = sampler
+
+
+def _worker_evaluate(choice_list):
+ """Top-level task function, therefore picklable.
+
+ Only the lightweight ``choice_list`` (a list of reagent indices) is sent
+ over the pipe. The sampler and evaluator are already resident in the
+ worker, installed by :func:`_init_worker`.
+ """
+ return _worker_sampler.evaluate(choice_list)
class ParallelEvaluator:
@@ -13,6 +71,16 @@ class ParallelEvaluator:
The pool is created once and reused across all evaluations to avoid
the overhead of repeatedly spawning processes.
+
+ Two execution paths:
+
+ - ``processes == 1``: sequential, calling ``evaluate_fn`` directly. No
+ pickling and no worker processes.
+ - ``processes > 1``: a persistent pool whose workers each construct their
+ own evaluator via :func:`_init_worker`. Requires a worker context to
+ have been bound with :meth:`bind_worker_context`; without one, falls
+ back to sending ``evaluate_fn`` itself, which works for picklable
+ evaluators but not for OpenEye-backed ones.
"""
def __init__(self, processes: int = 1):
@@ -23,12 +91,43 @@ def __init__(self, processes: int = 1):
processes: Number of CPU cores to use. If 1, uses sequential evaluation.
"""
self.processes = processes
- self._pool: Optional[Pool] = None
+ self._pool: Optional[Any] = None
+ self._worker_context: Optional[Tuple[Any, Any]] = None
+
+ def bind_worker_context(self, sampler, evaluator_config) -> None:
+ """Record what each worker needs in order to rebuild its own evaluator.
+
+ Called by ``ThompsonSampler.set_evaluator`` when the evaluator's config
+ is known (i.e. when the sampler was built by ``from_config``, or when
+ ``set_evaluator`` was given an explicit ``evaluator_config``).
+
+ Any existing pool is shut down, because its workers were initialized
+ from the previous context and would otherwise keep using a stale
+ evaluator.
+
+ Args:
+ sampler: The ThompsonSampler whose ``evaluate`` the workers run.
+ evaluator_config: Picklable evaluator config used to rebuild the
+ evaluator inside each worker. If None, the context is cleared.
+ """
+ self.close()
+ self._worker_context = (
+ None if evaluator_config is None else (sampler, evaluator_config)
+ )
def _ensure_pool(self):
"""Create the process pool if it doesn't exist."""
if self.processes > 1 and self._pool is None:
- self._pool = Pool(self.processes)
+ if self._worker_context is not None:
+ self._pool = multiprocessing.Pool(
+ self.processes,
+ initializer=_init_worker,
+ initargs=self._worker_context,
+ )
+ else:
+ # No config to rebuild from; workers get whatever the parent
+ # sends them. Fine for picklable evaluators, fatal for SWIG.
+ self._pool = multiprocessing.Pool(self.processes)
def close(self):
"""Close the process pool if it exists."""
@@ -44,25 +143,50 @@ def evaluate_batch(self,
Evaluate a batch of compounds in parallel.
Args:
- evaluate_fn: Function that takes a compound representation (e.g., list of reagent indices)
- and returns [score, smiles, name]
+ evaluate_fn: Function that takes a compound representation (e.g. list of reagent indices)
+ and returns [score, smiles, name]. Used directly when running
+ sequentially, and as the fallback task function when no worker
+ context has been bound.
compound_list: List of compound representations to evaluate
Returns:
List of evaluation results [score, smiles, name] for each compound
+
+ Raises:
+ TypeError: if ``processes > 1`` with no worker context bound and the
+ evaluator cannot be pickled. The message explains how to fix it.
"""
if not compound_list:
return []
- # Use parallel evaluation if processes > 1
- if self.processes > 1:
- self._ensure_pool()
- results = self._pool.map(evaluate_fn, compound_list)
- else:
- # Sequential evaluation for single process
- results = [evaluate_fn(compound) for compound in compound_list]
+ # Sequential evaluation for single process
+ if self.processes <= 1:
+ return [evaluate_fn(compound) for compound in compound_list]
+
+ self._ensure_pool()
+
+ # Preferred path: workers already hold their own evaluator.
+ if self._worker_context is not None:
+ return self._pool.map(_worker_evaluate, compound_list)
- return results
+ # Fallback path: ship the bound method. Works for picklable evaluators
+ # (Lookup, FP, MW, DB); fails loudly and usefully for SWIG-backed ones.
+ try:
+ return self._pool.map(evaluate_fn, compound_list)
+ except TypeError as exc:
+ if "pickle" not in str(exc).lower():
+ raise
+ raise TypeError(
+ "Cannot run this evaluator with processes > 1: it holds an "
+ "object that cannot be pickled (OpenEye OEDock and ROCS shape "
+ "objects are SWIG-wrapped C++ objects and are never "
+ "picklable).\n\n"
+ "Fix: build the sampler with ThompsonSampler.from_config(config), "
+ "or pass the evaluator's config explicitly via "
+ "sampler.set_evaluator(evaluator, evaluator_config=...). Either "
+ "lets each worker construct its own evaluator instead of "
+ "receiving one over the pipe."
+ ) from exc
def __del__(self):
"""Cleanup: close pool when evaluator is garbage collected."""
diff --git a/src/TACTICS/thompson_sampling/core/sampler.py b/src/TACTICS/thompson_sampling/core/sampler.py
index a838e5b..9680b20 100644
--- a/src/TACTICS/thompson_sampling/core/sampler.py
+++ b/src/TACTICS/thompson_sampling/core/sampler.py
@@ -74,6 +74,10 @@ def __init__(
self.warmup_strategy = warmup_strategy or StandardWarmup()
self.reagent_lists = []
self.evaluator = None
+ # Picklable recipe for self.evaluator, when known. Workers use this to
+ # build their own evaluator instead of receiving one over a pipe.
+ self.evaluator_config = None
+ self._log_filename = log_filename
self.logger = get_logger(__name__, filename=log_filename)
self._disallow_tracker = None
self.batch_size = batch_size
@@ -181,16 +185,18 @@ def from_config(cls, config: "ThompsonSamplingConfig") -> "ThompsonSampler":
log_filename=config.log_filename,
batch_size=config.batch_size,
max_resamples=config.max_resamples,
- processes=1, # Default to 1, compound generation is fast
- min_cpds_per_core=10,
+ processes=config.processes,
+ min_cpds_per_core=config.min_cpds_per_core,
product_library_file=config.product_library_file,
use_boltzmann_weighting=config.use_boltzmann_weighting,
seed=seed,
track_diagnostics=config.track_diagnostics,
)
- # Set up sampler
- sampler.set_evaluator(evaluator)
+ # Set up sampler. The evaluator config is passed through so that, when
+ # processes > 1, each worker can construct its own evaluator instead of
+ # receiving an unpicklable one over the pipe.
+ sampler.set_evaluator(evaluator, evaluator_config=config.evaluator_config)
sampler.read_reagents(pipeline.reagent_file_list)
# Auto-detect SMARTS compatibility if enabled
@@ -281,14 +287,69 @@ def get_num_prods(self) -> int:
"""Get the total number of possible products"""
return self.num_prods
- def set_evaluator(self, evaluator):
+ def __getstate__(self):
+ """Support pickling so workers can be initialized with this sampler.
+
+ Three attributes are dropped rather than pickled:
+
+ - ``evaluator``: may wrap a SWIG object (OpenEye ``OEDock``, ROCS shape
+ engine) that raises TypeError on pickle. Each worker rebuilds its own
+ from ``evaluator_config`` in ``parallel_evaluator._init_worker``.
+ - ``parallel_evaluator``: owns a ``multiprocessing.Pool``, which cannot
+ be pickled and must not be inherited by a worker (that would nest
+ pools recursively).
+ - ``logger``: a ``logging.Logger`` holding file handles and locks;
+ rebuilt from ``_log_filename`` in ``__setstate__``.
+ """
+ state = self.__dict__.copy()
+ state["evaluator"] = None
+ state["parallel_evaluator"] = None
+ state["logger"] = None
+ return state
+
+ def __setstate__(self, state):
+ """Restore a pickled sampler inside a worker process."""
+ self.__dict__.update(state)
+ # Rebuild the dropped logger. The evaluator is installed separately by
+ # _init_worker; parallel_evaluator stays None because a worker never
+ # spawns its own pool.
+ if self.logger is None:
+ self.logger = get_logger(
+ __name__, filename=self.__dict__.get("_log_filename")
+ )
+
+ def set_evaluator(self, evaluator, evaluator_config=None):
"""
Define the evaluator.
Automatically disables multiprocessing for fast evaluators (LookupEvaluator, DBEvaluator)
where pickle overhead exceeds evaluation time.
+
+ Args:
+ evaluator: The evaluator instance used to score compounds.
+ evaluator_config: Optional picklable config (Pydantic model) that can
+ recreate ``evaluator``. Required for ``processes > 1`` with
+ OpenEye-backed evaluators, whose SWIG objects cannot be pickled:
+ each worker builds its own evaluator from this config instead of
+ receiving one over the pipe. ``from_config`` supplies it
+ automatically.
"""
self.evaluator = evaluator
+ self.evaluator_config = evaluator_config
+
+ # Give workers the recipe they need to construct their own evaluator.
+ if self.parallel_evaluator is not None:
+ self.parallel_evaluator.bind_worker_context(self, evaluator_config)
+
+ if self.processes > 1 and evaluator_config is None:
+ self.logger.warning(
+ "processes=%d but no evaluator_config was provided. Workers "
+ "cannot rebuild the evaluator, so it must be picklable. "
+ "OpenEye-backed evaluators (Fred, ROCS) will fail; use "
+ "ThompsonSampler.from_config() or pass evaluator_config=... "
+ "to set_evaluator().",
+ self.processes,
+ )
# Auto-detect fast evaluators and warn about multiprocessing inefficiency
if self.processes > 1:
diff --git a/src/TACTICS/thompson_sampling/factories.py b/src/TACTICS/thompson_sampling/factories.py
index 54ee07d..508093a 100644
--- a/src/TACTICS/thompson_sampling/factories.py
+++ b/src/TACTICS/thompson_sampling/factories.py
@@ -231,6 +231,7 @@ def create_evaluator(config: EvaluatorConfig) -> Evaluator:
"ref_filename": config.ref_filename,
"compound_col": config.compound_col,
"score_col": config.score_col,
+ "default_score": config.default_score,
}
return LookupEvaluator(json.dumps(input_dict))
diff --git a/src/TACTICS/thompson_sampling/strategies/config.py b/src/TACTICS/thompson_sampling/strategies/config.py
index 028dda4..69a9324 100644
--- a/src/TACTICS/thompson_sampling/strategies/config.py
+++ b/src/TACTICS/thompson_sampling/strategies/config.py
@@ -208,7 +208,14 @@ class TopTwoConfig(BaseModel):
min_observations: int = Field(
default=5,
gt=0,
- description="Minimum observations per reagent before trusting GMIC criticality",
+ description=(
+ "DEPRECATED and inert. Formerly gated GMIC to 0.0 until every "
+ "active reagent had this many observations. The gate was removed "
+ "for consistency with RouletteWheelSelection, which never gated: "
+ "on large components a single under-observed reagent pinned the "
+ "whole component's GMIC to zero. Accepted only for backward "
+ "compatibility with existing configs."
+ ),
)
# Adaptive thermal cycling parameters
diff --git a/src/TACTICS/thompson_sampling/strategies/roulette_wheel.py b/src/TACTICS/thompson_sampling/strategies/roulette_wheel.py
index cddab6b..6ed0fd5 100644
--- a/src/TACTICS/thompson_sampling/strategies/roulette_wheel.py
+++ b/src/TACTICS/thompson_sampling/strategies/roulette_wheel.py
@@ -425,7 +425,13 @@ def select_reagent(self, reagent_list, disallow_mask=None, **kwargs):
# All scores identical, use uniform distribution
probs = np.ones(len(reagent_list)) / len(reagent_list)
else:
- scores = np.exp((scores - np.mean(scores)) / score_std / effective_temp)
+ # Numerically stable softmax: subtract the max of the exponent
+ # argument (shift-invariant, so probabilities are identical) to
+ # avoid exp() overflow when the sampled-score spread is large
+ # relative to a small CATS temperature — as on the heavy-tailed,
+ # zero-inflated read-count landscape.
+ z = (scores - np.mean(scores)) / score_std / effective_temp
+ scores = np.exp(z - np.max(z))
# Normalize to probabilities
probs = scores / np.sum(scores)
@@ -485,7 +491,13 @@ def select_batch(self, reagent_list, batch_size, disallow_mask=None, **kwargs):
# All scores identical, use uniform distribution
probs = np.ones(len(reagent_list)) / len(reagent_list)
else:
- scores = np.exp((scores - np.mean(scores)) / score_std / effective_temp)
+ # Numerically stable softmax: subtract the max of the exponent
+ # argument (shift-invariant, so probabilities are identical) to
+ # avoid exp() overflow when the sampled-score spread is large
+ # relative to a small CATS temperature — as on the heavy-tailed,
+ # zero-inflated read-count landscape.
+ z = (scores - np.mean(scores)) / score_std / effective_temp
+ scores = np.exp(z - np.max(z))
# Normalize to probabilities
probs = scores / np.sum(scores)
diff --git a/src/TACTICS/thompson_sampling/strategies/top_two_selection.py b/src/TACTICS/thompson_sampling/strategies/top_two_selection.py
index 9e490c1..5ce2290 100644
--- a/src/TACTICS/thompson_sampling/strategies/top_two_selection.py
+++ b/src/TACTICS/thompson_sampling/strategies/top_two_selection.py
@@ -44,7 +44,11 @@ class TopTwoSelection(SelectionStrategy):
>1 inflates uncertainty → more TT-TS disagreement → exploration.
cooled_scale: Multiplier on posterior std for cooled components.
<1 deflates uncertainty → more TT-TS agreement → exploitation.
- min_observations: Minimum observations before trusting GMIC.
+ min_observations: DEPRECATED and inert. Formerly gated GMIC to 0.0
+ until every active reagent had this many observations; that gate
+ was removed (see _calculate_gmic) because a single under-observed
+ reagent could pin a whole component's GMIC to zero. Accepted and
+ stored only for backward compatibility with existing configs.
"""
def __init__(
@@ -287,15 +291,24 @@ def _calculate_gmic(self, reagent_list: List) -> float:
or selection.
Returns:
- GMIC value >= 0. Returns 0.0 if insufficient data.
+ GMIC value >= 0. Returns 0.0 only when fewer than two reagents
+ have been observed.
"""
active = [r for r in reagent_list if r.n_samples > 0]
if len(active) < 2:
return 0.0
- if min(r.n_samples for r in active) < self.min_observations:
- return 0.0
-
+ # NOTE (2026-06): a min-observation gate used to live here — it returned
+ # 0.0 whenever the least-observed active reagent had fewer than
+ # self.min_observations samples. It was REMOVED for consistency with
+ # RouletteWheelSelection._calculate_gmic, which never gated. On large
+ # components (e.g. adenine's 688 isocyanides) a single sub-5-obs
+ # straggler forced the whole component's GMIC to 0 every cycle —
+ # firing on 25/28 benchmark libraries — which over-weighted that
+ # component in the GMIC rotation. A paired counterfactual showed
+ # removing the gate lifts adenine TT-TS top-100 recovery 87.1 -> 93.2
+ # and halves its replicate variance (sd 18.4 -> 10.6), with no change
+ # on the 3 libraries where the gate never fired.
means = np.array([r.mean for r in active])
signal = np.var(means)
noise = np.mean([r.std ** 2 for r in active])
diff --git a/tests/test_evaluator_integration.py b/tests/test_evaluator_integration.py
index fbc4d9d..fa67b0e 100644
--- a/tests/test_evaluator_integration.py
+++ b/tests/test_evaluator_integration.py
@@ -246,3 +246,83 @@ def evaluate(self, mol):
assert evaluator.counter == 1
sampler.close()
+
+
+class TestLookupEvaluatorDefaultScore:
+ """Test LookupEvaluator handling of product codes missing from the table.
+
+ Sparse lookup libraries (e.g. DEL read counts) only measure a small
+ fraction of the combinatorial space. There, an absent combination is a
+ true non-binder (score 0), not missing data. `default_score` makes that
+ distinction configurable while preserving the historical NaN default.
+ """
+
+ def setup_method(self):
+ self.temp_dir = tempfile.mkdtemp()
+ self.lookup_file = os.path.join(self.temp_dir, "sparse_scores.csv")
+ with open(self.lookup_file, "w") as f:
+ f.write("Product_Code,Scores\n")
+ f.write("measured_a,12.5\n")
+ f.write("measured_b,3.0\n")
+
+ def teardown_method(self):
+ shutil.rmtree(self.temp_dir)
+
+ def test_missing_product_returns_nan_by_default(self):
+ """Without default_score, absent products keep the historical NaN."""
+ evaluator = LookupEvaluator({"ref_filename": self.lookup_file})
+
+ assert np.isnan(evaluator.evaluate("never_measured"))
+ assert evaluator.evaluate("measured_a") == 12.5
+
+ def test_missing_product_returns_configured_default(self):
+ """With default_score=0.0, absent products score as true non-binders."""
+ evaluator = LookupEvaluator(
+ {"ref_filename": self.lookup_file, "default_score": 0.0}
+ )
+
+ assert evaluator.evaluate("never_measured") == 0.0
+ assert evaluator.evaluate("measured_a") == 12.5
+
+ def test_default_score_does_not_mask_measured_zeros(self):
+ """A measured score of 0.0 is returned from the table, not the default."""
+ zero_file = os.path.join(self.temp_dir, "with_zero.csv")
+ with open(zero_file, "w") as f:
+ f.write("Product_Code,Scores\n")
+ f.write("measured_zero,0.0\n")
+
+ evaluator = LookupEvaluator(
+ {"ref_filename": zero_file, "default_score": -99.0}
+ )
+
+ # Present-with-zero must not be confused with absent.
+ assert evaluator.evaluate("measured_zero") == 0.0
+ assert evaluator.evaluate("absent") == -99.0
+
+ def test_default_score_flows_through_config_and_factory(self):
+ """LookupEvaluatorConfig -> create_evaluator wires default_score through."""
+ from TACTICS.thompson_sampling.core.evaluator_config import (
+ LookupEvaluatorConfig,
+ )
+ from TACTICS.thompson_sampling.factories import create_evaluator
+
+ config = LookupEvaluatorConfig(
+ ref_filename=self.lookup_file, default_score=0.0
+ )
+ evaluator = create_evaluator(config)
+
+ assert evaluator.default_score == 0.0
+ assert evaluator.evaluate("never_measured") == 0.0
+
+ def test_config_default_is_none_preserving_nan_behavior(self):
+ """The config default must stay None so existing runs are unchanged."""
+ from TACTICS.thompson_sampling.core.evaluator_config import (
+ LookupEvaluatorConfig,
+ )
+ from TACTICS.thompson_sampling.factories import create_evaluator
+
+ config = LookupEvaluatorConfig(ref_filename=self.lookup_file)
+ evaluator = create_evaluator(config)
+
+ assert config.default_score is None
+ assert np.isnan(evaluator.evaluate("never_measured"))
diff --git a/tests/test_parallel_evaluation.py b/tests/test_parallel_evaluation.py
new file mode 100644
index 0000000..c5e8883
--- /dev/null
+++ b/tests/test_parallel_evaluation.py
@@ -0,0 +1,298 @@
+"""
+Tests for parallel batch evaluation (bug report 2026-06-23).
+
+Covers three defects that together made `processes > 1` unusable with slow
+evaluators such as Fred docking:
+
+1. `from_config()` hardcoded `processes=1`, so evaluation parallelism could
+ not be enabled through the config API at all.
+2. `pool.map(sampler.evaluate, ...)` pickled the sampler, and therefore the
+ evaluator, and therefore any SWIG-wrapped OpenEye object it held --
+ raising `TypeError: cannot pickle 'SwigPyObject' object` on the first batch.
+3. Importing TACTICS loaded the OpenEye toolkits at module level, which could
+ segfault the interpreter via a libexpat conflict with prompt_toolkit.
+
+Note on start methods: multiprocessing under `spawn` (the macOS/Windows
+default) re-imports the entry-point module in each worker, so these tests must
+run from an importable module -- which pytest provides.
+"""
+import os
+import pickle
+import sys
+import tempfile
+import shutil
+
+import numpy as np
+import pytest
+
+from TACTICS.library_enumeration import SynthesisPipeline, ReactionConfig, ReactionDef
+from TACTICS.thompson_sampling import ThompsonSamplingConfig, ThompsonSampler
+from TACTICS.thompson_sampling.core.evaluator_config import LookupEvaluatorConfig
+from TACTICS.thompson_sampling.core.parallel_evaluator import ParallelEvaluator
+from TACTICS.thompson_sampling.strategies.config import GreedyConfig
+
+REACTION_SMARTS = "[C:1](=O)[OH].[N:2]>>[C:1](=O)[N:2]"
+ALL_COMBOS = [[0, 0], [0, 1], [1, 0], [1, 1]]
+
+
+class _Unpicklable:
+ """Stands in for a SWIG-wrapped OpenEye object (OEDock, ROCS shape)."""
+
+ def __reduce__(self):
+ raise TypeError("cannot pickle 'SwigPyObject' object")
+
+
+class TestParallelEvaluationConfig:
+ """Bug 1: `processes` must be reachable through the config API."""
+
+ def setup_method(self):
+ self.temp_dir = tempfile.mkdtemp()
+ self.config = _make_config(self.temp_dir, processes=1)
+
+ def teardown_method(self):
+ shutil.rmtree(self.temp_dir)
+
+ def test_config_exposes_processes_with_safe_default(self):
+ assert self.config.processes == 1
+ assert self.config.min_cpds_per_core == 10
+
+ def test_processes_must_be_positive(self):
+ with pytest.raises(Exception):
+ _make_config(self.temp_dir, processes=0)
+
+ def test_from_config_propagates_processes(self):
+ """The regression: from_config used to hardcode processes=1."""
+ config = _make_config(self.temp_dir, processes=4)
+ sampler = ThompsonSampler.from_config(config)
+
+ assert sampler.processes == 4, (
+ "from_config ignored config.processes -- evaluation would run "
+ "single-threaded regardless of the allocated cores"
+ )
+ sampler.close()
+
+ def test_from_config_propagates_min_cpds_per_core(self):
+ config = _make_config(self.temp_dir, processes=2, min_cpds_per_core=3)
+ sampler = ThompsonSampler.from_config(config)
+
+ assert sampler.min_cpds_per_core == 3
+ sampler.close()
+
+ def test_from_config_binds_evaluator_config_for_workers(self):
+ """Workers need the config to rebuild their own evaluator."""
+ config = _make_config(self.temp_dir, processes=2)
+ sampler = ThompsonSampler.from_config(config)
+
+ assert sampler.evaluator_config is not None
+ assert sampler.parallel_evaluator._worker_context is not None
+ sampler.close()
+
+
+class TestSamplerPickling:
+ """Bug 2, part 1: the sampler must survive being sent to a worker."""
+
+ def setup_method(self):
+ self.temp_dir = tempfile.mkdtemp()
+
+ def teardown_method(self):
+ shutil.rmtree(self.temp_dir)
+
+ def test_sampler_pickles_without_evaluator_pool_or_logger(self):
+ config = _make_config(self.temp_dir, processes=2)
+ sampler = ThompsonSampler.from_config(config)
+
+ restored = pickle.loads(pickle.dumps(sampler))
+
+ # Dropped on the way out...
+ assert restored.evaluator is None, "evaluator must not be pickled"
+ assert restored.parallel_evaluator is None, "pool must not be pickled"
+ # ...and the logger is rebuilt on the way in.
+ assert restored.logger is not None
+ # Search state that workers do need is preserved.
+ assert len(restored.reagent_lists) == len(sampler.reagent_lists)
+ sampler.close()
+
+ def test_sampler_pickles_even_with_unpicklable_evaluator(self):
+ """A SWIG-holding evaluator must not block sending the sampler."""
+ config = _make_config(self.temp_dir, processes=2)
+ sampler = ThompsonSampler.from_config(config)
+ sampler.evaluator.swig_handle = _Unpicklable()
+
+ # Would raise TypeError if __getstate__ did not drop the evaluator.
+ restored = pickle.loads(pickle.dumps(sampler))
+
+ assert restored.evaluator is None
+ sampler.close()
+
+
+class TestParallelEvaluationCorrectness:
+ """Bug 2, part 2: parallel evaluation must run, and agree with sequential."""
+
+ def setup_method(self):
+ self.temp_dir = tempfile.mkdtemp()
+
+ def teardown_method(self):
+ shutil.rmtree(self.temp_dir)
+
+ def _evaluate(self, processes):
+ config = _make_config(self.temp_dir, processes=processes,
+ min_cpds_per_core=1)
+ sampler = ThompsonSampler.from_config(config)
+ try:
+ return sampler.parallel_evaluator.evaluate_batch(
+ sampler.evaluate, ALL_COMBOS)
+ finally:
+ sampler.close()
+
+ def test_parallel_matches_sequential(self):
+ sequential = self._evaluate(processes=1)
+ parallel = self._evaluate(processes=2)
+
+ assert len(parallel) == len(ALL_COMBOS)
+ assert parallel == sequential, (
+ "parallel evaluation returned different results than sequential"
+ )
+
+ def test_empty_batch_short_circuits(self):
+ evaluator = ParallelEvaluator(processes=2)
+ # Must not create a pool for an empty batch.
+ assert evaluator.evaluate_batch(lambda c: c, []) == []
+ assert evaluator._pool is None
+
+
+class TestWorkerRebuildsOwnEvaluator:
+ """Bug 2, part 3: the evaluator is constructed in the worker, not shipped.
+
+ This is what makes OpenEye-backed evaluators usable with processes > 1:
+ OEDock never crosses the pickle boundary, so its unpicklability is
+ irrelevant. These tests use a plain unpicklable stand-in so they run
+ without an OpenEye licence.
+ """
+
+ def setup_method(self):
+ self.temp_dir = tempfile.mkdtemp()
+
+ def teardown_method(self):
+ shutil.rmtree(self.temp_dir)
+
+ def test_unpicklable_evaluator_still_evaluates_in_parallel(self):
+ config = _make_config(self.temp_dir, processes=2, min_cpds_per_core=1)
+ sampler = ThompsonSampler.from_config(config)
+
+ # Attach a SWIG stand-in. Under the old implementation pool.map would
+ # pickle the bound method -> sampler -> evaluator -> this object, and
+ # raise TypeError on the very first batch.
+ sampler.evaluator.swig_handle = _Unpicklable()
+
+ results = sampler.parallel_evaluator.evaluate_batch(
+ sampler.evaluate, ALL_COMBOS)
+
+ assert len(results) == len(ALL_COMBOS)
+ assert all(np.isfinite(score) for _, _, score in results)
+ sampler.close()
+
+ def test_binding_a_context_resets_an_existing_pool(self):
+ """Rebinding must not leave workers holding a stale evaluator."""
+ evaluator = ParallelEvaluator(processes=2)
+ evaluator._pool = _DummyPool()
+ dummy = evaluator._pool
+
+ evaluator.bind_worker_context(sampler=None, evaluator_config=None)
+
+ assert dummy.closed, "existing pool should be shut down on rebind"
+ assert evaluator._pool is None
+ assert evaluator._worker_context is None
+
+
+class TestOpenEyeImportIsDeferred:
+ """Bug 3: importing TACTICS must not load the OpenEye toolkits.
+
+ Loading OpenEye at module scope initialises libexpat; prompt_toolkit
+ (via tqdm) later re-enters expat from pyexpat at module scope, and the
+ conflict segfaults the interpreter during `import TACTICS`.
+ """
+
+ def test_importing_evaluators_does_not_import_openeye(self):
+ # Importing the module must leave the OpenEye globals unbound. This
+ # holds whether or not openeye is installed.
+ import TACTICS.thompson_sampling.core.evaluators as evaluators
+
+ source = open(evaluators.__file__).read()
+ assert "from openeye import" not in source.split("def _ensure_openeye")[0], (
+ "OpenEye must not be imported at module scope"
+ )
+
+ def test_ensure_openeye_is_idempotent_or_raises_cleanly(self):
+ import TACTICS.thompson_sampling.core.evaluators as evaluators
+
+ try:
+ evaluators._ensure_openeye()
+ except ImportError as exc:
+ # No licence/install: the error must name the package and be actionable.
+ assert "openeye" in str(exc).lower()
+ return
+
+ # Installed: modules are cached and a second call is a no-op.
+ assert evaluators.oechem is not None
+ first = evaluators.oechem
+ evaluators._ensure_openeye()
+ assert evaluators.oechem is first
+
+ def test_ml_classifier_does_not_depend_on_openeye_import(self):
+ """joblib used to be imported inside the OpenEye try/except."""
+ import TACTICS.thompson_sampling.core.evaluators as evaluators
+
+ source = open(evaluators.__file__).read()
+ header = source.split("class Evaluator")[0]
+ assert "import joblib" not in header, (
+ "joblib must not be bound to the OpenEye import block"
+ )
+
+
+class _DummyPool:
+ """Minimal stand-in recording that close/join were called."""
+
+ def __init__(self):
+ self.closed = False
+
+ def close(self):
+ self.closed = True
+
+ def join(self):
+ pass
+
+
+def _make_config(temp_dir, processes=1, min_cpds_per_core=None):
+ """Build a small 2x2 lookup-scored config in ``temp_dir``."""
+ acids = os.path.join(temp_dir, "acids.smi")
+ amines = os.path.join(temp_dir, "amines.smi")
+ scores = os.path.join(temp_dir, "scores.csv")
+
+ if not os.path.exists(acids):
+ with open(acids, "w") as f:
+ f.write("CC(=O)O\tacid1\nCCC(=O)O\tacid2\n")
+ with open(amines, "w") as f:
+ f.write("CN\tam1\nCCN\tam2\n")
+ with open(scores, "w") as f:
+ f.write("Product_Code,Scores\n")
+ f.write("acid1_am1,0.1\nacid1_am2,0.2\n")
+ f.write("acid2_am1,0.3\nacid2_am2,0.4\n")
+
+ pipeline = SynthesisPipeline(ReactionConfig(
+ reactions=[ReactionDef(reaction_smarts=REACTION_SMARTS, step_index=0)],
+ reagent_file_list=[acids, amines],
+ ))
+
+ kwargs = dict(
+ synthesis_pipeline=pipeline,
+ num_ts_iterations=2,
+ strategy_config=GreedyConfig(mode="maximize"),
+ evaluator_config=LookupEvaluatorConfig(ref_filename=scores),
+ processes=processes,
+ batch_size=2,
+ hide_progress=True,
+ )
+ if min_cpds_per_core is not None:
+ kwargs["min_cpds_per_core"] = min_cpds_per_core
+
+ return ThompsonSamplingConfig(**kwargs)
diff --git a/tests/test_strategy_integration.py b/tests/test_strategy_integration.py
index 10ab2ad..3534c3d 100644
--- a/tests/test_strategy_integration.py
+++ b/tests/test_strategy_integration.py
@@ -383,3 +383,101 @@ def test_roulette_wheel_weighted_rotation(self):
f"Flexible component (idx=1) should be heated more than "
f"critical component (idx=0): {counts[1]} vs {counts[0]}"
)
+
+
+class TestRouletteWheelSoftmaxNumerics:
+ """Regression tests for exp() overflow in the CATS Boltzmann softmax.
+
+ The Boltzmann weight is exp(z) where z = (score - mean) / std / temperature.
+ Because z is standardized, it is O(1) for a well-behaved score distribution.
+ But on a heavy-tailed, zero-inflated landscape (e.g. DEL read counts, where
+ most reagents score 0 and a few score very high) a single extreme outlier
+ can push max(z) to ~sqrt(n), and a small CATS temperature divides that
+ further. The naive exp(z) then overflows to inf, and inf/sum(inf) = nan,
+ which makes rng.choice raise "probabilities contain NaN".
+
+ Subtracting max(z) before exponentiating is shift-invariant -- the
+ normalized probabilities are mathematically identical -- but keeps every
+ exponent <= 0, so the largest weight is exp(0) = 1 and overflow is
+ impossible.
+ """
+
+ def _heavy_tailed_reagents(self, n=500):
+ """Zero-inflated reagents: most near zero, one extreme outlier."""
+ from TACTICS.thompson_sampling.core.reagent import Reagent
+
+ reagents = []
+ for i in range(n):
+ r = Reagent(f"r_{i}", "C")
+ r.mean = 0.0
+ r.std = 1e-6 # near-deterministic, so sampled scores ~= means
+ r.n_samples = 10
+ reagents.append(r)
+ # One dominant reagent far out in the tail.
+ reagents[0].mean = 1e6
+ return reagents
+
+ def test_select_reagent_survives_extreme_z_scores(self, monkeypatch):
+ """A tiny temperature on a heavy tail must not produce NaN probabilities."""
+ strategy = RouletteWheelSelection(mode="maximize")
+ reagents = self._heavy_tailed_reagents()
+
+ # Force a very small temperature; the naive exp() would overflow here.
+ monkeypatch.setattr(
+ strategy, "_get_component_temperature", lambda *a, **k: 1e-3
+ )
+
+ rng = np.random.default_rng(42)
+ idx = strategy.select_reagent(reagents, rng=rng, component_idx=0)
+
+ assert 0 <= idx < len(reagents)
+
+ def test_select_batch_survives_extreme_z_scores(self, monkeypatch):
+ """Same guarantee for the batch selection path."""
+ strategy = RouletteWheelSelection(mode="maximize")
+ reagents = self._heavy_tailed_reagents()
+
+ monkeypatch.setattr(
+ strategy, "_get_component_temperature", lambda *a, **k: 1e-3
+ )
+
+ rng = np.random.default_rng(42)
+ indices = strategy.select_batch(reagents, batch_size=10, rng=rng,
+ component_idx=0)
+
+ assert len(indices) == 10
+ assert all(0 <= i < len(reagents) for i in indices)
+
+ def test_max_subtraction_is_shift_invariant(self):
+ """The stable form gives identical probabilities where naive doesn't overflow."""
+ rng = np.random.default_rng(0)
+ scores = rng.normal(size=200) * 3.0 + 10.0
+ z = (scores - np.mean(scores)) / np.std(scores) / 0.5
+
+ naive = np.exp(z)
+ naive_probs = naive / np.sum(naive)
+
+ stable = np.exp(z - np.max(z))
+ stable_probs = stable / np.sum(stable)
+
+ assert np.isfinite(naive_probs).all() # guard: naive is valid here
+ np.testing.assert_allclose(stable_probs, naive_probs, rtol=1e-12)
+
+ def test_naive_softmax_would_overflow_on_this_input(self):
+ """Documents the failure mode the max-subtraction prevents."""
+ z = np.zeros(500)
+ z[0] = 1e4 # the standardized outlier / small-temperature case
+
+ with np.errstate(over="ignore"):
+ naive = np.exp(z)
+ naive_probs = naive / np.sum(naive)
+
+ # Naive overflows to inf and yields NaN after normalization.
+ assert np.isinf(naive).any()
+ assert np.isnan(naive_probs).any()
+
+ # Stable form stays finite and normalized.
+ stable = np.exp(z - np.max(z))
+ stable_probs = stable / np.sum(stable)
+ assert np.isfinite(stable_probs).all()
+ np.testing.assert_allclose(np.sum(stable_probs), 1.0)
diff --git a/tutorials/diagnostic_benchmark_plots.py b/tutorials/diagnostic_benchmark_plots.py
new file mode 100644
index 0000000..dd71fe9
--- /dev/null
+++ b/tutorials/diagnostic_benchmark_plots.py
@@ -0,0 +1,812 @@
+"""
+TACTICS Diagnostic Benchmark Plots
+
+Visualises the per-query GMIC mechanism from the diagnostic benchmark
+(110 trials, 5 libraries, 11 queries, 5 replicates).
+
+Figure types:
+ - **RWS Diagnostic**: GMIC trajectory + effective temperature + divergence gate
+ - **TT-TS Diagnostic**: disagreement EMA + heated_scale adaptation
+ - **Layers 1 & 2**: GMIC component-routing + within-component intensity,
+ with per-library dropdowns for docking and ROCS landscapes
+
+Run as app: marimo run tutorials/diagnostic_benchmark_plots.py
+Edit mode: marimo edit tutorials/diagnostic_benchmark_plots.py
+"""
+
+import marimo
+
+__generated_with = "0.21.1"
+app = marimo.App(width="full", app_title="Diagnostic Benchmark Plots")
+
+
+@app.cell(hide_code=True)
+def _(mo):
+ mo.md(r"""
+ # Diagnostic Plots and Mechanism Plots
+ This notebook is designed to showcase the inner workings of TACTICS methods.
+ """)
+ return
+
+
+@app.cell(hide_code=True)
+def _():
+ import marimo as mo
+ import sys
+ from pathlib import Path
+
+ import matplotlib
+ matplotlib.rcParams.update({
+ "figure.facecolor": "white",
+ "axes.facecolor": "white",
+ "axes.edgecolor": "black",
+ "axes.labelcolor": "black",
+ "xtick.color": "black",
+ "ytick.color": "black",
+ "text.color": "black",
+ "savefig.facecolor": "white",
+ })
+ import matplotlib.pyplot as plt
+ import polars as pl
+
+ try:
+ project_root = Path(__file__).parent.parent.resolve()
+ except NameError:
+ project_root = Path.cwd()
+
+ sys.path.insert(0, str(project_root / "src"))
+
+ from TACTICS.library_analysis.diagnostic_plots import (
+ plot_reagent_usage_action_panel,
+ plot_rws_diagnostic,
+ plot_ttts_diagnostic,
+ )
+
+ diag_dir = project_root / "outputs" / "diagnostic_benchmark" / "diagnostics"
+ search_dir = project_root / "outputs" / "diagnostic_benchmark" / "search"
+ scores_dir = project_root / "data" / "scores"
+ reagents_dir = project_root / "data" / "reagents"
+ plot_dir = project_root / "plots"
+ plot_dir.mkdir(exist_ok=True)
+ return (
+ diag_dir,
+ mo,
+ pl,
+ plot_dir,
+ plot_reagent_usage_action_panel,
+ plot_rws_diagnostic,
+ plot_ttts_diagnostic,
+ plt,
+ reagents_dir,
+ search_dir,
+ )
+
+
+@app.cell
+def _(diag_dir, mo, pl):
+ """Load diagnostic data and build metadata index."""
+ bench_data = {"libraries": {}, "meta": {}}
+
+ for _lib_dir in sorted(diag_dir.iterdir()):
+ if not _lib_dir.is_dir():
+ continue
+ for _f in _lib_dir.iterdir():
+ if _f.suffix == ".parquet":
+ bench_data["libraries"][_lib_dir.name] = pl.read_parquet(_f)
+ break
+
+ bench_data["meta"] = {
+ "rxn206": {"gain": 6.1, "scoring": "ROCS", "n_comps": 2},
+ "mannich": {"gain": 9.1, "scoring": "ROCS", "n_comps": 3},
+ "orru": {"gain": 1.7, "scoring": "ROCS", "n_comps": 3},
+ "adenine": {"gain": 64.6, "scoring": "docking", "n_comps": 3},
+ "quinazoline": {"gain": -0.9, "scoring": "docking", "n_comps": 3},
+ }
+
+ mo.md(
+ f"**Loaded {len(bench_data['libraries'])} libraries**: "
+ f"{', '.join(bench_data['libraries'].keys())}\n\n"
+ + "\n".join(
+ f"- **{_name}**: {_df.shape[0]} rows, "
+ f"{_df['component_idx'].n_unique()} components, "
+ f"{_df['query_id'].n_unique()} queries, "
+ f"{_df['replicate'].n_unique()} replicates"
+ for _name, _df in bench_data["libraries"].items()
+ )
+ )
+ return (bench_data,)
+
+
+@app.cell
+def _(mo):
+ mo.md("""
+ # RWS Diagnostic: GMIC + Temperature + Divergence Gate
+ """)
+ return
+
+
+@app.cell
+def _(bench_data, plot_dir, plot_rws_diagnostic, plt):
+ for _lib, _df in bench_data["libraries"].items():
+ _rws = _df.filter(_df["method"].str.contains("RWS"))
+ if len(_rws) == 0:
+ continue
+ for _q in _rws["query_id"].unique().sort().to_list():
+ _q_data = _rws.filter(_rws["query_id"] == _q)
+ _fig = plot_rws_diagnostic(_q_data, title=f"{_lib} — {_q} (RWS)")
+ _fig.savefig(plot_dir / f"rws_{_lib}_{_q}.pdf", bbox_inches="tight")
+ plt.show()
+ plt.close(_fig)
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md("""
+ # TT-TS Diagnostic: Disagreement EMA + Heated Scale
+ """)
+ return
+
+
+@app.cell
+def _(bench_data, plot_dir, plot_ttts_diagnostic, plt):
+ for _lib, _df in bench_data["libraries"].items():
+ _ttts = _df.filter(_df["method"].str.contains("TT-TS"))
+ if len(_ttts) == 0:
+ continue
+ for _q in _ttts["query_id"].unique().sort().to_list():
+ _q_data = _ttts.filter(_ttts["query_id"] == _q)
+ _fig = plot_ttts_diagnostic(_q_data, title=f"{_lib} — {_q} (TT-TS)")
+ _fig.savefig(plot_dir / f"ttts_{_lib}_{_q}.pdf", bbox_inches="tight")
+ plt.show()
+ plt.close(_fig)
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md("""
+ # Action Panel: Per-Batch Reagent Diversity + Heated Component
+
+ Shows the GMIC/TT-TS rotation mechanism in action batch by batch.
+
+ - **Grouped bars**: one bar per component per batch. Bar height
+ = fraction of that component's reagents sampled in the batch.
+ Each bar is split into a *revisit* segment (bottom, dimmer) and
+ a *new* segment (top, brighter) — "new" means first-time-seen
+ this search.
+ - **Heated encoding**: the component marked ``is_heated`` that
+ batch is rendered at full opacity; cooled components' bars are
+ muted. Mechanism working → heated bars stand out batch by
+ batch.
+ """)
+ return
+
+
+@app.cell
+def _(
+ bench_data,
+ pl,
+ plot_dir,
+ plot_reagent_usage_action_panel,
+ plt,
+ reagents_dir,
+ search_dir,
+):
+ _FOLDER_MAP = {
+ "amide-suzuki": "amide_suzuki",
+ "groebke-blackburn-bienayme": "groebke",
+ }
+ _WINDOW = {
+ "mannich": 1, "orru": 1, "rxn206": 1,
+ "adenine": 1, "quinazoline": 1,
+ }
+ _FIGSIZE = {
+ "mannich": (16, 5.5), "orru": (16, 5.5), "rxn206": (16, 5.5),
+ "adenine": (16, 5.5), "quinazoline": (16, 5.5),
+ }
+ # Imbalanced libraries — normalize per batch (unique / min(batch, n_reagents))
+ # so bar heights are comparable across differently-sized components.
+ _NORMALIZE = {
+ "mannich": "library", "orru": "library", "rxn206": "library",
+ "adenine": "batch", "quinazoline": "batch",
+ }
+ # Component labels derived from reaction SMARTS in data/reactions.csv.
+ # Order matches reagent_0, reagent_1, ... Adenine uses auto-inferred prefix
+ # names (amidine/isocyanide/aldehyde) and is left to the default path.
+ _COMPONENT_NAMES = {
+ "mannich": ["Amine", "Aldehyde", "Ketone"],
+ "orru": ["Aldehyde", "Cyanoester", "Amine"],
+ "rxn206": ["Alcohol", "Sulfonamide"],
+ "quinazoline": ["Aminobenzoic acid", "Amine", "Carboxylic acid"],
+ }
+
+ # Quinazoline has too many batches (20M library, ~1000+ batches) for the
+ # action-panel diagnostic to be legible — skip it here.
+ _SKIP_LIBS = {"quinazoline"}
+
+ for _lib, _df in bench_data["libraries"].items():
+ if _lib in _SKIP_LIBS:
+ continue
+ _folder = _FOLDER_MAP.get(_lib, _lib)
+ _rfiles = sorted((reagents_dir / _folder).glob(f"{_folder}_reagent_*.smi"))
+ if not _rfiles:
+ _rfiles = sorted((reagents_dir / _folder).glob(f"{_lib}_reagent_*.smi"))
+ if not _rfiles:
+ continue
+
+ _search_path = search_dir / _lib / f"{_lib}.parquet"
+ if not _search_path.exists():
+ continue
+ _search_df = pl.read_parquet(_search_path)
+ _win = _WINDOW.get(_lib, 1)
+
+ for _method_pat, _method_tag in [("RWS", "rws"), ("TT-TS", "ttts")]:
+ _m_df = _df.filter(_df["method"].str.contains(_method_pat))
+ if len(_m_df) == 0:
+ continue
+ for _q in _m_df["query_id"].unique().sort().to_list():
+ _q_diag = _m_df.filter(
+ (_m_df["query_id"] == _q) & (pl.col("replicate") == 0)
+ )
+ _q_search = _search_df.filter(
+ (pl.col("query_id") == _q)
+ & (pl.col("method").str.contains(_method_pat))
+ & (pl.col("replicate") == 0)
+ )
+ if len(_q_search) == 0 or len(_q_diag) == 0:
+ continue
+
+ _heated_label = "Explore" if _method_pat == "TT-TS" else "Heated"
+ _fig = plot_reagent_usage_action_panel(
+ _q_search, _q_diag,
+ reagent_files=_rfiles,
+ batch_size=100,
+ aggregate_window=_win,
+ normalize=_NORMALIZE.get(_lib, "library"),
+ figsize=_FIGSIZE.get(_lib, (16, 5)),
+ component_names=_COMPONENT_NAMES.get(_lib),
+ title=f"{_lib} — {_q} — {_method_pat} (rep 0)",
+ heated_label=_heated_label,
+ )
+ _fig.savefig(
+ plot_dir / f"action_{_lib}_{_q}_{_method_tag}.pdf",
+ bbox_inches="tight",
+ )
+ plt.show()
+ plt.close(_fig)
+ return
+
+
+@app.cell
+def _():
+ """Shared deps for the MECH cells: numpy, a fig→PDF-bytes helper (this
+ notebook's original import cell defines neither), the canonical-benchmark
+ paths, and the heterogeneous-schema diagnostics loader.
+
+ These are kept separate from the original import cell so none of the
+ existing cells are modified.
+ """
+ import io as _io
+ from pathlib import Path as _Path
+
+ import numpy as np
+
+ def fig_to_pdf_bytes(fig):
+ """Convert a matplotlib figure to PDF bytes in memory (tight bbox)."""
+ _buf = _io.BytesIO()
+ fig.savefig(_buf, format="pdf", bbox_inches="tight", dpi=300)
+ return _buf.getvalue()
+
+ try:
+ _root = _Path(__file__).parent.parent.resolve()
+ except NameError:
+ _root = _Path.cwd()
+
+ # Diagnostics: the gate-fixed 2% TT-TS GMIC/disagreement trajectories live in
+ # full_benchmark_nogate (merged per-library, all 28 libs incl. hybrids).
+ # ema_cats' adenine TT-TS diagnostics are STALE (pre-merge, gated) and were
+ # never regenerated. Recovery stays on the canonical ema_cats: its TT-TS is
+ # the spliced nogate TT-TS (identical), only non-TT-TS RNG differs.
+ mech_bench = _root / "outputs" / "full_benchmark_nogate"
+ mech_diag_dir = mech_bench / "diagnostics"
+ mech_recovery_path = (
+ _root / "outputs" / "full_benchmark_ema_cats" / "analysis" / "recovery_summary.parquet"
+ )
+ return fig_to_pdf_bytes, mech_diag_dir, np
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Layers 1 & 2 — Exploration Mechanism for Any Docking Library
+
+ Pick a docking library from the dropdown; **both** mechanism layers below
+ update to that library.
+
+ - **Layer 1 — *which* component?** Signal → Decision → Result. GMIC reads
+ each component as resolved (high GMIC → exploit) or unresolved (low GMIC →
+ explore); the *explored* strip follows GMIC, and **cumulative coverage**
+ (running total of reagents sampled) shows the consequence.
+ - **Layer 2 — *how hard* within that component?** Intensity → Decision →
+ Result, one level down. Both knobs are **dimensionless ×-multipliers
+ centred on 1.0** (1.0 = no adjustment; >1 = amplify exploration): **RWS**
+ tunes the **CATS temperature multiplier** (`cats_multiplier`, ×base
+ temperature, GMIC-driven); **TT-TS** tunes the **σ-inflation multiplier**
+ (`heated_scale`, ×posterior σ, disagreement-driven). The Result row is
+ **new reagents per batch** — the *per-batch increment* of coverage (it falls
+ to zero as a component's reagent pool is exhausted, even while coverage is
+ still climbing).
+
+ **Data source (per library, labelled on each figure):** the **2% canonical**
+ benchmark (`full_benchmark_ema_cats`) — the manuscript operating point — is
+ used for the 4 base docking libraries (thrombin, amide, adenine, quinazoline).
+ The 4 docking *hybrids* have no 2% diagnostics saved (search only), so they
+ currently fall back to the **5%** sweep (`full_benchmark_5pct`) — flagged in
+ the title — until 2% diagnostics are regenerated. Replicate 0. Reagent files
+ resolve via the benchmark folder convention (hybrids reuse the base library's
+ reagents).
+ """)
+ return
+
+
+@app.cell
+def _(mo):
+ """Dropdown selector over the 8 docking libraries."""
+ _dock_libs = [
+ "thrombin", "thrombin_hybrid", "amide", "amide_hybrid",
+ "adenine", "adenine_hybrid", "quinazoline", "quinazoline_hybrid",
+ ]
+ dock_lib_dd = mo.ui.dropdown(
+ options=_dock_libs, value="thrombin", label="Docking library"
+ )
+ dock_lib_dd
+ return (dock_lib_dd,)
+
+
+@app.cell
+def _(dock_lib_dd, fig_to_pdf_bytes, mo, pl, plt, reagents_dir):
+ """Render both layer plots for the selected docking library."""
+ import glob as _glob3
+ from pathlib import Path as _Path
+
+ from TACTICS.library_analysis.diagnostic_plots import (
+ plot_adaptive_intensity as _ai,
+ plot_gmic_directed_exploration as _gde,
+ )
+
+ _lib = dock_lib_dd.value
+ # Reagent-folder convention (Local Norm #11): hybrids reuse the base folder.
+ _FOLDER = {
+ "adenine_hybrid": "adenine", "thrombin_hybrid": "thrombin",
+ "amide_hybrid": "amide", "quinazoline_hybrid": "quinazoline",
+ }
+ # Friendly component names where reagent IDs are numeric (else auto-inferred).
+ _NAMES = {
+ "thrombin": ["Acids", "Dipeptides"],
+ "thrombin_hybrid": ["Acids", "Dipeptides"],
+ "amide": ["Acids", "Amines"],
+ "amide_hybrid": ["Acids", "Amines"],
+ "quinazoline": ["Aminobenzoic acid", "Amine", "Carboxylic acid"],
+ "quinazoline_hybrid": ["Aminobenzoic acid", "Amine", "Carboxylic acid"],
+ "adenine": ["Amidine", "Isocyanide", "Aldehyde"],
+ "adenine_hybrid": ["Amidine", "Isocyanide", "Aldehyde"],
+ }
+ _folder = _FOLDER.get(_lib, _lib)
+ _key = _folder if _lib.endswith("_hybrid") else _lib
+ _rf = sorted(
+ _glob3.glob(str(reagents_dir / _folder / f"{_key}_reagent_*.smi"))
+ )
+
+ # Budget resolution: prefer any 2% source — the CANONICAL benchmark
+ # (full_benchmark_ema_cats, base docking libs) or the regenerated hybrid
+ # diagnostics (hybrid_diag_2pct, docking hybrids) — both the manuscript 2%
+ # operating point. Fall back to the 5% sweep only if no 2% diagnostics
+ # exist yet for this library.
+ _outputs = reagents_dir.parent.parent / "outputs"
+
+ def _parqs(_d):
+ # Glob .parquet but skip hidden/AppleDouble (._*) sidecar files.
+ return [
+ f for f in sorted(_glob3.glob(str(_d / "*.parquet")))
+ if not _Path(f).name.startswith(".")
+ ]
+
+ def _shards(_bdir):
+ return _parqs(_bdir / "diagnostics" / _lib)
+
+ _candidates = [
+ (_outputs / "full_benchmark_nogate", "2% gate-fixed"),
+ (_outputs / "full_benchmark_ema_cats", "2% canonical (gated TT-TS — fallback)"),
+ (_outputs / "hybrid_diag_2pct", "2% canonical"),
+ (_outputs / "full_benchmark_5pct", "5% (2% diagnostics not yet generated)"),
+ ]
+ _bench, _budget = next(
+ ((b, lbl) for b, lbl in _candidates if _shards(b)), (None, None)
+ )
+ _dfiles = _shards(_bench) if _bench else []
+ _sfiles = _parqs(_bench / "search" / _lib) if _bench else []
+
+ if not _dfiles or not _sfiles or not _rf:
+ _out3 = mo.md(f"⚠️ diagnostics/search/reagents not found for **{_lib}**.")
+ else:
+ # ema_cats shards are per-method heterogeneous-schema; union-concat them
+ # (null-fill missing columns) so RWS + TT-TS rows coexist.
+ _cols = set()
+ for _f in _dfiles:
+ _cols |= set(pl.read_parquet_schema(_f).keys())
+ _cols = sorted(_cols)
+ _parts = []
+ for _f in _dfiles:
+ _d = pl.read_parquet(_f)
+ for _c in _cols:
+ if _c not in _d.columns:
+ _d = _d.with_columns(pl.lit(None).alias(_c))
+ _parts.append(_d.select(_cols))
+ _diag = pl.concat(_parts, how="vertical_relaxed")
+ _search = pl.concat(
+ [pl.read_parquet(_sf, columns=["Name", "method", "replicate", "phase"])
+ for _sf in _sfiles],
+ how="vertical_relaxed",
+ )
+ _cn = _NAMES.get(_lib) # None -> auto-infer from reagent-ID prefixes
+ _methods = ["TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"]
+ _mlabels = {
+ "TACTICS Enhanced-RWS (GMIC)": "TACTICS RWS",
+ "TACTICS Enhanced-TT-TS (GMIC)": "TACTICS TT-TS",
+ }
+
+ _f1 = _gde(
+ _search, _diag, _rf, methods=_methods, method_labels=_mlabels,
+ component_names=_cn,
+ title=f"Layer 1 — GMIC directs WHICH component to explore ({_lib}, {_budget})",
+ )
+ _f2 = _ai(
+ _search, _diag, _rf, methods=_methods, method_labels=_mlabels,
+ component_names=_cn,
+ title=(
+ f"Layer 2 — within-component exploration intensity ({_lib}, {_budget})\n"
+ "both knobs are ×-multipliers centred on 1.0: "
+ "RWS = CATS temperature multiplier; TT-TS = σ-inflation multiplier"
+ ),
+ )
+ # Rasterise each figure explicitly. Embedding live matplotlib Figure
+ # objects in a single marimo cell is unreliable (only the last tends to
+ # show), so we convert both to PNG, then close them.
+ import io as _io3
+
+ def _png(_f):
+ _b = _io3.BytesIO()
+ _f.savefig(_b, format="png", dpi=110, bbox_inches="tight")
+ return _b.getvalue()
+
+ _png1, _pdf1 = _png(_f1), fig_to_pdf_bytes(_f1)
+ _png2, _pdf2 = _png(_f2), fig_to_pdf_bytes(_f2)
+ plt.close(_f1)
+ plt.close(_f2)
+ _out3 = mo.vstack([
+ mo.md(f"### Layer 1 — GMIC directs *which* component ({_lib} · {_budget})"),
+ mo.image(_png1),
+ mo.download(
+ data=_pdf1, filename=f"diag_gmic_directed_{_lib}.pdf",
+ mimetype="application/pdf", label="Download Layer 1 PDF",
+ ),
+ mo.md(f"### Layer 2 — within-component intensity ({_lib} · {_budget})"),
+ mo.image(_png2),
+ mo.download(
+ data=_pdf2, filename=f"diag_adaptive_intensity_{_lib}.pdf",
+ mimetype="application/pdf", label="Download Layer 2 PDF",
+ ),
+ ])
+ _out3
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Layers 1 & 2 — Exploration Mechanism for Any ROCS Library
+
+ The same two mechanism layers as the docking section, for the ROCS
+ (similarity) libraries. On ROCS the GMIC *direction* is query-specific
+ (different query molecules flag different components as flexible), so each
+ library shows a **single representative query** — picked for the clearest
+ sustained between-component GMIC separation with cross-replicate consensus
+ = 1.0 (`experiments/_debug/pick_rocs_demo_queries.py`; `rxn206`/`groebke`
+ reproduce the earlier manual demo-query picks). Pick a library from the dropdown;
+ replicate 0.
+
+ - **Layer 1 — *which* component?** GMIC reads each component as resolved
+ (high GMIC → exploit) or unresolved (low GMIC → explore); the *explored*
+ strip follows GMIC and cumulative coverage shows the consequence.
+ - **Layer 2 — *how hard* within that component?** RWS tunes the CATS
+ temperature multiplier; TT-TS tunes the σ-inflation multiplier (both
+ dimensionless ×-multipliers centred on 1.0).
+
+ **Data source:** the 2% canonical benchmark (`full_benchmark_nogate`,
+ gate-fixed; falls back to `full_benchmark_ema_cats`).
+ """)
+ return
+
+
+@app.cell
+def _(mo):
+ """Dropdown selector over the 20 ROCS libraries."""
+ _rocs_libs = [
+ "amide-suzuki", "betti", "dobener", "groebke-blackburn-bienayme",
+ "mannich", "niementowski", "orru", "passerini", "petasis", "poparov",
+ "rxn101", "rxn102a", "rxn108b", "rxn111b", "rxn114b", "rxn203",
+ "rxn205", "rxn206", "rxn207", "rxn208",
+ ]
+ rocs_lib_dd = mo.ui.dropdown(
+ options=_rocs_libs, value="rxn206", label="ROCS library"
+ )
+ rocs_lib_dd
+ return (rocs_lib_dd,)
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, pl, plt, reagents_dir, rocs_lib_dd):
+ """Render both layer plots for the selected ROCS library at its chosen
+ representative query (clearest GMIC component separation; replicate 0)."""
+ import glob as _glob4
+ import io as _io4
+ from pathlib import Path as _Path4
+
+ from TACTICS.library_analysis.diagnostic_plots import (
+ plot_adaptive_intensity as _ai_r,
+ plot_gmic_directed_exploration as _gde_r,
+ )
+
+ # Representative query per library (experiments/_debug/pick_rocs_demo_queries.py):
+ # clearest sustained between-component GMIC separation, cross-rep consensus 1.0.
+ _RQUERY = {
+ "amide-suzuki": "query_066", "betti": "query_098", "dobener": "query_018",
+ "groebke-blackburn-bienayme": "query_035", "mannich": "query_035",
+ "niementowski": "query_004", "orru": "query_103", "passerini": "query_037",
+ "petasis": "query_020", "poparov": "query_012", "rxn101": "query_018",
+ "rxn102a": "query_081", "rxn108b": "query_104", "rxn111b": "query_060",
+ "rxn114b": "query_018", "rxn203": "query_036", "rxn205": "query_024",
+ "rxn206": "query_047", "rxn207": "query_038", "rxn208": "query_024",
+ }
+ # Chemical reactant name per component (reagent_0, reagent_1, [reagent_2]),
+ # derived from each reaction's SMARTS in data/reactions.csv. Order follows the
+ # reagent-file order (reagent_0 = first SMARTS reactant). Edit freely — these
+ # replace the default "Component {idx}" labels on the layer-plot components.
+ _ROCS_COMPONENT_NAMES = {
+ # 2-component (rxn*) libraries
+ "rxn102a": ["Aryl halide", "Amine"],
+ "rxn108b": ["Aryl/vinyl halide", "Alkyne"],
+ "rxn111b": ["Alkene", "Aryl/vinyl halide"],
+ "rxn114b": ["Amine", "Amine"],
+ "rxn203": ["Amine", "Amine"],
+ "rxn205": ["Alcohol", "Phenol"],
+ "rxn206": ["Alcohol", "Sulfonamide"],
+ "rxn207": ["Carbonyl", "Amine"],
+ "rxn208": ["Aryl halide", "Amine"],
+ # rxn101: SMARTS absent from data/reactions.csv -> falls back to "Component {idx}"
+ # 3-component (named multicomponent) reactions
+ "amide-suzuki": ["Boronic acid", "Aryl halide", "Carboxylic acid"],
+ "betti": ["Naphthol", "Aldehyde", "Amine"],
+ "dobener": ["Aniline", "Aldehyde", "Ketoester"],
+ "groebke-blackburn-bienayme": ["Aminoazine", "Aldehyde", "Isocyanide"],
+ "mannich": ["Secondary amine", "Aldehyde", "Ketone"],
+ "niementowski": ["Anthranilic acid", "Amine", "Carboxylic acid"],
+ "orru": ["Aldehyde", "Cyanoacetate", "Amine"],
+ "passerini": ["Carboxylic acid", "Carbonyl", "Isocyanide"],
+ "petasis": ["Amine", "Carbonyl", "Boronic acid"],
+ "poparov": ["Aniline", "Alkene", "Aldehyde"],
+ }
+ # Reagent-folder convention (Local Norm #11): a couple of ROCS libs live in a
+ # differently-named folder; the reagent-file prefix stays the lib_id.
+ _FOLDER = {"amide-suzuki": "amide_suzuki",
+ "groebke-blackburn-bienayme": "groebke"}
+
+ _lib = rocs_lib_dd.value
+ _rcn = _ROCS_COMPONENT_NAMES.get(_lib) # chemical names; None -> "Component {idx}"
+ _q = _RQUERY[_lib]
+ _folder = _FOLDER.get(_lib, _lib)
+ _rf = sorted(
+ f for f in _glob4.glob(str(reagents_dir / _folder / f"{_lib}_reagent_*.smi"))
+ if not _Path4(f).name.startswith(".")
+ )
+ _outputs = reagents_dir.parent.parent / "outputs"
+
+ def _parqs(_d):
+ # Glob .parquet but skip hidden/AppleDouble (._*) sidecar files.
+ return [
+ f for f in sorted(_glob4.glob(str(_d / "*.parquet")))
+ if not _Path4(f).name.startswith(".")
+ ]
+
+ _candidates = [
+ (_outputs / "full_benchmark_nogate", "2% gate-fixed"),
+ (_outputs / "full_benchmark_ema_cats", "2% canonical"),
+ ]
+ _bench, _budget = next(
+ ((b, lbl) for b, lbl in _candidates if _parqs(b / "diagnostics" / _lib)),
+ (None, None),
+ )
+ _dfiles = _parqs(_bench / "diagnostics" / _lib) if _bench else []
+ _sfiles = _parqs(_bench / "search" / _lib) if _bench else []
+
+ if not _dfiles or not _sfiles or not _rf:
+ _outR = mo.md(f"⚠️ diagnostics/search/reagents not found for **{_lib}**.")
+ else:
+ # Per-shard column-union (RWS + TT-TS schemas differ), then filter to the
+ # single representative query so the layer plots see one trial per method.
+ _cols = set()
+ for _f in _dfiles:
+ _cols |= set(pl.read_parquet_schema(_f).keys())
+ _cols = sorted(_cols)
+ _parts = []
+ for _f in _dfiles:
+ _d = pl.read_parquet(_f)
+ for _c in _cols:
+ if _c not in _d.columns:
+ _d = _d.with_columns(pl.lit(None).alias(_c))
+ _parts.append(_d.select(_cols))
+ _diag = (
+ pl.concat(_parts, how="vertical_relaxed")
+ .filter(pl.col("query_id") == _q)
+ )
+ _search = (
+ pl.concat(
+ [pl.read_parquet(
+ _sf, columns=["Name", "method", "replicate", "phase", "query_id"])
+ for _sf in _sfiles],
+ how="vertical_relaxed",
+ )
+ .filter(pl.col("query_id") == _q)
+ .select(["Name", "method", "replicate", "phase"])
+ )
+ _methods = ["TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"]
+ _mlabels = {
+ "TACTICS Enhanced-RWS (GMIC)": "TACTICS RWS",
+ "TACTICS Enhanced-TT-TS (GMIC)": "TACTICS TT-TS",
+ }
+
+ _f1 = _gde_r(
+ _search, _diag, _rf, methods=_methods, method_labels=_mlabels,
+ component_names=_rcn,
+ title=f"Layer 1 — GMIC directs WHICH component to explore ({_lib} · {_q}, {_budget})",
+ )
+ _f2 = _ai_r(
+ _search, _diag, _rf, methods=_methods, method_labels=_mlabels,
+ component_names=_rcn,
+ title=(
+ f"Layer 2 — within-component exploration intensity ({_lib} · {_q}, {_budget})\n"
+ "both knobs are ×-multipliers centred on 1.0: "
+ "RWS = CATS temperature multiplier; TT-TS = σ-inflation multiplier"
+ ),
+ )
+
+ def _png(_f):
+ _b = _io4.BytesIO()
+ _f.savefig(_b, format="png", dpi=110, bbox_inches="tight")
+ return _b.getvalue()
+
+ _png1, _pdf1 = _png(_f1), fig_to_pdf_bytes(_f1)
+ _png2, _pdf2 = _png(_f2), fig_to_pdf_bytes(_f2)
+ plt.close(_f1)
+ plt.close(_f2)
+ _outR = mo.vstack([
+ mo.md(f"### Layer 1 — GMIC directs *which* component ({_lib} · {_q} · {_budget})"),
+ mo.image(_png1),
+ mo.download(
+ data=_pdf1, filename=f"diag_gmic_directed_{_lib}_{_q}.pdf",
+ mimetype="application/pdf", label="Download Layer 1 PDF",
+ ),
+ mo.md(f"### Layer 2 — within-component intensity ({_lib} · {_q} · {_budget})"),
+ mo.image(_png2),
+ mo.download(
+ data=_pdf2, filename=f"diag_adaptive_intensity_{_lib}_{_q}.pdf",
+ mimetype="application/pdf", label="Download Layer 2 PDF",
+ ),
+ ])
+ _outR
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ ## SI · GMIC routing is query-dependent on ROCS (scales with separation)
+
+ Backing figure for the one-line main-text claim that GMIC selects a
+ *different* component to explore depending on the query. One point per
+ (ROCS library x query): **x** = between-component GMIC separation
+ (max - min late-search GMIC); **y** = fraction of search cycles the
+ *lowest-GMIC* (explored) component was heated. Dashed line = round-robin
+ (1/n_components). GMIC tilts exploration toward the unresolved component
+ **in proportion to how separated the components are** -- real adaptive
+ routing, but modest, and near round-robin when the components are similar
+ (the typical ROCS query). The Layer-1 demo queries sit in the
+ high-separation tail. Loads all 20 ROCS libraries (~1-2 min).
+ """)
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mech_diag_dir, mo, np, pl, plt):
+ """SI: per-query GMIC routing tilt vs between-component separation, pooled
+ over the 20 ROCS libraries and split by component count. Reads the merged
+ per-library RWS diagnostics from full_benchmark_nogate."""
+ _RWS = "TACTICS Enhanced-RWS (GMIC)"
+ _LAST_N = 20
+ _ROCS = [
+ "rxn101", "rxn102a", "rxn108b", "rxn111b", "rxn114b", "rxn203", "rxn205",
+ "rxn206", "rxn207", "rxn208", "amide-suzuki", "betti", "dobener",
+ "groebke-blackburn-bienayme", "mannich", "niementowski", "orru",
+ "passerini", "petasis", "poparov",
+ ]
+ _cols = ["method", "query_id", "component_idx", "replicate",
+ "current_cycle", "total_cycles", "gmic", "is_heated"]
+
+ _rows = []
+ for _lib in _ROCS:
+ _p = mech_diag_dir / _lib / f"{_lib}.parquet"
+ if not _p.exists():
+ continue
+ _df = pl.read_parquet(_p, columns=_cols).filter(pl.col("method") == _RWS)
+ _nc = _df["component_idx"].n_unique()
+ _late = _df.filter(pl.col("current_cycle") > (pl.col("total_cycles") - _LAST_N))
+ _g = (
+ _late.group_by(["query_id", "component_idx", "replicate"])
+ .agg(pl.col("gmic").mean().alias("g"))
+ .group_by(["query_id", "component_idx"])
+ .agg(pl.col("g").mean().alias("lg"))
+ )
+ _hf = _df.group_by(["query_id", "component_idx"]).agg(
+ pl.col("is_heated").cast(pl.Float64).mean().alias("hf")
+ )
+ _gq = _g.group_by("query_id").agg(
+ (pl.col("lg").max() - pl.col("lg").min()).alias("gap"),
+ pl.col("component_idx").sort_by("lg").first().alias("chosen"),
+ )
+ _j = _gq.join(_hf, left_on=["query_id", "chosen"],
+ right_on=["query_id", "component_idx"])
+ for _r in _j.iter_rows(named=True):
+ _rows.append(dict(lib=_lib, n_comp=_nc, gap=_r["gap"],
+ chosen_share=_r["hf"]))
+
+ _data = pl.DataFrame(_rows)
+
+ _fig, _axes = plt.subplots(1, 2, figsize=(11, 4.4))
+ for _ax, _ncomp, _base, _title in [
+ (_axes[0], 2, 0.50, "2-component ROCS"),
+ (_axes[1], 3, 1.0 / 3.0, "3-component ROCS"),
+ ]:
+ _d = _data.filter(pl.col("n_comp") == _ncomp)
+ _x, _y = _d["gap"].to_numpy(), _d["chosen_share"].to_numpy()
+ _ax.scatter(_x, _y, s=12, alpha=0.30, color="#3b6fb6", edgecolors="none")
+ _ax.axhline(_base, color="crimson", ls="--", lw=1.4,
+ label=f"round-robin = {_base:.2f}")
+ _m, _b = np.polyfit(_x, _y, 1)
+ _xs = np.linspace(_x.min(), _x.max(), 50)
+ _ax.plot(_xs, _m * _xs + _b, color="black", lw=1.6)
+ _ax.text(0.96, 0.07,
+ f"r = {np.corrcoef(_x, _y)[0, 1]:.2f}\n"
+ f"{len(_x)} queries, {_d['lib'].n_unique()} libs",
+ transform=_ax.transAxes, ha="right", va="bottom", fontsize=9)
+ _ax.set_title(_title, fontsize=11)
+ _ax.set_xlabel("between-component GMIC separation (max - min)")
+ _ax.set_ylabel("heating share of lowest-GMIC component")
+ _ax.legend(loc="upper left", fontsize=8, frameon=False)
+ _fig.suptitle("GMIC exploration routing on ROCS is query-dependent and "
+ "scales with component separation", fontsize=12)
+ _fig.tight_layout(rect=[0, 0, 1, 0.95])
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="SI_gmic_routing_query_dependence.pdf",
+ mimetype="application/pdf", label="Download as PDF",
+ ),
+ ])
+ return
+
+
+if __name__ == "__main__":
+ app.run()
diff --git a/tutorials/interactive_sar_explorer.py b/tutorials/interactive_sar_explorer.py
new file mode 100644
index 0000000..a9eef11
--- /dev/null
+++ b/tutorials/interactive_sar_explorer.py
@@ -0,0 +1,342 @@
+"""
+Interactive SAR Explorer — Hover-to-Structure with molplotly
+
+Hover over reagent dots to see molecular structures rendered by RDKit.
+Per-component oracle GMIC shows whether a component needs exploration
+(low GMIC) or can be exploited (high GMIC).
+
+Run as app: marimo run tutorials/interactive_sar_explorer.py
+Edit mode: marimo edit tutorials/interactive_sar_explorer.py
+"""
+
+import marimo
+
+__generated_with = "0.21.1"
+app = marimo.App(width="full", app_title="Interactive SAR Explorer")
+
+
+@app.cell
+def _():
+ import marimo as mo
+ import socket
+ import threading
+ from pathlib import Path
+ from io import BytesIO
+
+ import numpy as np
+ import pandas as pd
+ import polars as pl
+ import plotly.express as px
+ import molplotly
+ import re
+
+ try:
+ project_root = Path(__file__).parent.parent.resolve()
+ except NameError:
+ project_root = Path.cwd()
+
+ SCORES_DIR = project_root / "data" / "scores"
+ REAGENTS_DIR = project_root / "data" / "reagents"
+
+ return (
+ SCORES_DIR, REAGENTS_DIR, BytesIO, mo, molplotly,
+ np, pd, pl, project_root, px, re, socket, threading,
+ )
+
+
+@app.cell
+def _(mo):
+ library_selector = mo.ui.dropdown(
+ options={
+ "Adenine (3-comp docking, +62.1)": "adenine",
+ "Quinazoline (3-comp docking, +0.1 ns)": "quinazoline",
+ "Thrombin (2-comp docking, +3.2)": "thrombin",
+ "Amide (2-comp docking, +5.1)": "amide",
+ "rxn206 (2-comp ROCS, +7.6)": "rxn206",
+ },
+ value="Adenine (3-comp docking, +62.1)",
+ label="Library",
+ )
+ top_n_selector = mo.ui.slider(
+ start=25, stop=250, step=25, value=100, label="Top-N threshold",
+ )
+ mo.vstack([
+ mo.md(r"""
+# Interactive SAR Explorer
+
+**Hover** over dots to see reagent structures.
+Each panel is one reaction component. Oracle GMIC quantifies
+how informative per-reagent mean scores are: **low GMIC** = high
+flexibility (reagent means don't distinguish hits) /
+**high GMIC** = high criticality (reagent means separate hits).
+ """),
+ mo.hstack([library_selector, top_n_selector], gap=2),
+ ])
+ return library_selector, top_n_selector
+
+
+@app.cell
+def _(library_selector, mo, pl, re, SCORES_DIR, REAGENTS_DIR):
+ """Load scores and reagent SMILES; create query selector for ROCS."""
+ _lib_id = library_selector.value
+
+ def _parse_adenine(code):
+ _m = re.match(r"(amidine_\d+)_(isocyanide_db_\d+)_(aldehyde_\d+)", code)
+ return list(_m.groups()) if _m else None
+
+ def _parse_thrombin(code):
+ _parts = code.split("_")
+ return [_parts[0], _parts[1] + "_" + _parts[2]] if len(_parts) == 3 else None
+
+ def _parse_split(code, n):
+ _parts = code.split("_")
+ return _parts if len(_parts) == n else None
+
+ _CONFIGS = {
+ "adenine": {
+ "components": ["Amidines", "Isocyanides", "Aldehydes"],
+ "reagent_files": [
+ "adenine_reagent_0.smi", "adenine_reagent_1.smi",
+ "adenine_reagent_2.smi",
+ ],
+ "code_col": "Product_Code", "mode": "minimize",
+ "parse": _parse_adenine,
+ },
+ "quinazoline": {
+ "components": ["Aminobenzoics", "Amines", "Acids"],
+ "reagent_files": [
+ "quinazoline_reagent_0.smi", "quinazoline_reagent_1.smi",
+ "quinazoline_reagent_2.smi",
+ ],
+ "code_col": "Product_Code", "mode": "minimize",
+ "parse": lambda c: _parse_split(c, 3),
+ },
+ "thrombin": {
+ "components": ["Acids", "Dipeptides"],
+ "reagent_files": [
+ "thrombin_reagent_0.smi", "thrombin_reagent_1.smi",
+ ],
+ "code_col": "Product_Code", "mode": "minimize",
+ "parse": _parse_thrombin,
+ },
+ "amide": {
+ "components": ["Amines", "Acids"],
+ "reagent_files": [
+ "amide_reagent_0.smi", "amide_reagent_1.smi",
+ ],
+ "code_col": "Product_Code", "mode": "minimize",
+ "parse": lambda c: _parse_split(c, 2),
+ },
+ "rxn206": {
+ "components": ["Component 0", "Component 1"],
+ "reagent_files": [
+ "rxn206_reagent_0.smi", "rxn206_reagent_1.smi",
+ ],
+ "code_col": "Name", "mode": "maximize",
+ "parse": lambda c: _parse_split(c, 2),
+ },
+ }
+
+ lib_cfg = _CONFIGS[_lib_id]
+ score_df = pl.read_parquet(SCORES_DIR / f"{_lib_id}.parquet")
+
+ if lib_cfg["mode"] == "maximize":
+ _qcols = sorted(c for c in score_df.columns if c.startswith("query_"))
+ query_selector = mo.ui.dropdown(
+ options=_qcols, value=_qcols[0], label="Query molecule",
+ )
+ else:
+ query_selector = mo.ui.dropdown(
+ options=["Scores"], value="Scores", label="Score column",
+ )
+
+ smiles_maps = []
+ for _fname in lib_cfg["reagent_files"]:
+ _smap = {}
+ with open(REAGENTS_DIR / _lib_id / _fname, errors="replace") as _f:
+ for _line in _f:
+ _line = _line.strip()
+ if not _line:
+ continue
+ _parts = _line.split("\t", 1) if "\t" in _line else _line.split(None, 1)
+ if len(_parts) >= 2:
+ _smap[_parts[1].strip()] = _parts[0].strip()
+ smiles_maps.append(_smap)
+
+ query_selector if lib_cfg["mode"] == "maximize" else mo.md("")
+ return lib_cfg, query_selector, score_df, smiles_maps
+
+
+@app.cell
+def _(lib_cfg, query_selector, top_n_selector, score_df, smiles_maps, np, pd, pl):
+ """Compute per-reagent stats and build chart DataFrame."""
+ _score_col = query_selector.value
+ _top_n = top_n_selector.value
+ _mode = lib_cfg["mode"]
+ _parse = lib_cfg["parse"]
+ _code_col = lib_cfg["code_col"]
+ _comp_names = lib_cfg["components"]
+
+ _df = score_df.filter(pl.col(_score_col).is_not_nan())
+ _ascending = _mode == "minimize"
+ _top_codes = set(
+ _df.sort(_score_col, descending=not _ascending)
+ .head(_top_n)[_code_col]
+ .to_list()
+ )
+
+ _codes = _df[_code_col].to_list()
+ _scores = _df[_score_col].to_numpy()
+ _parsed = [_parse(c) for c in _codes]
+ _parsed_top = {c: _parse(c) for c in _top_codes}
+
+ _rows = []
+ for _ci, _cname in enumerate(_comp_names):
+ _r2s = {}
+ for _p, _s in zip(_parsed, _scores):
+ if _p is not None:
+ _r2s.setdefault(_p[_ci], []).append(_s)
+
+ _top_counts = {}
+ for _c, _p in _parsed_top.items():
+ if _p is not None:
+ _top_counts[_p[_ci]] = _top_counts.get(_p[_ci], 0) + 1
+
+ _means = {r: float(np.mean(s)) for r, s in _r2s.items()}
+ _stds = {r: float(np.std(s)) for r, s in _r2s.items()}
+
+ _all_means = np.array(list(_means.values()))
+ _all_stds = np.array(list(_stds.values()))
+ _sig_var = float(np.var(_all_means))
+ _noise_var = float(np.mean(_all_stds**2))
+ _oracle_gmic = 0.5 * np.log(1 + _sig_var / _noise_var) if _noise_var > 0 else 0.0
+
+ # Min-max normalization to [0, 1] for cross-component comparability
+ _m_min, _m_max = float(np.min(_all_means)), float(np.max(_all_means))
+ _s_min, _s_max = float(np.min(_all_stds)), float(np.max(_all_stds))
+ _m_range = _m_max - _m_min if _m_max > _m_min else 1.0
+ _s_range = _s_max - _s_min if _s_max > _s_min else 1.0
+
+ _comp_title = f"{_cname} \u2014 GMIC {_oracle_gmic:.3f}"
+
+ _smap = smiles_maps[_ci]
+ for _rid in sorted(_r2s.keys()):
+ _nm = (_means[_rid] - _m_min) / _m_range
+ _ns = (_stds[_rid] - _s_min) / _s_range
+ _rows.append({
+ "component": _comp_title,
+ "reagent_id": str(_rid),
+ "smiles": _smap.get(str(_rid), ""),
+ "mean": round(_means[_rid], 3),
+ "std": round(_stds[_rid], 3),
+ "norm_mean": round(_nm, 3),
+ "norm_std": round(_ns, 3),
+ "count": _top_counts.get(_rid, 0),
+ "n_obs": len(_r2s[_rid]),
+ "oracle_gmic": round(_oracle_gmic, 3),
+ })
+
+ chart_df = pd.DataFrame(_rows)
+ return (chart_df,)
+
+
+@app.cell
+def _(chart_df, px, top_n_selector):
+ """Build Plotly figure with faceted scatter per component."""
+ _top_n = top_n_selector.value
+
+ plotly_fig = px.scatter(
+ chart_df,
+ x="norm_mean",
+ y="norm_std",
+ color="count",
+ facet_col="component",
+ facet_col_wrap=3,
+ hover_data=["reagent_id", "mean", "std", "n_obs", "oracle_gmic"],
+ color_continuous_scale="YlOrRd",
+ labels={
+ "norm_mean": "Reagent Mean (normalized)",
+ "norm_std": "Reagent Stdev (normalized)",
+ "count": f"Top-{_top_n}",
+ },
+ )
+ plotly_fig.update_traces(
+ marker=dict(
+ size=8,
+ line=dict(width=0.5, color="#666"),
+ ),
+ )
+ plotly_fig.update_layout(
+ template="plotly_white",
+ width=400 * chart_df["component"].nunique(),
+ height=450,
+ font=dict(size=13),
+ )
+ # Clean up facet titles (remove "component = " prefix)
+ plotly_fig.for_each_annotation(
+ lambda a: a.update(text=a.text.split("=", 1)[-1].strip())
+ )
+ return (plotly_fig,)
+
+
+@app.cell
+def _(plotly_fig, chart_df, molplotly, mo, socket, threading, top_n_selector, BytesIO):
+ """Launch molplotly Dash app for hover-to-structure and embed via iframe."""
+ _top_n = top_n_selector.value
+
+ # Find a free port
+ _sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ _sock.bind(("127.0.0.1", 0))
+ _dash_port = _sock.getsockname()[1]
+ _sock.close()
+
+ _dash_app = molplotly.add_molecules(
+ fig=plotly_fig,
+ df=chart_df,
+ smiles_col="smiles",
+ title_col="reagent_id",
+ show_coords=False,
+ caption_cols=["count", "mean", "std", "oracle_gmic"],
+ caption_transform={
+ "count": lambda x: f"Top-{_top_n}: {x}",
+ "oracle_gmic": lambda x: f"GMIC: {x}",
+ },
+ svg_size=250,
+ )
+
+ _thread = threading.Thread(
+ target=lambda: _dash_app.run(
+ host="127.0.0.1", port=_dash_port,
+ debug=False, use_reloader=False,
+ ),
+ daemon=True,
+ )
+ _thread.start()
+
+ # PDF export via kaleido
+ _pdf_buf = BytesIO()
+ plotly_fig.write_image(_pdf_buf, format="pdf", scale=2)
+ _pdf_bytes = _pdf_buf.getvalue()
+
+ mo.vstack([
+ mo.md(
+ f"**Interactive viewer** (hover for structures) — "
+ f"[open in new tab](http://localhost:{_dash_port})"
+ ),
+ mo.Html(
+ f''
+ ),
+ mo.download(
+ data=_pdf_bytes,
+ filename="sar_scatter.pdf",
+ mimetype="application/pdf",
+ label="Download PDF",
+ ),
+ ])
+ return
+
+
+if __name__ == "__main__":
+ app.run()
diff --git a/tutorials/legacy_vs_current_comparison.py b/tutorials/legacy_vs_current_comparison.py
deleted file mode 100644
index 14dcf40..0000000
--- a/tutorials/legacy_vs_current_comparison.py
+++ /dev/null
@@ -1,1177 +0,0 @@
-"""
-TACTICS Legacy vs Current Implementation Benchmark
-
-This notebook compares the performance of:
-1. Legacy StandardThompsonSampler (greedy selection)
-2. Legacy EnhancedThompsonSampler (roulette wheel with CATS)
-3. Current ThompsonSampler with various strategies
-
-Dataset: Thrombin (amide coupling)
-
-Run as app: marimo run notebooks/legacy_vs_current_benchmark.py
-Edit mode: marimo edit notebooks/legacy_vs_current_benchmark.py
-"""
-
-import marimo
-
-__generated_with = "0.19.7"
-app = marimo.App(width="full", app_title="TACTICS Legacy vs Current Benchmark")
-
-
-@app.cell
-def _():
- """Imports and project setup."""
- import marimo as mo
- import sys
- from pathlib import Path
- import time
- import tempfile
- import numpy as np
-
- # Add TACTICS project paths
- try:
- project_root = Path(__file__).parent.parent.resolve()
- except NameError:
- project_root = Path("/Users/aakankschitnandkeolyar/Desktop/TACTICS")
-
- sys.path.insert(0, str(project_root))
- sys.path.insert(0, str(project_root / "src"))
-
- import polars as pl
-
- # Current TACTICS imports
- from TACTICS.thompson_sampling import ThompsonSampler
- from TACTICS.thompson_sampling.config import ThompsonSamplingConfig
- from TACTICS.thompson_sampling.strategies.config import (
- GreedyConfig,
- RouletteWheelConfig,
- EpsilonGreedyConfig,
- TopTwoConfig,
- )
- from TACTICS.thompson_sampling.warmup.config import (
- StandardWarmupConfig,
- BalancedWarmupConfig,
- EnhancedWarmupConfig,
- )
- from TACTICS.thompson_sampling.core.evaluator_config import LookupEvaluatorConfig
-
- from TACTICS.library_enumeration import SynthesisPipeline
- from TACTICS.library_enumeration.smarts_toolkit import ReactionConfig, ReactionDef
-
- from TACTICS.library_analysis.visualization import TS_Benchmarks
-
- # Legacy TACTICS imports
- from TACTICS.thompson_sampling.legacy.standard_thompson_sampling import StandardThompsonSampler
- from TACTICS.thompson_sampling.legacy.enhanced_thompson_sampling import EnhancedThompsonSampler
- from TACTICS.thompson_sampling.legacy.evaluators import LookupEvaluator as LegacyLookupEvaluator
- return (
- BalancedWarmupConfig,
- EnhancedThompsonSampler,
- EnhancedWarmupConfig,
- EpsilonGreedyConfig,
- GreedyConfig,
- LegacyLookupEvaluator,
- LookupEvaluatorConfig,
- ReactionConfig,
- ReactionDef,
- RouletteWheelConfig,
- StandardThompsonSampler,
- StandardWarmupConfig,
- SynthesisPipeline,
- TS_Benchmarks,
- ThompsonSampler,
- ThompsonSamplingConfig,
- TopTwoConfig,
- mo,
- pl,
- tempfile,
- time,
- )
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- # Legacy vs Current Thompson Sampling Benchmark
-
- Compare the original TACTICS Thompson Sampling implementations with the current
- refactored version to validate performance parity and identify improvements.
-
- ## Implementations Compared
-
- | Implementation | Description | Selection Method |
- |----------------|-------------|------------------|
- | **Legacy Standard** | Original greedy TS | argmax/argmin of samples |
- | **Legacy Enhanced** | CATS with roulette wheel | Thermal cycling + % library |
- | **Current Greedy** | Refactored greedy | Same as legacy standard |
- | **Current Epsilon-Greedy** | Exploration-exploitation | Random with epsilon prob |
- | **Current Roulette Wheel** | RWS + GMIC criticality | Thermal cycling + criticality rotation |
- | **Current TT-TS** | Top-Two TS + thermal cycling | Two-sample disagreement + criticality rotation |
-
- **Dataset:** Thrombin Linear Amide Library (~500K products)
- """)
- return
-
-
-@app.cell
-def _():
- """Load bundled Thrombin dataset paths."""
- import importlib.resources
-
- _data_files = importlib.resources.files("TACTICS.data.thrombin")
- ACIDS_FILE = str(_data_files / "acids.smi")
- AMINES_FILE = str(_data_files / "coupled_aa_sub.smi")
- SCORES_FILE = str(_data_files / "product_scores.csv")
- REAGENT_FILES = [ACIDS_FILE, AMINES_FILE]
- AMIDE_COUPLING_SMARTS = "[#6:1](=[O:2])[OH].[#7X3;H1,H2;!$(N[!#6]);!$(N[#6]=[O]);!$(N[#6]~[!#6;!#16]):3]>>[#6:1](=[O:2])[#7:3]"
- return AMIDE_COUPLING_SMARTS, REAGENT_FILES, SCORES_FILE
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Configuration
-
- ### Select Implementations to Compare
- """)
- return
-
-
-@app.cell
-def _(mo):
- """Implementation selection checkboxes."""
- # Legacy implementations
- run_legacy_standard = mo.ui.checkbox(value=True, label="Legacy Standard (Greedy)")
- run_legacy_enhanced = mo.ui.checkbox(value=False, label="Legacy Enhanced (CATS/RWS)")
-
- # Current implementations
- run_current_greedy = mo.ui.checkbox(value=True, label="Current Greedy")
- run_current_epsilon = mo.ui.checkbox(value=True, label="Current Epsilon-Greedy")
- run_current_roulette = mo.ui.checkbox(value=True, label="Current Roulette Wheel (GMIC)")
- run_current_ttts = mo.ui.checkbox(value=True, label="Current TT-TS")
- return (
- run_current_epsilon,
- run_current_greedy,
- run_current_roulette,
- run_current_ttts,
- run_legacy_enhanced,
- run_legacy_standard,
- )
-
-
-@app.cell
-def _(
- mo,
- run_current_epsilon,
- run_current_greedy,
- run_current_roulette,
- run_current_ttts,
- run_legacy_enhanced,
- run_legacy_standard,
-):
- """Display implementation selection."""
- mo.vstack([
- mo.md("**Legacy Implementations:**"),
- mo.hstack([run_legacy_standard, run_legacy_enhanced], justify="start", gap=2),
- mo.md("**Current Implementations:**"),
- mo.hstack([run_current_greedy, run_current_epsilon, run_current_roulette, run_current_ttts], justify="start", gap=2),
- ])
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Shared Parameters
- """)
- return
-
-
-@app.cell
-def _(mo):
- """Shared parameters for all methods."""
- warmup_k_slider = mo.ui.slider(
- start=1, stop=10, value=3, step=1,
- label="Warmup K (evaluations per reagent)"
- )
- cycles_slider = mo.ui.slider(
- start=1, stop=10, value=3, step=1,
- label="Benchmark cycles"
- )
- top_n_slider = mo.ui.slider(
- start=50, stop=500, value=100, step=50,
- label="Top N for recovery"
- )
- return cycles_slider, top_n_slider, warmup_k_slider
-
-
-@app.cell
-def _(cycles_slider, mo, top_n_slider, warmup_k_slider):
- """Display shared parameter controls."""
- mo.hstack([warmup_k_slider, cycles_slider, top_n_slider], justify="start", gap=2)
- return
-
-
-@app.cell
-def _(mo):
- """Legacy Standard search parameters."""
- legacy_iterations_slider = mo.ui.slider(
- start=100, stop=20000, value=500, step=100,
- label="Search iterations (1 compound/cycle)"
- )
- return (legacy_iterations_slider,)
-
-
-@app.cell
-def _(legacy_iterations_slider, mo, run_legacy_standard, warmup_k_slider):
- """Display Legacy Standard parameters (conditional)."""
- mo.stop(not run_legacy_standard.value)
- _output = mo.vstack([
- mo.md(
- f"### Legacy Standard (Greedy) Parameters\n"
- f"- **Warmup:** Standard (K={warmup_k_slider.value}) — each reagent sampled K times with random partners\n"
- f"- **Search:** {legacy_iterations_slider.value:,} iterations, 1 compound per cycle"
- ),
- legacy_iterations_slider,
- ])
- _output
- return
-
-
-@app.cell
-def _(mo):
- """Current TACTICS search parameters — widget definitions."""
- iterations_slider = mo.ui.slider(start=1, stop=500, value=5, step=1, label="Search iterations")
- batch_size_slider = mo.ui.slider(start=10, stop=500, value=100, step=10, label="Batch size (compounds per iteration)")
- warmup_dropdown = mo.ui.dropdown(
- options={"Standard (random)": "standard", "Balanced (stratified)": "balanced", "Enhanced (stochastic)": "enhanced"},
- value="Balanced (stratified)",
- label="Warmup Strategy",
- )
- return batch_size_slider, iterations_slider, warmup_dropdown
-
-
-@app.cell
-def _(
- batch_size_slider,
- iterations_slider,
- mo,
- run_current_epsilon,
- run_current_greedy,
- run_current_roulette,
- run_current_ttts,
- warmup_dropdown,
- warmup_k_slider,
-):
- """Display Current TACTICS parameters (conditional)."""
- _any_current = (
- run_current_greedy.value or run_current_epsilon.value
- or run_current_roulette.value or run_current_ttts.value
- )
- mo.stop(not _any_current)
- _search_evals = iterations_slider.value * batch_size_slider.value
- _output = mo.vstack([
- mo.md(
- "### Current TACTICS Parameters\n"
- "- **Warmup:** " + str(warmup_dropdown.value) + " (K=" + str(warmup_k_slider.value) + ")\n"
- "- **Search:** " + str(iterations_slider.value) + " iterations x "
- + str(batch_size_slider.value) + " batch = **" + str(f"{_search_evals:,}") + "** search evaluations"
- ),
- warmup_dropdown,
- mo.hstack([iterations_slider, batch_size_slider]),
- ])
- _output
- return
-
-
-@app.cell
-def _(mo):
- mo.md(r"""
- ### Iteration Budget Breakdown
-
- Shows how the total evaluation budget is distributed between warmup and search
- phases for each selected method, based on the current parameter settings.
- """)
- return
-
-
-@app.cell
-def _(
- REAGENT_FILES,
- batch_size_slider,
- iterations_slider,
- legacy_iterations_slider,
- legacy_percent_lib,
- mo,
- run_current_epsilon,
- run_current_greedy,
- run_current_roulette,
- run_current_ttts,
- run_legacy_enhanced,
- run_legacy_standard,
- warmup_dropdown,
- warmup_k_slider,
-):
- """Compute and display iteration budget breakdown per method."""
- # Read reagent file sizes
- _reagent_sizes = []
- for _f in REAGENT_FILES:
- with open(_f) as _fh:
- _reagent_sizes.append(sum(1 for _line in _fh if _line.strip()))
-
- _sum_sizes = sum(_reagent_sizes)
- _max_size = max(_reagent_sizes)
- _library_size = 1
- for _s in _reagent_sizes:
- _library_size *= _s
-
- _K = warmup_k_slider.value
- _warmup_name = warmup_dropdown.value
-
- _rows = []
-
- # Legacy Standard: Standard warmup, 1 compound per search cycle
- if run_legacy_standard.value:
- _ls_warmup = _sum_sizes * _K
- _ls_search = legacy_iterations_slider.value
- _ls_total = _ls_warmup + _ls_search
- _ls_pct = _ls_total / _library_size * 100
- _rows.append(
- f"| Legacy Standard | Standard (K={_K}) | "
- f"{_sum_sizes} x {_K} = **{_ls_warmup:,}** | "
- f"**{_ls_search:,}** (1/cycle) | **{_ls_total:,}** | {_ls_pct:.2f}% |"
- )
-
- # Legacy Enhanced: Enhanced warmup, batch=100 per search cycle
- # percent_of_library is the TOTAL budget (warmup + search)
- # The legacy search method computes: nsearch = pct * library_size - num_warmup
- if run_legacy_enhanced.value:
- _le_warmup = _max_size * _K
- _le_total = int(_library_size * legacy_percent_lib.value)
- _le_search = max(_le_total - _le_warmup, 0)
- _le_pct = _le_total / _library_size * 100
- _le_warn = " **[warmup > budget!]**" if _le_search == 0 else ""
- _rows.append(
- f"| Legacy Enhanced | Enhanced (K={_K}) | "
- f"{_max_size} x {_K} = **{_le_warmup:,}** | "
- f"{_le_total:,} - {_le_warmup:,} = **{_le_search:,}** (batch=100){_le_warn} | "
- f"**{_le_total:,}** | {_le_pct:.2f}% |"
- )
-
- # Current TACTICS: configurable warmup, batch_size per iteration
- _tactics_search = iterations_slider.value * batch_size_slider.value
- if _warmup_name == "enhanced":
- _current_warmup = _max_size * _K
- _current_warmup_label = f"Enhanced (K={_K})"
- else: # standard or balanced — both use sum(sizes) * K
- _current_warmup = _sum_sizes * _K
- _current_warmup_label = f"{_warmup_name.title()} (K={_K})"
-
- _current_methods = []
- if run_current_greedy.value:
- _current_methods.append("Current Greedy")
- if run_current_epsilon.value:
- _current_methods.append("Current Epsilon-Greedy")
- if run_current_roulette.value:
- _current_methods.append("Current Roulette Wheel")
- if run_current_ttts.value:
- _current_methods.append("Current TT-TS")
-
- for _method in _current_methods:
- _c_total = _current_warmup + _tactics_search
- _c_pct = _c_total / _library_size * 100
- _rows.append(
- f"| {_method} | {_current_warmup_label} | "
- f"**{_current_warmup:,}** | "
- f"{iterations_slider.value} x {batch_size_slider.value} = **{_tactics_search:,}** | "
- f"**{_c_total:,}** | {_c_pct:.2f}% |"
- )
-
- _sizes_str = " + ".join(str(s) for s in _reagent_sizes)
- _table = "\n".join(_rows) if _rows else ""
-
- _output = mo.md(
- f"**Library:** {_sizes_str} = **{_library_size:,}** products "
- f"({len(_reagent_sizes)} components: {', '.join(f'{s:,}' for s in _reagent_sizes)})\n\n"
- f"| Method | Warmup Strategy | Warmup Evaluations | Search Evaluations | Total Evaluations | % Library |\n"
- f"|--------|----------------|-------------------|-------------------|-------------------|-----------|\n"
- f"{_table}\n\n"
- f"**How evaluations are calculated:**\n\n"
- f"| Phase | Method | Formula | This library |\n"
- f"|-------|--------|---------|-------------|\n"
- f"| Warmup | Standard / Balanced | `sum(component_sizes) x K` | {_sum_sizes} x K |\n"
- f"| Warmup | Enhanced | `max(component_sizes) x K` | {_max_size} x K |\n"
- f"| Search | Legacy Standard | `iterations` (1 compound/cycle) | iterations |\n"
- f"| Search | Legacy Enhanced | `pct x library_size - warmup` (batch=100/cycle) | total budget - warmup |\n"
- f"| Search | TACTICS | `iterations x batch_size` | iterations x batch_size |"
- ) if _rows else mo.md("*Select at least one implementation above to see the iteration budget.*")
- _output
- return
-
-
-@app.cell
-def _(mo):
- """Legacy Enhanced specific parameters."""
- legacy_percent_lib = mo.ui.slider(
- start=0.01, stop=0.10, value=0.05, step=0.01,
- label="Percent of library to screen (total budget incl. warmup)"
- )
- legacy_scaling = mo.ui.slider(
- start=-1, stop=1, value=-1, step=1,
- label="Scaling (-1=minimize, 1=maximize)"
- )
- return legacy_percent_lib, legacy_scaling
-
-
-@app.cell
-def _(
- legacy_percent_lib,
- legacy_scaling,
- mo,
- run_legacy_enhanced,
- warmup_k_slider,
-):
- """Display Legacy Enhanced parameters (conditional)."""
- mo.stop(not run_legacy_enhanced.value)
- _output = mo.vstack([
- mo.md(
- f"### Legacy Enhanced (CATS/RWS) Parameters\n"
- f"- **Warmup:** Enhanced (K={warmup_k_slider.value}) — stochastic parallel pairing\n"
- f"- **Search:** {legacy_percent_lib.value*100:.0f}% of library (total budget), batch size = 100 per cycle"
- ),
- mo.hstack([legacy_percent_lib, legacy_scaling]),
- ])
- _output
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Run Benchmark
- """)
- return
-
-
-@app.cell
-def _(mo):
- """Run button."""
- run_button = mo.ui.run_button(label="Run Legacy vs Current Benchmark")
- run_button
- return (run_button,)
-
-
-@app.cell
-def _(
- AMIDE_COUPLING_SMARTS,
- BalancedWarmupConfig,
- EnhancedThompsonSampler,
- EnhancedWarmupConfig,
- EpsilonGreedyConfig,
- GreedyConfig,
- LegacyLookupEvaluator,
- LookupEvaluatorConfig,
- REAGENT_FILES,
- ReactionConfig,
- ReactionDef,
- RouletteWheelConfig,
- SCORES_FILE,
- StandardThompsonSampler,
- StandardWarmupConfig,
- SynthesisPipeline,
- ThompsonSampler,
- ThompsonSamplingConfig,
- TopTwoConfig,
- batch_size_slider,
- cycles_slider,
- iterations_slider,
- legacy_iterations_slider,
- legacy_percent_lib,
- legacy_scaling,
- mo,
- pl,
- run_button,
- run_current_epsilon,
- run_current_greedy,
- run_current_roulette,
- run_current_ttts,
- run_legacy_enhanced,
- run_legacy_standard,
- tempfile,
- time,
- warmup_dropdown,
- warmup_k_slider,
-):
- """Execute benchmark for all selected implementations."""
- mo.stop(not run_button.value, mo.md("*Click 'Run Legacy vs Current Benchmark' to start*"))
-
- # Check at least one is selected
- any_selected = (
- run_legacy_standard.value or
- run_legacy_enhanced.value or
- run_current_greedy.value or
- run_current_epsilon.value or
- run_current_roulette.value or
- run_current_ttts.value
- )
- if not any_selected:
- mo.stop(True, mo.md("**Error:** Select at least one implementation."))
-
- # Parameters
- _num_cycles = cycles_slider.value
- _K = warmup_k_slider.value
-
- # Results storage
- all_results = {}
- run_metadata = []
- start_time = time.time()
-
- # =========================================================================
- # LEGACY STANDARD THOMPSON SAMPLER
- # =========================================================================
- if run_legacy_standard.value:
- method_name = "Legacy Standard"
- _cycle_results = []
-
- for _cycle in range(_num_cycles):
- # Create legacy evaluator
- _legacy_eval = LegacyLookupEvaluator({
- "ref_filename": SCORES_FILE,
- "ref_colname": "Scores",
- "compound_col": "Product_Code",
- })
-
- # Create and configure legacy sampler
- _legacy_sampler = StandardThompsonSampler(mode="minimize")
- _legacy_sampler.set_hide_progress(True)
- _legacy_sampler.set_evaluator(_legacy_eval)
- _legacy_sampler.read_reagents(REAGENT_FILES)
- _legacy_sampler.set_reaction(AMIDE_COUPLING_SMARTS)
-
- # Run warmup and search
- _warmup_results = _legacy_sampler.warm_up(num_warmup_trials=_K)
- _search_results = _legacy_sampler.search(num_cycles=legacy_iterations_slider.value)
-
- # Convert to polars DataFrame
- _all_legacy = _warmup_results + _search_results
- _legacy_df = pl.DataFrame({
- "Name": [r[2] for r in _all_legacy],
- "score": [r[0] for r in _all_legacy],
- }).filter(pl.col("score").is_not_nan())
-
- _cycle_results.append(_legacy_df)
-
- all_results[method_name] = _cycle_results
- _best_scores = [r["score"].min() for r in _cycle_results]
- run_metadata.append({
- "method": method_name,
- "type": "Legacy",
- "warmup": f"Standard (K={_K})",
- "cycles": _num_cycles,
- "best_score": min(_best_scores),
- "avg_best": sum(_best_scores) / len(_best_scores),
- })
-
- # =========================================================================
- # LEGACY ENHANCED THOMPSON SAMPLER (CATS/RWS)
- # =========================================================================
- if run_legacy_enhanced.value:
- method_name = "Legacy Enhanced"
- _cycle_results = []
-
- for _cycle in range(_num_cycles):
- # Create a temp file for results (required by legacy enhanced)
- _temp_file = tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False)
- _temp_filename = _temp_file.name
- _temp_file.close()
-
- # Create legacy evaluator
- _legacy_eval = LegacyLookupEvaluator({
- "ref_filename": SCORES_FILE,
- "ref_colname": "Scores",
- "compound_col": "Product_Code",
- })
-
- # Create and configure legacy enhanced sampler
- _legacy_enhanced = EnhancedThompsonSampler(
- processes=1,
- scaling=legacy_scaling.value,
- percent_lib=legacy_percent_lib.value,
- search_stop=6000,
- min_cpds_per_core=50,
- results_filename=_temp_filename,
- )
- _legacy_enhanced.set_hide_progress(True)
- _legacy_enhanced.set_evaluator(_legacy_eval)
- _legacy_enhanced.read_reagents(REAGENT_FILES)
- _legacy_enhanced.set_reaction(AMIDE_COUPLING_SMARTS)
-
- # Run warmup and search
- _legacy_enhanced.warm_up(num_warmup_trials=_K)
- _search_results = _legacy_enhanced.search(
- percent_of_library=legacy_percent_lib.value,
- stop=6000,
- min_cpds_per_core=50,
- )
-
- # Read results from temp file and convert
- try:
- _legacy_df = pl.read_csv(_temp_filename, null_values=["nan"])
- _legacy_df = _legacy_df.rename({"SMILES": "smiles"}).select(["Name", "score"])
- _legacy_df = _legacy_df.filter(pl.col("score").is_not_nan())
- except Exception:
- # If file reading fails, create from search results
- _legacy_df = pl.DataFrame({
- "Name": [r[2] for r in _search_results],
- "score": [r[0] for r in _search_results],
- }).filter(pl.col("score").is_not_nan())
-
- _cycle_results.append(_legacy_df)
-
- # Clean up temp file
- import os
- try:
- os.unlink(_temp_filename)
- except Exception:
- pass
-
- all_results[method_name] = _cycle_results
- _best_scores = [r["score"].min() for r in _cycle_results if len(r) > 0]
- if _best_scores:
- run_metadata.append({
- "method": method_name,
- "type": "Legacy",
- "warmup": f"Enhanced (K={_K})",
- "cycles": _num_cycles,
- "best_score": min(_best_scores),
- "avg_best": sum(_best_scores) / len(_best_scores),
- })
-
- # =========================================================================
- # CURRENT IMPLEMENTATIONS
- # =========================================================================
-
- # Shared pipeline and evaluator config for current implementations
- _reaction_config = ReactionConfig(
- reactions=[ReactionDef(reaction_smarts=AMIDE_COUPLING_SMARTS, step_index=0)],
- reagent_file_list=REAGENT_FILES,
- )
- _pipeline = SynthesisPipeline(_reaction_config)
- _evaluator_config = LookupEvaluatorConfig(
- ref_filename=SCORES_FILE,
- compound_col="Product_Code",
- score_col="Scores",
- )
-
- # Build warmup config based on selection
- _warmup_name = warmup_dropdown.value
- if _warmup_name == "standard":
- _warmup_config = StandardWarmupConfig()
- _warmup_display = f"Standard (K={_K})"
- elif _warmup_name == "balanced":
- _warmup_config = BalancedWarmupConfig(observations_per_reagent=_K)
- _warmup_display = f"Balanced (K={_K})"
- else: # enhanced
- _warmup_config = EnhancedWarmupConfig()
- _warmup_display = f"Enhanced (K={_K})"
-
- # Helper function for current implementations
- def run_current_implementation(method_name, strategy_config):
- _cycle_results = []
- for _cycle in range(_num_cycles):
- _config = ThompsonSamplingConfig(
- synthesis_pipeline=_pipeline,
- num_ts_iterations=iterations_slider.value,
- num_warmup_trials=_K,
- strategy_config=strategy_config,
- warmup_config=_warmup_config,
- evaluator_config=_evaluator_config,
- batch_size=batch_size_slider.value,
- max_resamples=1000,
- hide_progress=True,
- use_boltzmann_weighting=True,
- )
- _sampler = ThompsonSampler.from_config(_config)
- _warmup_df = _sampler.warm_up(num_warmup_trials=_config.num_warmup_trials)
- _search_df = _sampler.search(num_cycles=_config.num_ts_iterations)
- _combined = pl.concat([_warmup_df, _search_df])
- _result = _combined.select(["Name", "score"])
- _cycle_results.append(_result)
- _sampler.close()
-
- all_results[method_name] = _cycle_results
- _best_scores = [r["score"].min() for r in _cycle_results]
- run_metadata.append({
- "method": method_name,
- "type": "Current",
- "warmup": _warmup_display,
- "cycles": _num_cycles,
- "best_score": min(_best_scores),
- "avg_best": sum(_best_scores) / len(_best_scores),
- })
-
- # Current Greedy
- if run_current_greedy.value:
- run_current_implementation("Current Greedy", GreedyConfig(mode="minimize"))
-
- # Current Epsilon-Greedy
- if run_current_epsilon.value:
- run_current_implementation(
- "Current Epsilon-Greedy",
- EpsilonGreedyConfig(mode="minimize", epsilon=0.2, decay=0.995)
- )
-
- # Current Roulette Wheel (GMIC)
- if run_current_roulette.value:
- run_current_implementation(
- "Current Roulette Wheel",
- RouletteWheelConfig(mode="minimize", alpha=0.1, beta=0.05, divergence_threshold=0.1)
- )
-
- # Current TT-TS
- if run_current_ttts.value:
- run_current_implementation(
- "Current TT-TS",
- TopTwoConfig(mode="minimize", beta=0.5, heated_scale=1.5, cooled_scale=0.75)
- )
-
- total_time = time.time() - start_time
- method_names = list(all_results.keys())
- return all_results, method_names, run_metadata, total_time
-
-
-@app.cell(hide_code=True)
-def _(mo, run_button, run_metadata, total_time):
- """Display results summary."""
- mo.stop(not run_button.value)
-
- _rows = []
- for m in run_metadata:
- _rows.append(
- f"| {m['method']} | {m['type']} | {m['warmup']} | {m['cycles']} | "
- f"{m['best_score']:.3f} | {m['avg_best']:.3f} |"
- )
- _table = "\n".join(_rows)
-
- mo.md(f"""
- ---
- ## Results Summary
-
- **Total runtime:** {total_time:.1f}s
-
- | Method | Type | Warmup | Cycles | Best Score | Avg Best |
- |--------|------|--------|--------|------------|----------|
- {_table}
- """)
- return
-
-
-@app.cell
-def _(mo, run_button):
- """Save benchmark results to CSV."""
- mo.stop(not run_button.value)
-
- import pathlib as _pathlib
- import datetime as _dt
-
- _timestamp = _dt.datetime.now().strftime("%Y%m%d_%H%M%S")
- _default_dir = str(_pathlib.Path.cwd().parent / "outputs" / "benchmark_results")
-
- save_dir_input = mo.ui.text(
- value=_default_dir,
- label="Output directory",
- full_width=True,
- )
- save_button = mo.ui.run_button(label="Save Results to CSV")
-
- mo.vstack([
- mo.md("### Save Results"),
- save_dir_input,
- save_button,
- ])
- return save_button, save_dir_input
-
-
-@app.cell
-def _(
- all_results,
- method_names,
- mo,
- pl,
- run_button,
- run_metadata,
- save_button,
- save_dir_input,
-):
- """Write CSV files when save button is clicked."""
- mo.stop(not run_button.value)
- mo.stop(not save_button.value, mo.md("*Click 'Save Results to CSV' to export.*"))
-
- import pathlib as _pathlib
- import datetime as _dt
-
- _timestamp = _dt.datetime.now().strftime("%Y%m%d_%H%M%S")
- _out_dir = _pathlib.Path(save_dir_input.value) / _timestamp
- _out_dir.mkdir(parents=True, exist_ok=True)
-
- _saved = []
-
- # 1. Run metadata summary
- _meta_df = pl.DataFrame(run_metadata)
- _meta_path = _out_dir / "run_summary.csv"
- _meta_df.write_csv(_meta_path)
- _saved.append(f"`run_summary.csv` — {len(run_metadata)} methods")
-
- # 2. Per-method, per-cycle compound results
- for _name in method_names:
- _cycle_results = all_results[_name]
- _safe_name = _name.lower().replace(" ", "_").replace("-", "_")
- for _i, _df in enumerate(_cycle_results):
- if len(_df) > 0:
- _fname = f"{_safe_name}_cycle{_i + 1}.csv"
- _df.write_csv(_out_dir / _fname)
- _saved.append(f"`{_safe_name}_cycle*.csv` — {len(_cycle_results)} cycles")
-
- # 3. Combined results across all methods and cycles
- _all_dfs = []
- for _name in method_names:
- for _i, _df in enumerate(all_results[_name]):
- if len(_df) > 0:
- _all_dfs.append(
- _df.with_columns(
- pl.lit(_name).alias("method"),
- pl.lit(_i + 1).alias("cycle"),
- )
- )
- if _all_dfs:
- _combined = pl.concat(_all_dfs)
- _combined.write_csv(_out_dir / "all_results_combined.csv")
- _saved.append(f"`all_results_combined.csv` — {len(_combined):,} rows")
-
- _file_list = "\n".join(f"- {s}" for s in _saved)
- mo.md(f"""
- **Saved to** `{_out_dir}`
-
- {_file_list}
- """)
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Performance Comparison Analysis
-
- Key questions to answer:
- 1. Does **Current Greedy** match **Legacy Standard** performance?
- 2. Does **Current Roulette Wheel** match **Legacy Enhanced** performance?
- 3. How does **Current Epsilon-Greedy** compare to both?
- 4. How does **warmup strategy** affect results?
- """)
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Top Compounds Found
- """)
- return
-
-
-@app.cell(hide_code=True)
-def _(all_results, method_names, mo, pl, run_button):
- """Display top compounds table."""
- mo.stop(not run_button.value)
-
- _all_dfs = []
- for _name in method_names:
- _results = all_results[_name]
- for _i, _df in enumerate(_results):
- if len(_df) > 0:
- _labeled = _df.with_columns([
- pl.lit(_name).alias("method"),
- pl.lit(_i + 1).alias("cycle"),
- ])
- _all_dfs.append(_labeled)
-
- if _all_dfs:
- combined_all = pl.concat(_all_dfs)
- top_20 = (
- combined_all
- .sort("score")
- .head(20)
- .select(["Name", "score", "method", "cycle"])
- )
- top_20
- else:
- mo.md("*No results to display*")
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Benchmark Visualizations
-
- ### Toggle Methods for Comparison
- """)
- return
-
-
-@app.cell
-def _(SCORES_FILE, mo, pl, run_button):
- """Load reference data."""
- mo.stop(not run_button.value)
-
- reference_data = (
- pl.read_csv(SCORES_FILE)
- .rename({"Product_Code": "Name", "Scores": "score"})
- .select(["Name", "score"])
- )
- return (reference_data,)
-
-
-@app.cell
-def _(method_names, mo, run_button):
- """Create toggle checkboxes for visualization."""
- mo.stop(not run_button.value)
-
- viz_toggles = mo.ui.array([
- mo.ui.checkbox(value=True, label=name)
- for name in method_names
- ])
- return (viz_toggles,)
-
-
-@app.cell(hide_code=True)
-def _(mo, run_button, viz_toggles):
- """Display visualization toggles."""
- mo.stop(not run_button.value)
-
- mo.vstack([
- mo.md("**Select methods to visualize:**"),
- mo.hstack(viz_toggles, justify="start", gap=2, wrap=True),
- ])
- return
-
-
-@app.cell
-def _(all_results, method_names, mo, run_button, viz_toggles):
- """Get selected methods for visualization."""
- mo.stop(not run_button.value)
-
- viz_methods = [
- name for i, name in enumerate(method_names)
- if viz_toggles[i].value
- ]
- viz_results = {name: all_results[name] for name in viz_methods}
-
- mo.md(f"**Visualizing:** {', '.join(viz_methods) if viz_methods else 'None'}")
- return viz_methods, viz_results
-
-
-@app.cell
-def _(
- TS_Benchmarks,
- cycles_slider,
- mo,
- reference_data,
- run_button,
- top_n_slider,
- viz_methods,
- viz_results,
-):
- """Create TS_Benchmarks instance - reactive to toggle changes."""
- mo.stop(not run_button.value)
-
- # Handle empty selection gracefully without breaking reactivity
- if len(viz_methods) == 0:
- benchmarks = None
- else:
- # Filter out empty results
- _filtered_results = {
- k: v for k, v in viz_results.items()
- if all(len(df) > 0 for df in v)
- }
- _filtered_methods = list(_filtered_results.keys())
-
- if not _filtered_methods:
- benchmarks = None
- else:
- benchmarks = TS_Benchmarks(
- no_of_cycles=cycles_slider.value,
- methods_list=_filtered_methods,
- TS_runs_data=_filtered_results,
- reference_data=reference_data,
- top_n=top_n_slider.value,
- sort_type="minimize",
- top_ns=[25, 50, 100, 200, 300],
- )
- return (benchmarks,)
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Strip Plot: Score Distributions
- """)
- return
-
-
-@app.cell
-def _(benchmarks, mo, run_button):
- """Generate strip plot - reactive to toggle changes."""
- mo.stop(not run_button.value)
-
- # Use conditional rendering to maintain reactivity
- _output = (
- mo.md("*Select at least one method to visualize*")
- if benchmarks is None
- else benchmarks.stripplot_TS_results(
- width=900,
- height=400,
- show_plot=True,
- legend_position="bottom"
- )
- )
- _output
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Bar Plot: Hit Recovery by Cycle
-
- Compare how legacy and current implementations recover top hits.
- """)
- return
-
-
-@app.cell
-def _(benchmarks, mo, run_button):
- """Generate bar plot - reactive to toggle changes."""
- mo.stop(not run_button.value)
-
- # Use conditional rendering to maintain reactivity
- _output = (
- mo.md("*Select at least one method to visualize*")
- if benchmarks is None
- else benchmarks.plot_barplot_TS_results(
- width=900,
- height=400,
- show_plot=True,
- legend_position="bottom",
- dark_mode=False
- )
- )
- _output
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Line Plot: Performance vs Top-N
-
- This plot shows recovery rates at different top-N thresholds.
- Legacy and current implementations should show similar curves if they're equivalent.
- """)
- return
-
-
-@app.cell
-def _(benchmarks, mo, run_button):
- """Generate line plot - reactive to toggle changes."""
- mo.stop(not run_button.value)
-
- # Use conditional rendering to maintain reactivity
- _output = (
- mo.md("*Select at least one method to visualize*")
- if benchmarks is None
- else benchmarks.plot_line_performance_with_error_bars(
- width=900,
- height=450,
- show_plot=True,
- legend_position="bottom"
- )
- )
- _output
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Performance Statistics
- """)
- return
-
-
-@app.cell(hide_code=True)
-def _(benchmarks, mo, run_button):
- """Display grouped statistics - reactive to toggle changes."""
- mo.stop(not run_button.value)
-
- # Use conditional rendering to maintain reactivity
- if benchmarks is None:
- _stats = mo.md("*Select at least one method to visualize*")
- elif benchmarks.grouped_stats is not None:
- _stats = mo.vstack([
- mo.md("**Mean Performance by Top-N:**"),
- benchmarks.grouped_stats,
- ])
- else:
- _stats = mo.md("*Statistics not available*")
- _stats
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Implementation Comparison Notes
-
- ### Legacy vs Current Mapping
-
- | Legacy | Current Equivalent | Key Differences |
- |--------|-------------------|-----------------|
- | `StandardThompsonSampler(mode="minimize")` | `ThompsonSampler` + `GreedyConfig` | Same algorithm, refactored |
- | `StandardThompsonSampler(mode="minimize_boltzmann")` | `ThompsonSampler` + `BoltzmannConfig` | Boltzmann temperature handling |
- | `EnhancedThompsonSampler` | `ThompsonSampler` + `RouletteWheelConfig` | GMIC criticality + thermal cycling |
- | — | `ThompsonSampler` + `TopTwoConfig` | TT-TS best-arm ID + thermal cycling |
-
- ### Warmup Strategy Comparison
-
- | Warmup | Description | When to Use |
- |--------|-------------|-------------|
- | **Standard** | Random partner selection | Quick baseline |
- | **Balanced** | K observations per reagent | Best coverage (recommended) |
- | **Enhanced** | Stochastic parallel pairing | Large libraries |
-
- ### Expected Results
-
- - **Legacy Standard** and **Current Greedy** should produce statistically similar results
- - **Current Epsilon-Greedy** typically shows better exploration (more diverse hits)
- - **Current Roulette Wheel** uses GMIC criticality-weighted rotation (improved over Legacy Enhanced)
- - **Current TT-TS** targets best-arm identification with thermal cycling — best recovery on manuscript benchmarks
-
- ### Key Improvements in Current Version
-
- 1. **Unified API**: Single `ThompsonSampler` class with pluggable strategies
- 2. **Pydantic Configuration**: Type-safe, validated configuration objects
- 3. **SynthesisPipeline Integration**: Cleaner molecule synthesis workflow
- 4. **Polars DataFrames**: Better performance for large result sets
- 5. **Strategy Pattern**: Easy to add new selection strategies
- 6. **Warmup Flexibility**: Choose warmup strategy independently of selection strategy
- """)
- return
-
-
-if __name__ == "__main__":
- app.run()
diff --git a/tutorials/library_component_comparison.py b/tutorials/library_component_comparison.py
deleted file mode 100644
index 7d20d6d..0000000
--- a/tutorials/library_component_comparison.py
+++ /dev/null
@@ -1,888 +0,0 @@
-"""
-TACTICS Library Component Comparison App
-
-An interactive benchmark tool comparing Thompson Sampling behavior between
-2-component and 3-component representations of the Thrombin Linear Amide Library.
-
-This notebook explores whether modeling the library as a 3-component synthesis
-(amino acid + amino acid + carboxylic acid) produces different optimization
-behavior than the 2-component approach (dipeptide + carboxylic acid).
-
-Features:
-1. Side-by-side comparison of 2-component vs 3-component library representation
-2. Selection strategy configuration
-3. TS_Benchmarks visualizations for both library types
-4. Hit recovery analysis comparison
-
-Dataset: Thrombin Linear Amide Library (~500K products)
-
-Run as app: marimo run notebooks/library_component_comparison.py
-Edit mode: marimo edit notebooks/library_component_comparison.py
-"""
-
-import marimo
-
-__generated_with = "0.18.2"
-app = marimo.App(width="full", app_title="TACTICS Library Component Comparison")
-
-
-@app.cell
-def _():
- """Imports and project setup."""
- import marimo as mo
- import sys
- from pathlib import Path
-
- # Add TACTICS project paths
- try:
- project_root = Path(__file__).parent.parent.resolve()
- except NameError:
- project_root = Path("/Users/aakankschitnandkeolyar/Desktop/TACTICS")
-
- sys.path.insert(0, str(project_root))
- sys.path.insert(0, str(project_root / "src"))
-
- # Standard imports
- import polars as pl
- import time
-
- # TACTICS Thompson Sampling imports
- from TACTICS.thompson_sampling import ThompsonSampler
- from TACTICS.thompson_sampling.config import ThompsonSamplingConfig
- from TACTICS.thompson_sampling.strategies.config import (
- GreedyConfig,
- RouletteWheelConfig,
- UCBConfig,
- EpsilonGreedyConfig,
- BoltzmannConfig,
- )
- from TACTICS.thompson_sampling.warmup.config import (
- BalancedWarmupConfig,
- StandardWarmupConfig,
- EnhancedWarmupConfig,
- )
- from TACTICS.thompson_sampling.core.evaluator_config import LookupEvaluatorConfig
-
- # TACTICS Library Enumeration imports
- from TACTICS.library_enumeration import SynthesisPipeline
- from TACTICS.library_enumeration.smarts_toolkit import (
- ReactionConfig,
- ReactionDef,
- StepInput,
- InputSource,
- )
-
- # TACTICS Library Analysis imports
- from TACTICS.library_analysis.visualization import TS_Benchmarks
- return (
- BalancedWarmupConfig,
- BoltzmannConfig,
- EnhancedWarmupConfig,
- EpsilonGreedyConfig,
- GreedyConfig,
- InputSource,
- LookupEvaluatorConfig,
- ReactionConfig,
- ReactionDef,
- RouletteWheelConfig,
- StandardWarmupConfig,
- StepInput,
- SynthesisPipeline,
- TS_Benchmarks,
- ThompsonSampler,
- ThompsonSamplingConfig,
- UCBConfig,
- mo,
- pl,
- time,
- )
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- # TACTICS Library Component Comparison
-
- This notebook compares Thompson Sampling optimization behavior when the
- Thrombin Linear Amide Library is modeled as:
-
- - **2-Component Library**: Dipeptide + Carboxylic Acid (pre-coupled dipeptides)
- - **3-Component Library**: Amino Acid + Amino Acid + Carboxylic Acid (synthesized in-situ)
-
- Both representations describe the same ~500K product library, but the component
- structure affects how the Thompson Sampling algorithm explores the chemical space.
-
- **Key Question**: Does the 3-component representation provide better exploration
- or different optimization dynamics compared to the 2-component approach?
- """)
- return
-
-
-@app.cell
-def _():
- """Load bundled Thrombin dataset paths."""
- import importlib.resources
-
- _data_files = importlib.resources.files("TACTICS.data.thrombin")
- ACIDS_FILE = str(_data_files / "acids.smi")
- DIPEPTIDES_FILE = str(_data_files / "coupled_aa_sub.smi")
- AMINO_ACIDS_FILE = str(_data_files / "amino_acids_no_fmoc.smi")
- SCORES_FILE = str(_data_files / "product_scores.csv")
-
- # 2-Component Library Files
- REAGENT_FILES_2COMP = [ACIDS_FILE, DIPEPTIDES_FILE]
-
- # 3-Component Library Files
- REAGENT_FILES_3COMP = [AMINO_ACIDS_FILE, AMINO_ACIDS_FILE, ACIDS_FILE]
-
- # SMARTS patterns
- AMIDE_COUPLING_SMARTS = "[#6:1](=[O:2])[OH].[#7X3;H1,H2;!$(N[!#6]);!$(N[#6]=[O]);!$(N[#6]~[!#6;!#16]):3]>>[#6:1](=[O:2])[#7:3]"
- PEPTIDE_COUPLING_SMARTS = "[#6X3:1](=[#8X1])[OH].[#7X3;H1,H2;!$(N[!#6]);!$(N[#6]=[O]);!$(N[#6]~[!#6;!#16]):2]>>[#6X3:1](=[#8X1])[#7X3:2]"
- AMINE_SUBSTITUTION_SMARTS = "[#6:1](=O)[OH]>>[#6:1](=O)[NH2]"
- return (
- ACIDS_FILE,
- AMIDE_COUPLING_SMARTS,
- AMINE_SUBSTITUTION_SMARTS,
- AMINO_ACIDS_FILE,
- DIPEPTIDES_FILE,
- PEPTIDE_COUPLING_SMARTS,
- REAGENT_FILES_2COMP,
- REAGENT_FILES_3COMP,
- SCORES_FILE,
- )
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Configuration
-
- ### Selection Strategy
- """)
- return
-
-
-@app.cell
-def _(mo):
- """Strategy selection with parameter customization."""
- strategy_dropdown = mo.ui.dropdown(
- options=["Epsilon-Greedy", "Greedy", "UCB", "Roulette Wheel", "Boltzmann"],
- value="Epsilon-Greedy",
- label="Selection Strategy"
- )
-
- # Strategy parameters
- epsilon_value = mo.ui.slider(start=0.05, stop=0.5, value=0.2, step=0.05, label="Epsilon")
- epsilon_decay = mo.ui.slider(start=0.9, stop=1.0, value=0.995, step=0.005, label="Decay")
- ucb_c = mo.ui.slider(start=0.5, stop=4.0, value=2.0, step=0.5, label="UCB c")
- rw_alpha = mo.ui.slider(start=0.05, stop=0.5, value=0.1, step=0.05, label="RW alpha")
- boltz_temp = mo.ui.slider(start=0.1, stop=2.0, value=1.0, step=0.1, label="Temperature")
-
- # Warmup type dropdowns for each library
- warmup_type_2comp = mo.ui.dropdown(
- options=["Balanced", "Standard", "Enhanced"],
- value="Balanced",
- label="2-Component Warmup Type"
- )
- warmup_type_3comp = mo.ui.dropdown(
- options=["Balanced", "Standard", "Enhanced"],
- value="Balanced",
- label="3-Component Warmup Type"
- )
-
- # K sliders for balanced warmup (shown conditionally in display cell)
- warmup_k_2comp = mo.ui.slider(
- start=1, stop=20, value=3, step=1,
- label="K (obs/reagent)"
- )
- warmup_k_3comp = mo.ui.slider(
- start=1, stop=20, value=5, step=1,
- label="K (obs/reagent)"
- )
- return (
- boltz_temp,
- epsilon_decay,
- epsilon_value,
- rw_alpha,
- strategy_dropdown,
- ucb_c,
- warmup_k_2comp,
- warmup_k_3comp,
- warmup_type_2comp,
- warmup_type_3comp,
- )
-
-
-@app.cell
-def _(
- boltz_temp,
- epsilon_decay,
- epsilon_value,
- mo,
- rw_alpha,
- strategy_dropdown,
- ucb_c,
- warmup_k_2comp,
- warmup_k_3comp,
- warmup_type_2comp,
- warmup_type_3comp,
-):
- """Display strategy controls."""
- _strategy_params = mo.hstack([
- mo.vstack([mo.md("*Epsilon-Greedy:*"), epsilon_value, epsilon_decay]),
- mo.vstack([mo.md("*UCB:*"), ucb_c]),
- mo.vstack([mo.md("*Roulette Wheel:*"), rw_alpha]),
- mo.vstack([mo.md("*Boltzmann:*"), boltz_temp]),
- ], justify="start", gap=4)
-
- # Conditionally show K slider only when Balanced is selected
- _2comp_warmup_ui = (
- mo.vstack([warmup_type_2comp, warmup_k_2comp])
- if warmup_type_2comp.value == "Balanced"
- else mo.vstack([warmup_type_2comp])
- )
- _3comp_warmup_ui = (
- mo.vstack([warmup_type_3comp, warmup_k_3comp])
- if warmup_type_3comp.value == "Balanced"
- else mo.vstack([warmup_type_3comp])
- )
-
- _warmup_controls = mo.hstack([
- mo.vstack([mo.md("**2-Component Warmup:**"), _2comp_warmup_ui]),
- mo.vstack([mo.md("**3-Component Warmup:**"), _3comp_warmup_ui]),
- ], justify="start", gap=4)
-
- _controls = mo.vstack([
- strategy_dropdown,
- _warmup_controls,
- mo.md("**Strategy Parameters:**"),
- _strategy_params,
- ])
- _controls
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Search Parameters
- """)
- return
-
-
-@app.cell
-def _(mo):
- """Search parameters."""
- iterations_slider = mo.ui.slider(
- start=100, stop=2000, value=500, step=100,
- label="Iterations per cycle"
- )
- cycles_slider = mo.ui.slider(
- start=1, stop=10, value=3, step=1,
- label="Benchmark cycles"
- )
- top_n_slider = mo.ui.slider(
- start=50, stop=500, value=100, step=50,
- label="Top N for recovery"
- )
- return cycles_slider, iterations_slider, top_n_slider
-
-
-@app.cell(hide_code=True)
-def _(cycles_slider, iterations_slider, mo, top_n_slider):
- """Display search parameter controls."""
- mo.hstack([iterations_slider, cycles_slider, top_n_slider], justify="start", gap=2)
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Run Comparison Benchmark
- """)
- return
-
-
-@app.cell
-def _(mo):
- """Run button."""
- run_button = mo.ui.run_button(label="Run Library Comparison Benchmark")
- run_button
- return (run_button,)
-
-
-@app.cell
-def _(
- ACIDS_FILE,
- AMIDE_COUPLING_SMARTS,
- AMINE_SUBSTITUTION_SMARTS,
- AMINO_ACIDS_FILE,
- DIPEPTIDES_FILE,
- InputSource,
- PEPTIDE_COUPLING_SMARTS,
- ReactionConfig,
- ReactionDef,
- StepInput,
- SynthesisPipeline,
-):
- """Create synthesis pipelines for both library representations."""
- # 2-Component Pipeline: Dipeptide + Carboxylic Acid
- reaction_config_2comp = ReactionConfig(
- reactions=[ReactionDef(reaction_smarts=AMIDE_COUPLING_SMARTS, step_index=0)],
- reagent_file_list=[ACIDS_FILE, DIPEPTIDES_FILE],
- )
- pipeline_2comp = SynthesisPipeline(reaction_config_2comp)
-
- # 3-Component Pipeline: Carboxylic Acid + Amino Acid + Amino Acid
- # Note: Reagent order is [Acid, AA1, AA2] to match scores file naming (CA_AA_AA)
- reaction_config_3comp = ReactionConfig(
- reactions=[
- # Step 0: amino acid + amino acid → dipeptide
- ReactionDef(
- reaction_smarts=PEPTIDE_COUPLING_SMARTS,
- step_index=0,
- description="peptide coupling",
- ),
- # Step 1: hydroxyl → amine substitution
- ReactionDef(
- reaction_smarts=AMINE_SUBSTITUTION_SMARTS,
- step_index=1,
- description="Amine substitution",
- ),
- # Step 2: dipeptide + carboxylic acid → final product
- ReactionDef(
- reaction_smarts=AMIDE_COUPLING_SMARTS,
- step_index=2,
- description="linear amide coupling",
- ),
- ],
- reagent_file_list=[
- ACIDS_FILE, # Index 0: Carboxylic acids (for naming: CA_...)
- AMINO_ACIDS_FILE, # Index 1: First amino acid
- AMINO_ACIDS_FILE, # Index 2: Second amino acid
- ],
- step_inputs={
- 0: [
- StepInput(source=InputSource.REAGENT_FILE, file_index=1), # First AA
- StepInput(source=InputSource.REAGENT_FILE, file_index=2), # Second AA
- ],
- 1: [
- StepInput(source=InputSource.PREVIOUS_STEP, step_index=0)
- ],
- 2: [
- StepInput(source=InputSource.REAGENT_FILE, file_index=0), # Acid
- StepInput(source=InputSource.PREVIOUS_STEP, step_index=1)
- ],
- },
- )
- pipeline_3comp = SynthesisPipeline(reaction_config_3comp)
-
- print(f"2-Component Pipeline: {pipeline_2comp.num_components} components, {pipeline_2comp.num_steps} steps")
- print(f"3-Component Pipeline: {pipeline_3comp.num_components} components, {pipeline_3comp.num_steps} steps")
- return pipeline_2comp, pipeline_3comp, reaction_config_2comp, reaction_config_3comp
-
-
-@app.cell
-def _(
- BalancedWarmupConfig,
- BoltzmannConfig,
- EnhancedWarmupConfig,
- EpsilonGreedyConfig,
- GreedyConfig,
- LookupEvaluatorConfig,
- RouletteWheelConfig,
- SCORES_FILE,
- StandardWarmupConfig,
- ThompsonSampler,
- ThompsonSamplingConfig,
- UCBConfig,
- boltz_temp,
- cycles_slider,
- epsilon_decay,
- epsilon_value,
- iterations_slider,
- mo,
- pipeline_2comp,
- pipeline_3comp,
- pl,
- run_button,
- rw_alpha,
- strategy_dropdown,
- time,
- ucb_c,
- warmup_k_2comp,
- warmup_k_3comp,
- warmup_type_2comp,
- warmup_type_3comp,
-):
- """Execute Thompson Sampling benchmark for both library representations."""
- mo.stop(not run_button.value, mo.md("*Click 'Run Library Comparison Benchmark' to start*"))
-
- # Build strategy config based on selection
- strategy_name = strategy_dropdown.value
- if strategy_name == "Epsilon-Greedy":
- strategy_config = EpsilonGreedyConfig(
- mode="minimize", epsilon=epsilon_value.value, decay=epsilon_decay.value
- )
- elif strategy_name == "Greedy":
- strategy_config = GreedyConfig(mode="minimize")
- elif strategy_name == "UCB":
- strategy_config = UCBConfig(mode="minimize", c=ucb_c.value)
- elif strategy_name == "Roulette Wheel":
- strategy_config = RouletteWheelConfig(
- mode="minimize", alpha=rw_alpha.value, beta=rw_alpha.value
- )
- else: # Boltzmann
- strategy_config = BoltzmannConfig(
- mode="minimize", temperature=boltz_temp.value
- )
-
- # Build warmup config for 2-component library
- _warmup_type_2comp = warmup_type_2comp.value
- _k_2comp = warmup_k_2comp.value
- if _warmup_type_2comp == "Balanced":
- warmup_config_2comp = BalancedWarmupConfig(observations_per_reagent=_k_2comp)
- _warmup_label_2comp = f"Balanced-K{_k_2comp}"
- elif _warmup_type_2comp == "Standard":
- warmup_config_2comp = StandardWarmupConfig()
- _warmup_label_2comp = "Standard"
- else: # Enhanced
- warmup_config_2comp = EnhancedWarmupConfig()
- _warmup_label_2comp = "Enhanced"
-
- # Build warmup config for 3-component library
- _warmup_type_3comp = warmup_type_3comp.value
- _k_3comp = warmup_k_3comp.value
- if _warmup_type_3comp == "Balanced":
- warmup_config_3comp = BalancedWarmupConfig(observations_per_reagent=_k_3comp)
- _warmup_label_3comp = f"Balanced-K{_k_3comp}"
- elif _warmup_type_3comp == "Standard":
- warmup_config_3comp = StandardWarmupConfig()
- _warmup_label_3comp = "Standard"
- else: # Enhanced
- warmup_config_3comp = EnhancedWarmupConfig()
- _warmup_label_3comp = "Enhanced"
-
- # Evaluator config
- evaluator_config = LookupEvaluatorConfig(
- ref_filename=SCORES_FILE,
- compound_col="Product_Code",
- score_col="Scores",
- )
-
- # Run parameters
- _num_cycles = cycles_slider.value
- _num_iterations = iterations_slider.value
-
- # Results storage
- results_2comp = {}
- results_3comp = {}
- start_time = time.time()
-
- # --- Run 2-Component Library Benchmark ---
- _cycle_results_2comp = []
- for _cycle in range(_num_cycles):
- _config = ThompsonSamplingConfig(
- synthesis_pipeline=pipeline_2comp,
- num_ts_iterations=_num_iterations,
- num_warmup_trials=3,
- strategy_config=strategy_config,
- warmup_config=warmup_config_2comp,
- evaluator_config=evaluator_config,
- batch_size=1,
- max_resamples=1000,
- hide_progress=True,
- )
-
- _sampler = ThompsonSampler.from_config(_config)
- _warmup_df = _sampler.warm_up(num_warmup_trials=_config.num_warmup_trials)
- _search_df = _sampler.search(num_cycles=_config.num_ts_iterations)
-
- _combined = pl.concat([_warmup_df, _search_df])
- _result = _combined.select(["Name", "score"])
- _cycle_results_2comp.append(_result)
- _sampler.close()
-
- results_2comp[f"2-Comp ({_warmup_label_2comp})"] = _cycle_results_2comp
-
- # --- Run 3-Component Library Benchmark ---
- _cycle_results_3comp = []
- for _cycle in range(_num_cycles):
- _config = ThompsonSamplingConfig(
- synthesis_pipeline=pipeline_3comp,
- num_ts_iterations=_num_iterations,
- num_warmup_trials=3,
- strategy_config=strategy_config,
- warmup_config=warmup_config_3comp,
- evaluator_config=evaluator_config,
- batch_size=1,
- max_resamples=1000,
- hide_progress=True,
- )
-
- _sampler = ThompsonSampler.from_config(_config)
- _warmup_df = _sampler.warm_up(num_warmup_trials=_config.num_warmup_trials)
- _search_df = _sampler.search(num_cycles=_config.num_ts_iterations)
-
- _combined = pl.concat([_warmup_df, _search_df])
- _result = _combined.select(["Name", "score"])
- _cycle_results_3comp.append(_result)
- _sampler.close()
-
- results_3comp[f"3-Comp ({_warmup_label_3comp})"] = _cycle_results_3comp
-
- total_time = time.time() - start_time
-
- # Compute summary statistics
- _best_2comp = [r["score"].min() for r in _cycle_results_2comp]
- _best_3comp = [r["score"].min() for r in _cycle_results_3comp]
-
- run_metadata = {
- "strategy": strategy_name,
- "warmup_2comp": _warmup_label_2comp,
- "warmup_3comp": _warmup_label_3comp,
- "cycles": _num_cycles,
- "iterations": _num_iterations,
- "2comp_best": min(_best_2comp),
- "2comp_avg_best": sum(_best_2comp) / len(_best_2comp),
- "3comp_best": min(_best_3comp),
- "3comp_avg_best": sum(_best_3comp) / len(_best_3comp),
- }
- return (
- evaluator_config,
- results_2comp,
- results_3comp,
- run_metadata,
- strategy_config,
- strategy_name,
- total_time,
- )
-
-
-@app.cell
-def _(mo, run_button, run_metadata, total_time):
- """Display results summary."""
- mo.stop(not run_button.value)
-
- mo.md(f"""
- ---
- ## Results Summary
-
- **Strategy:** {run_metadata['strategy']}
-
- **Total runtime:** {total_time:.1f}s ({run_metadata['cycles']} cycles × {run_metadata['iterations']} iterations)
-
- | Library Type | Warmup | Best Score | Avg Best Score |
- |--------------|--------|------------|----------------|
- | 2-Component | {run_metadata['warmup_2comp']} | {run_metadata['2comp_best']:.3f} | {run_metadata['2comp_avg_best']:.3f} |
- | 3-Component | {run_metadata['warmup_3comp']} | {run_metadata['3comp_best']:.3f} | {run_metadata['3comp_avg_best']:.3f} |
-
- **Difference:** {abs(run_metadata['2comp_avg_best'] - run_metadata['3comp_avg_best']):.3f}
- ({"3-Component better" if run_metadata['3comp_avg_best'] < run_metadata['2comp_avg_best'] else "2-Component better"})
- """)
- return
-
-
-@app.cell
-def _(SCORES_FILE, mo, pl, run_button):
- """Load reference data for visualizations."""
- mo.stop(not run_button.value)
-
- reference_data = (
- pl.read_csv(SCORES_FILE)
- .rename({"Product_Code": "Name", "Scores": "score"})
- .select(["Name", "score"])
- )
- return (reference_data,)
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Comparison Visualizations
-
- ### Strip Plot: Score Distributions
- """)
- return
-
-
-@app.cell
-def _(
- TS_Benchmarks,
- cycles_slider,
- mo,
- reference_data,
- results_2comp,
- results_3comp,
- run_button,
- top_n_slider,
-):
- """Create combined TS_Benchmarks for side-by-side comparison."""
- mo.stop(not run_button.value)
-
- # Merge results from both libraries
- combined_results = {**results_2comp, **results_3comp}
- combined_methods = list(combined_results.keys())
-
- benchmarks_combined = TS_Benchmarks(
- no_of_cycles=cycles_slider.value,
- methods_list=combined_methods,
- TS_runs_data=combined_results,
- reference_data=reference_data,
- top_n=top_n_slider.value,
- sort_type="minimize",
- top_ns=[25, 50, 100, 200, 300],
- )
- return benchmarks_combined, combined_methods, combined_results
-
-
-@app.cell
-def _(benchmarks_combined, mo, run_button):
- """Generate combined strip plot."""
- mo.stop(not run_button.value)
-
- benchmarks_combined.stripplot_TS_results(
- width=900,
- height=450,
- show_plot=True,
- legend_position="bottom"
- )
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Bar Plot: Hit Recovery by Cycle
- """)
- return
-
-
-@app.cell
-def _(benchmarks_combined, mo, run_button):
- """Generate combined bar plot."""
- mo.stop(not run_button.value)
-
- benchmarks_combined.plot_barplot_TS_results(
- width=900,
- height=450,
- show_plot=True,
- legend_position="bottom",
- dark_mode=False
- )
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Line Plot: Performance vs Top-N
- """)
- return
-
-
-@app.cell
-def _(benchmarks_combined, mo, run_button):
- """Generate combined line plot."""
- mo.stop(not run_button.value)
-
- benchmarks_combined.plot_line_performance_with_error_bars(
- width=900,
- height=450,
- show_plot=True,
- legend_position="bottom"
- )
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ### Performance Statistics
- """)
- return
-
-
-@app.cell(hide_code=True)
-def _(benchmarks_combined, mo, run_button):
- """Display grouped statistics."""
- mo.stop(not run_button.value)
-
- _output = (
- mo.vstack([
- mo.md("**Mean Performance by Top-N:**"),
- benchmarks_combined.grouped_stats,
- ])
- if benchmarks_combined.grouped_stats is not None
- else mo.md("*Statistics not available*")
- )
- _output
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Top Compounds Found
- """)
- return
-
-
-@app.cell
-def _(combined_results, mo, pl, run_button):
- """Display top compounds from both library representations."""
- mo.stop(not run_button.value)
-
- _all_dfs = []
- for _name, _results in combined_results.items():
- for _i, _df in enumerate(_results):
- _labeled = _df.with_columns([
- pl.lit(_name).alias("library_type"),
- pl.lit(_i + 1).alias("cycle"),
- ])
- _all_dfs.append(_labeled)
-
- combined_df = pl.concat(_all_dfs)
- top_compounds = (
- combined_df
- .sort("score")
- .unique(subset=["Name"], keep="first")
- .head(20)
- .select(["Name", "score", "library_type", "cycle"])
- )
- top_compounds
- return combined_df, top_compounds
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Additivity Analysis
- """)
- return
-
-
-@app.cell
-def _(SCORES_FILE, mo, pl, run_button):
- """Compute additivity metrics to explain performance differences."""
- mo.stop(not run_button.value)
-
- # Load and parse scores
- _scores = pl.read_csv(SCORES_FILE)
- _scores = _scores.with_columns([
- pl.col("Product_Code").str.split("_").list.get(0).alias("Acid"),
- pl.col("Product_Code").str.split("_").list.get(1).alias("AA1"),
- pl.col("Product_Code").str.split("_").list.get(2).alias("AA2"),
- (pl.col("Product_Code").str.split("_").list.get(1) + "_" +
- pl.col("Product_Code").str.split("_").list.get(2)).alias("Dipeptide"),
- ])
-
- # Calculate variance components
- _total_var = _scores["Scores"].var()
- _acid_var = _scores.group_by("Acid").agg(pl.col("Scores").mean())["Scores"].var()
- _aa1_var = _scores.group_by("AA1").agg(pl.col("Scores").mean())["Scores"].var()
- _aa2_var = _scores.group_by("AA2").agg(pl.col("Scores").mean())["Scores"].var()
- _dip_var = _scores.group_by("Dipeptide").agg(pl.col("Scores").mean())["Scores"].var()
-
- _additive_var = _aa1_var + _aa2_var
- _interaction_ratio = _dip_var / _additive_var if _additive_var > 0 else float('inf')
-
- # Determine recommendation
- if _interaction_ratio > 1.3:
- _recommendation = "**Strong interactions detected** → 2-Component representation recommended"
- _color = "red"
- elif _interaction_ratio > 1.0:
- _recommendation = "**Moderate interactions** → 2-Component likely better"
- _color = "orange"
- else:
- _recommendation = "**Approximately additive** → 3-Component may be viable"
- _color = "green"
-
- mo.md(f"""
- ### Variance Component Analysis
-
- | Component | Variance | % of Total |
- |-----------|----------|------------|
- | Total Score | {_total_var:.4f} | 100% |
- | Acid (mean per acid) | {_acid_var:.4f} | {100*_acid_var/_total_var:.1f}% |
- | AA1 (mean per AA1) | {_aa1_var:.4f} | {100*_aa1_var/_total_var:.1f}% |
- | AA2 (mean per AA2) | {_aa2_var:.4f} | {100*_aa2_var/_total_var:.1f}% |
- | Dipeptide (mean per dipeptide) | {_dip_var:.4f} | {100*_dip_var/_total_var:.1f}% |
-
- ### Additivity Test
-
- - **Sum of individual AA variances**: {_additive_var:.4f}
- - **Dipeptide variance**: {_dip_var:.4f}
- - **Interaction ratio** (dipeptide / sum): **{_interaction_ratio:.2f}**
-
- {_recommendation}
-
- *Ratio > 1.0 indicates interaction effects beyond what individual amino acids explain.*
- """)
- return
-
-
-@app.cell(hide_code=True)
-def _(mo):
- mo.md(r"""
- ---
- ## Understanding the Comparison
-
- ### Library Representations
-
- | Aspect | 2-Component | 3-Component |
- |--------|-------------|-------------|
- | **Reagent Files** | Dipeptides (3844) + Acids (130) | Amino Acids (62) × 2 + Acids (130) |
- | **Posteriors Tracked** | 130 + 3844 = **3974** | 130 + 62 + 62 = **254** |
- | **Synthesis Steps** | 1 (amide coupling) | 3 (peptide → amine sub → amide) |
- | **Implicit Assumption** | Dipeptides are atomic units | AA contributions are additive |
-
- ### Why 3-Component May Underperform
-
- Thompson Sampling with independent posteriors assumes **additivity**:
- ```
- score ≈ f(acid) + g(AA1) + h(AA2)
- ```
-
- But if there are strong **interaction effects** between amino acids:
- ```
- score ≈ f(acid) + interaction(AA1, AA2)
- ```
-
- The 3-component model fails because:
-
- 1. **Information Loss**: When `CA5_AA32_AA61` scores well, the model credits AA32 and AA61
- independently. But AA32 may only be good *with* AA61, not generally.
-
- 2. **Noisy Posteriors**: Each amino acid's posterior is updated based on combinations with
- many different partners, creating conflicting signals.
-
- 3. **Fewer Parameters**: 254 posteriors must capture 499,720 product preferences, vs
- 3974 posteriors in the 2-component case.
-
- ### Additivity Test for Your Data
-
- To determine which representation is appropriate, check:
- - **Dipeptide variance** vs **AA1 variance + AA2 variance**
- - If dipeptide variance >> sum of individual variances → strong interactions → use 2-component
- - If roughly equal → additive → 3-component may work
-
- For Thrombin data: Dipeptide variance (0.89) >> AA1 (0.25) + AA2 (0.31) = 0.56
- → **Strong interactions exist** → 2-component is more appropriate
-
- ### Usage Recommendations
-
- - **2-Component**: Use when component interactions matter (most drug discovery cases)
- - **3-Component**: Use when contributions are approximately additive, or for initial
- exploration to identify promising component classes before detailed optimization
- """)
- return
-
-
-if __name__ == "__main__":
- app.run()
diff --git a/tutorials/manuscript_plots.py b/tutorials/manuscript_plots.py
deleted file mode 100644
index 338ebe2..0000000
--- a/tutorials/manuscript_plots.py
+++ /dev/null
@@ -1,3218 +0,0 @@
-"""
-TACTICS Manuscript Plots
-
-Loads the manuscript benchmark results (109,050 trials, 5 methods, 21 libraries)
-plus thrombin docking recovery data, and provides DataFrames ready for plotting.
-
-Run as app: marimo run tutorials/manuscript_plots.py
-Edit mode: marimo edit tutorials/manuscript_plots.py
-"""
-
-import marimo
-
-__generated_with = "0.21.1"
-app = marimo.App(width="full", app_title="TACTICS Manuscript Plots")
-
-
-@app.cell
-def _():
- """Imports."""
- import io
- import marimo as mo
- import sys
- from pathlib import Path
-
- import matplotlib.pyplot as plt
- from matplotlib.lines import Line2D
- from matplotlib.patches import Patch
- import numpy as np
- import polars as pl
- from scipy import stats
-
- def fig_to_pdf_bytes(fig):
- """Convert a matplotlib figure to PDF bytes in memory."""
- buf = io.BytesIO()
- fig.savefig(buf, format="pdf", bbox_inches="tight", dpi=300)
- return buf.getvalue()
-
- try:
- project_root = Path(__file__).parent.parent.resolve()
- except NameError:
- project_root = Path.cwd()
-
- sys.path.insert(0, str(project_root))
- sys.path.insert(0, str(project_root / "src"))
- return (
- Line2D,
- Patch,
- fig_to_pdf_bytes,
- mo,
- np,
- pl,
- plt,
- project_root,
- stats,
- )
-
-
-@app.cell
-def _(mo):
- """Top-N slider for filtering pre-computed recovery data."""
- topn_slider = mo.ui.slider(
- start=25, stop=250, step=25, value=100,
- label="Top-N Recovery Threshold",
- )
- topn_slider
- return (topn_slider,)
-
-
-@app.cell
-def _(mo, pl, project_root, topn_slider):
- """Load pre-computed recovery and statistical analysis results."""
- analysis_dir = project_root / "outputs" / "manuscript_analysis" / "post_hoc"
-
- TOP_N = topn_slider.value
-
- # --- Recovery summary (all libraries × all top-N thresholds) ---
- recovery_all = pl.read_parquet(analysis_dir / "recovery_summary.parquet")
-
- # Filter to selected top-N threshold
- df = recovery_all.filter(pl.col("top_n") == TOP_N)
-
- # --- Statistical test results ---
- tukey_df = pl.read_parquet(analysis_dir / "tukey_hsd_results.parquet")
- tukey_at_n = tukey_df.filter(pl.col("top_n") == TOP_N)
-
- anova_df = pl.read_parquet(analysis_dir / "anova_results.parquet")
- anova_at_n = anova_df.filter(pl.col("top_n") == TOP_N)
-
- n_trials = len(df)
- n_libraries = df["library_id"].n_unique()
- libraries = sorted(df["library_id"].unique().to_list())
- methods = sorted(df["method"].unique().to_list())
-
- mo.md(
- f"""
- ## Data Loaded (Pre-Computed Post-Hoc Analysis)
-
- - **Source**: `{analysis_dir}`
- - **Top-N threshold**: {TOP_N} (use slider to change)
- - **Available thresholds**: {sorted(recovery_all['top_n'].unique().to_list())}
- - **Total trials at top-{TOP_N}**: {n_trials:,}
- - **Libraries**: {n_libraries} ({', '.join(libraries)})
- - **Methods**: {', '.join(methods)}
- - **Replicates**: {df['replicate'].n_unique()}
- - **Tukey HSD comparisons**: {len(tukey_at_n):,} ({tukey_at_n.filter(pl.col('reject')).height} significant)
- - **ANOVA tests**: {len(anova_at_n)} ({anova_at_n.filter(pl.col('significance') != 'ns').height} significant)
- """
- )
- return TOP_N, df, recovery_all, tukey_at_n
-
-
-@app.cell
-def _(df, mo, pl):
- """Overall method summary."""
- method_summary = (
- df.group_by("method")
- .agg([
- pl.col("recovered").mean().round(2).alias("mean_recovery"),
- pl.col("recovered").std().round(2).alias("std_recovery"),
- pl.col("recovered").median().alias("median_recovery"),
- pl.col("n_evaluated").mean().round(0).alias("mean_evaluations"),
- ])
- .sort("mean_recovery", descending=True)
- )
-
- mo.md("## Overall Method Summary")
- mo.ui.table(method_summary.to_pandas())
- return
-
-
-@app.cell
-def _(df, mo, pl):
- """Per-library per-method mean recovery (wide format for plotting)."""
- lib_method = (
- df.group_by(["library_id", "n_components", "method"])
- .agg(pl.col("recovered").mean().round(1).alias("mean_recovery"))
- .sort(["library_id", "method"])
- )
-
- # Wide format: one column per method
- lib_wide = lib_method.pivot(
- on="method",
- index=["library_id", "n_components"],
- values="mean_recovery",
- ).sort("library_id")
-
- mo.vstack([
- mo.md("## Per-Library Recovery (Wide Format)"),
- mo.ui.table(lib_wide.to_pandas()),
- ])
- return
-
-
-@app.cell
-def _(df, mo, pl):
- """Per-library per-method recovery with std (for error bars)."""
- lib_method_stats = (
- df.group_by(["library_id", "n_components", "method"])
- .agg([
- pl.col("recovered").mean().round(2).alias("mean_recovery"),
- pl.col("recovered").std().round(2).alias("std_recovery"),
- pl.col("recovered").count().alias("n_trials"),
- ])
- .with_columns(
- (pl.col("std_recovery") / pl.col("n_trials").sqrt())
- .round(2)
- .alias("se_recovery")
- )
- .sort(["library_id", "method"])
- )
-
- mo.vstack([
- mo.md("## Per-Library Recovery with Error Bars"),
- mo.ui.table(lib_method_stats.to_pandas()),
- ])
- return
-
-
-@app.cell
-def _(df, mo, pl):
- """Component-count breakdown."""
- component_summary = (
- df.group_by(["n_components", "method"])
- .agg([
- pl.col("recovered").mean().round(2).alias("mean_recovery"),
- pl.col("recovered").std().round(2).alias("std_recovery"),
- ])
- .sort(["n_components", "mean_recovery"], descending=[False, True])
- )
-
- mo.vstack([
- mo.md("## Component-Count Breakdown"),
- mo.ui.table(component_summary.to_pandas()),
- ])
- return
-
-
-@app.cell
-def _(mo, tukey_at_n):
- """Pairwise significance tests (Tukey HSD, pre-computed)."""
- mo.vstack([
- mo.md(
- """
- ## Pairwise Significance Tests (Tukey HSD)
-
- FWER-controlled pairwise comparisons.
- Following Ash et al. 2025 (JCIM) guidelines.
- """
- ),
- mo.ui.table(
- tukey_at_n.select([
- "library_id", "n_components", "group1", "group2",
- "mean_diff", "p_adj", "significance", "ci_low", "ci_high",
- "n_pairs",
- ]).to_pandas()
- ),
- ])
- return
-
-
-@app.cell
-def _(mo):
- mo.md(r"""
- ## Per-Library Recovery Differences: TACTICS vs Legacy Baselines
-
- Grouped bar plots showing per-library Δ recovery for **TT-TS** and **RWS (GMIC)**
- relative to the appropriate legacy baseline:
- - **ROCS**: baseline = Legacy Enhanced-RWS
- - **Docking**: baseline = Legacy Standard-Greedy
-
- Each panel has independent y-axes so smaller gains remain visible.
- """)
- return
-
-
-@app.cell
-def _(Patch, fig_to_pdf_bytes, mo, np, pl, plt, tukey_at_n):
- """Per-library grouped difference bars — ROCS (baseline = Legacy RWS)."""
-
- _DOCKING_LIBS = {"adenine", "quinazoline", "thrombin", "amide"}
- _COMP_COUNTS = {
- "amide-suzuki": 3, "betti": 3, "dobener": 3,
- "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
- "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
- "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2,
- "rxn114b": 2, "rxn203": 2, "rxn205": 2, "rxn206": 2,
- "rxn207": 2, "rxn208": 2,
- "adenine": 3, "quinazoline": 3, "thrombin": 2, "amide": 2,
- }
- _baseline = "Legacy Enhanced-RWS"
- _challengers = [
- ("TACTICS Enhanced-TT-TS (GMIC)", "TT-TS", "#4C78A8"),
- ("TACTICS Enhanced-RWS (GMIC)", "RWS (GMIC)", "#F28E2B"),
- ]
-
- def _get_diffs(_baseline_name, _challenger_name, _tukey):
- """Extract per-library diffs from Tukey HSD, normalized so positive = challenger better."""
- _match = _tukey.filter(
- ((pl.col("group1") == _baseline_name) & (pl.col("group2") == _challenger_name))
- | ((pl.col("group1") == _challenger_name) & (pl.col("group2") == _baseline_name))
- )
- _rows = {}
- for _r in _match.iter_rows(named=True):
- _lib = _r["library_id"]
- _nc = _COMP_COUNTS.get(_lib, _r["n_components"])
- if _r["group1"] == _baseline_name:
- _rows[_lib] = {"mean_diff": _r["mean_diff"], "ci_low": _r["ci_low"],
- "ci_high": _r["ci_high"], "sig": _r["significance"], "n_comp": _nc}
- else:
- _rows[_lib] = {"mean_diff": -_r["mean_diff"], "ci_low": -_r["ci_high"],
- "ci_high": -_r["ci_low"], "sig": _r["significance"], "n_comp": _nc}
- return _rows
-
- # Collect diffs for both challengers
- _all_diffs = {}
- for _ch_method, _ch_label, _ch_color in _challengers:
- _all_diffs[_ch_label] = _get_diffs(_baseline, _ch_method, tukey_at_n)
-
- # ROCS only
- _rocs_libs = sorted([lib for lib in _all_diffs["TT-TS"] if lib not in _DOCKING_LIBS])
-
- _fig, _axes = plt.subplots(1, 2, figsize=(18, 7), facecolor="white")
- _bar_width = 0.35
-
- for _ax, _nc in zip(_axes, [2, 3]):
- _ax.set_facecolor("white")
- _libs = [lib for lib in _rocs_libs if _COMP_COUNTS.get(lib, 0) == _nc]
- _libs.sort(key=lambda lib: _all_diffs["TT-TS"].get(lib, {}).get("mean_diff", 0), reverse=True)
- _n = len(_libs)
-
- if _n == 0:
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14,
- fontweight="bold", color="black")
- _ax.text(0.5, 0.5, "No data", ha="center", va="center",
- fontsize=14, color="grey", transform=_ax.transAxes)
- _ax.set_xticks([])
- continue
-
- _x = np.arange(_n)
- _all_star_tops = []
- _all_star_bots = []
-
- for _gi, (_ch_method, _ch_label, _ch_color) in enumerate(_challengers):
- _diffs = _all_diffs[_ch_label]
- _means = np.array([_diffs.get(lib, {}).get("mean_diff", 0) for lib in _libs])
- _ci_lo = np.array([_diffs.get(lib, {}).get("ci_low", 0) for lib in _libs])
- _ci_hi = np.array([_diffs.get(lib, {}).get("ci_high", 0) for lib in _libs])
- _sigs = [_diffs.get(lib, {}).get("sig", "ns") for lib in _libs]
- _yerr = np.array([_means - _ci_lo, _ci_hi - _means])
-
- _offset = (_gi - 0.5) * _bar_width
- _ax.bar(
- _x + _offset, _means, width=_bar_width, yerr=_yerr,
- color=_ch_color, edgecolor="white", linewidth=0.5,
- capsize=3, error_kw={"linewidth": 1.0, "color": "black"},
- zorder=3, label=_ch_label,
- )
-
- _all_star_tops.extend(_ci_hi.tolist())
- _all_star_bots.extend(_ci_lo.tolist())
-
- # Significance stars
- _star_offset = max(abs(_ci_hi.max()), abs(_ci_lo.min()), 1) * 0.04
- for _i, (_sig, _ci_h, _ci_l, _m) in enumerate(zip(_sigs, _ci_hi, _ci_lo, _means)):
- _y_pos = _ci_h + _star_offset if _m >= 0 else _ci_l - _star_offset
- _va = "bottom" if _m >= 0 else "top"
- _ax.text(_x[_i] + _offset, _y_pos, _sig, ha="center", va=_va,
- fontsize=12, fontweight="bold", color="black", rotation=90)
-
- # Set y-limits with enough room for stars
- _max_top = max(_all_star_tops) if _all_star_tops else 1
- _min_bot = min(_all_star_bots) if _all_star_bots else 0
- _range = max(abs(_max_top), abs(_min_bot), 1)
- _ax.set_ylim(_min_bot - _range * 0.3, _max_top + _range * 0.3)
-
- _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
- _ax.set_xticks(_x)
- _ax.set_xticklabels(_libs, rotation=45, ha="right", fontsize=12, color="black")
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14, fontweight="bold", color="black")
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="y", alpha=0.3, zorder=1, color="grey")
- _ax.set_xlim(-0.6, _n - 0.4)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _axes[0].set_ylabel("\u0394 % Recovery of Top-N Hits\n(vs Legacy RWS)", fontsize=14, color="black")
-
- _legend_elements = [
- Patch(facecolor=c, edgecolor="black", label=l)
- for _, l, c in _challengers
- ]
- _axes[1].legend(handles=_legend_elements, loc="upper right", fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black")
-
- _fig.suptitle("Per-Library Recovery Difference: TACTICS vs Legacy RWS (ROCS Libraries)",
- fontsize=15, fontweight="bold", color="black", y=1.01)
- _fig.tight_layout()
-
- mo.vstack([
- _fig,
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="per_library_diff_rocs.pdf",
- mimetype="application/pdf",
- label="Download ROCS as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(Patch, fig_to_pdf_bytes, mo, np, pl, plt, tukey_at_n):
- """Per-library grouped difference bars — Docking (baseline = Legacy Greedy)."""
-
- _DOCKING_LIBS = {"adenine", "quinazoline", "thrombin", "amide"}
- _COMP_COUNTS = {
- "adenine": 3, "quinazoline": 3, "thrombin": 2, "amide": 2,
- }
- _baseline = "Legacy Standard-Greedy"
- _challengers = [
- ("TACTICS Enhanced-TT-TS (GMIC)", "TT-TS", "#4C78A8"),
- ("TACTICS Enhanced-RWS (GMIC)", "RWS (GMIC)", "#F28E2B"),
- ]
-
- def _get_diffs(_baseline_name, _challenger_name, _tukey):
- _match = _tukey.filter(
- ((pl.col("group1") == _baseline_name) & (pl.col("group2") == _challenger_name))
- | ((pl.col("group1") == _challenger_name) & (pl.col("group2") == _baseline_name))
- )
- _rows = {}
- for _r in _match.iter_rows(named=True):
- _lib = _r["library_id"]
- _nc = _COMP_COUNTS.get(_lib, _r["n_components"])
- if _r["group1"] == _baseline_name:
- _rows[_lib] = {"mean_diff": _r["mean_diff"], "ci_low": _r["ci_low"],
- "ci_high": _r["ci_high"], "sig": _r["significance"], "n_comp": _nc}
- else:
- _rows[_lib] = {"mean_diff": -_r["mean_diff"], "ci_low": -_r["ci_high"],
- "ci_high": -_r["ci_low"], "sig": _r["significance"], "n_comp": _nc}
- return _rows
-
- _all_diffs = {}
- for _ch_method, _ch_label, _ch_color in _challengers:
- _all_diffs[_ch_label] = _get_diffs(_baseline, _ch_method, tukey_at_n)
-
- _dock_libs = sorted([lib for lib in _all_diffs["TT-TS"] if lib in _DOCKING_LIBS])
-
- _fig, _axes = plt.subplots(1, 2, figsize=(18, 7), facecolor="white")
- _bar_width = 0.35
-
- for _ax, _nc in zip(_axes, [2, 3]):
- _ax.set_facecolor("white")
- _libs = [lib for lib in _dock_libs if _COMP_COUNTS.get(lib, 0) == _nc]
- _libs.sort(key=lambda lib: _all_diffs["TT-TS"].get(lib, {}).get("mean_diff", 0), reverse=True)
- _n = len(_libs)
-
- if _n == 0:
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14,
- fontweight="bold", color="black")
- _ax.text(0.5, 0.5, "No data", ha="center", va="center",
- fontsize=14, color="grey", transform=_ax.transAxes)
- _ax.set_xticks([])
- continue
-
- _x = np.arange(_n)
- _all_star_tops = []
- _all_star_bots = []
-
- for _gi, (_ch_method, _ch_label, _ch_color) in enumerate(_challengers):
- _diffs = _all_diffs[_ch_label]
- _means = np.array([_diffs.get(lib, {}).get("mean_diff", 0) for lib in _libs])
- _ci_lo = np.array([_diffs.get(lib, {}).get("ci_low", 0) for lib in _libs])
- _ci_hi = np.array([_diffs.get(lib, {}).get("ci_high", 0) for lib in _libs])
- _sigs = [_diffs.get(lib, {}).get("sig", "ns") for lib in _libs]
- _yerr = np.array([_means - _ci_lo, _ci_hi - _means])
-
- _offset = (_gi - 0.5) * _bar_width
- _ax.bar(
- _x + _offset, _means, width=_bar_width, yerr=_yerr,
- color=_ch_color, edgecolor="white", linewidth=0.5,
- capsize=3, error_kw={"linewidth": 1.0, "color": "black"},
- zorder=3, label=_ch_label,
- )
-
- _all_star_tops.extend(_ci_hi.tolist())
- _all_star_bots.extend(_ci_lo.tolist())
-
- _star_offset = max(abs(_ci_hi.max()), abs(_ci_lo.min()), 1) * 0.04
- for _i, (_sig, _ci_h, _ci_l, _m) in enumerate(zip(_sigs, _ci_hi, _ci_lo, _means)):
- _y_pos = _ci_h + _star_offset if _m >= 0 else _ci_l - _star_offset
- _va = "bottom" if _m >= 0 else "top"
- _ax.text(_x[_i] + _offset, _y_pos, _sig, ha="center", va=_va,
- fontsize=12, fontweight="bold", color="black", rotation=90)
-
- # Per-panel y-limits with room for stars
- _max_top = max(_all_star_tops) if _all_star_tops else 1
- _min_bot = min(_all_star_bots) if _all_star_bots else 0
- _range = max(abs(_max_top), abs(_min_bot), 1)
- _ax.set_ylim(_min_bot - _range * 0.3, _max_top + _range * 0.3)
-
- _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
- _ax.set_xticks(_x)
- _ax.set_xticklabels(_libs, rotation=45, ha="right", fontsize=12, color="black")
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14, fontweight="bold", color="black")
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="y", alpha=0.3, zorder=1, color="grey")
- _ax.set_xlim(-0.6, _n - 0.4)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _axes[0].set_ylabel("\u0394 % Recovery of Top-N Hits\n(vs Legacy Greedy)", fontsize=14, color="black")
-
- _legend_elements = [
- Patch(facecolor=c, edgecolor="black", label=l)
- for _, l, c in _challengers
- ]
- _axes[1].legend(handles=_legend_elements, loc="upper right", fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black")
-
- _fig.suptitle("Per-Library Recovery Difference: TACTICS vs Legacy Greedy (Docking Libraries)",
- fontsize=15, fontweight="bold", color="black", y=1.01)
- _fig.tight_layout()
-
- mo.vstack([
- _fig,
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="per_library_diff_docking.pdf",
- mimetype="application/pdf",
- label="Download Docking as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(df, mo, pl):
- """Gain decomposition with scoring-type-appropriate baselines.
-
- ROCS: baseline = Legacy Enhanced-RWS
- total = TT-TS - Legacy RWS
- GMIC rotation = TACTICS RWS - Legacy RWS
- TT-TS selection = TT-TS - TACTICS RWS
-
- Docking: baseline = Legacy Standard-Greedy
- total = TT-TS - Legacy Greedy
- GMIC rotation = TACTICS RWS - Legacy Greedy
- TT-TS selection = TT-TS - TACTICS RWS
- """
- _COMP_COUNTS = {
- "amide-suzuki": 3, "betti": 3, "dobener": 3,
- "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
- "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
- "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2,
- "rxn114b": 2, "rxn203": 2, "rxn205": 2, "rxn206": 2,
- "rxn207": 2, "rxn208": 2,
- "adenine": 3, "quinazoline": 3, "thrombin": 2, "amide": 2,
- }
- _DOCKING_LIBS = {"adenine", "quinazoline", "thrombin", "amide"}
-
- libs_list = sorted(df["library_id"].unique().to_list())
-
- decomp_rows = []
- for _lib_id in libs_list:
- _lib_data = df.filter(pl.col("library_id") == _lib_id)
- _n_comp = _COMP_COUNTS.get(_lib_id, _lib_data["n_components"].to_list()[0])
- _lib_methods = set(_lib_data["method"].unique().to_list())
- _is_docking = _lib_id in _DOCKING_LIBS
-
- # Require pivot method (TACTICS RWS) and TT-TS in all cases
- _required = {"TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"}
- if _is_docking:
- _required.add("Legacy Standard-Greedy")
- else:
- _required.add("Legacy Enhanced-RWS")
- if not _required.issubset(_lib_methods):
- continue
-
- _trws = float(_lib_data.filter(pl.col("method") == "TACTICS Enhanced-RWS (GMIC)")["recovered"].mean())
- _ttts = float(_lib_data.filter(pl.col("method") == "TACTICS Enhanced-TT-TS (GMIC)")["recovered"].mean())
-
- if _is_docking:
- _baseline = float(_lib_data.filter(pl.col("method") == "Legacy Standard-Greedy")["recovered"].mean())
- _baseline_name = "Legacy Greedy"
- else:
- _baseline = float(_lib_data.filter(pl.col("method") == "Legacy Enhanced-RWS")["recovered"].mean())
- _baseline_name = "Legacy RWS"
-
- _d_gmic = _trws - _baseline # GMIC rotation gain over baseline
- _d_ttts = _ttts - _trws # TT-TS selection gain over TACTICS RWS
- _d_total = _ttts - _baseline
- _abs_sum = abs(_d_gmic) + abs(_d_ttts)
-
- _share_gmic = round(abs(_d_gmic) / _abs_sum * 100, 0) if _abs_sum > 0.01 else 0
- _share_ttts = round(abs(_d_ttts) / _abs_sum * 100, 0) if _abs_sum > 0.01 else 0
-
- _signed_gmic = _share_gmic if _d_gmic >= 0 else -_share_gmic
- _signed_ttts = _share_ttts if _d_ttts >= 0 else -_share_ttts
-
- decomp_rows.append({
- "library_id": _lib_id,
- "n_components": _n_comp,
- "scoring_type": "Docking" if _is_docking else "ROCS",
- "baseline": _baseline_name,
- "baseline_recovery": round(_baseline, 2),
- "delta_gmic": round(_d_gmic, 2),
- "delta_ttts": round(_d_ttts, 2),
- "delta_total": round(_d_total, 2),
- "share_gmic_%": _share_gmic,
- "share_ttts_%": _share_ttts,
- "signed_gmic_%": _signed_gmic,
- "signed_ttts_%": _signed_ttts,
- })
-
- gain_decomposition = pl.DataFrame(decomp_rows)
-
- mo.vstack([
- mo.md(
- """
- ## Gain Decomposition: GMIC Rotation vs TT-TS Selection
-
- Decomposition uses **scoring-type-appropriate baselines**:
- - **ROCS**: baseline = Legacy Enhanced-RWS (competitive legacy method)
- - **Docking**: baseline = Legacy Standard-Greedy (Legacy RWS is broken on docking)
-
- Both decompose as:
- - **GMIC rotation** = TACTICS RWS − baseline (what GMIC-weighted rotation adds)
- - **TT-TS selection** = TACTICS TT-TS − TACTICS RWS (what TT-TS adds over RWS, both using GMIC)
-
- Signed shares express each component as a percentage of the total gain.
- """
- ),
- mo.ui.table(gain_decomposition.to_pandas()),
- ])
- return (gain_decomposition,)
-
-
-@app.cell
-def _(
- Line2D,
- Patch,
- fig_to_pdf_bytes,
- gain_decomposition,
- mo,
- np,
- pl,
- plt,
- tukey_at_n,
-):
- """Gain decomposition: ROCS (baseline = Legacy RWS)."""
-
- _gmic_color = "#59A14F" # green
- _ttts_color = "#F28E2B" # orange
-
- # Significance: TT-TS vs Legacy RWS for ROCS
- _sig_map = {}
- for _lib_id in gain_decomposition["library_id"].to_list():
- _match = tukey_at_n.filter(
- (pl.col("library_id") == _lib_id)
- & (
- ((pl.col("group1") == "Legacy Enhanced-RWS") & (pl.col("group2") == "TACTICS Enhanced-TT-TS (GMIC)"))
- | ((pl.col("group1") == "TACTICS Enhanced-TT-TS (GMIC)") & (pl.col("group2") == "Legacy Enhanced-RWS"))
- )
- )
- if len(_match) > 0:
- _sig_map[_lib_id] = _match["significance"].to_list()[0]
- else:
- _sig_map[_lib_id] = "ns"
-
- _rocs_gd = gain_decomposition.filter(pl.col("scoring_type") == "ROCS")
-
- _groups = {
- 2: _rocs_gd.filter(pl.col("n_components") == 2).sort("delta_total", descending=True),
- 3: _rocs_gd.filter(pl.col("n_components") == 3).sort("delta_total", descending=True),
- }
-
- _fig, _axes = plt.subplots(1, 2, figsize=(18, 7), facecolor="white")
-
- _global_top = 0.0
- _global_bot = 0.0
- _panel_data = []
-
- for _ax, (_nc, _gdf) in zip(_axes, _groups.items()):
- _ax.set_facecolor("white")
- _n = len(_gdf)
- if _n == 0:
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14,
- fontweight="bold", color="black")
- _ax.text(0.5, 0.5, "No data", ha="center", va="center",
- fontsize=14, color="grey", transform=_ax.transAxes)
- _ax.set_xticks([])
- _panel_data.append(None)
- continue
-
- _labels = _gdf["library_id"].to_list()
- _d_gmic = _gdf["delta_gmic"].to_numpy().astype(float)
- _d_ttts = _gdf["delta_ttts"].to_numpy().astype(float)
- _d_total = _gdf["delta_total"].to_numpy().astype(float)
- _x = np.arange(_n)
-
- _bar_top = np.zeros(_n)
- _bar_bot = np.zeros(_n)
-
- for _i in range(_n):
- _pos_bottom = 0.0
- _neg_bottom = 0.0
-
- _dg = _d_gmic[_i]
- if _dg >= 0:
- _ax.bar(_i, _dg, bottom=_pos_bottom, color=_gmic_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _pos_bottom += _dg
- else:
- _ax.bar(_i, _dg, bottom=_neg_bottom, color=_gmic_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _neg_bottom += _dg
-
- _dt = _d_ttts[_i]
- if _dt >= 0:
- _ax.bar(_i, _dt, bottom=_pos_bottom, color=_ttts_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _pos_bottom += _dt
- else:
- _ax.bar(_i, _dt, bottom=_neg_bottom, color=_ttts_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _neg_bottom += _dt
-
- _bar_top[_i] = _pos_bottom
- _bar_bot[_i] = _neg_bottom
-
- _global_top = max(_global_top, _bar_top.max())
- _global_bot = min(_global_bot, _bar_bot.min())
- _panel_data.append((_labels, _d_total, _bar_top, _bar_bot, _x, _n))
-
- _ax.scatter(_x, _d_total, marker="D", color="black", s=30, zorder=5)
- _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
- _ax.set_xticks(_x)
- _ax.set_xticklabels(_labels, rotation=45, ha="right", fontsize=12, color="black")
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14, fontweight="bold", color="black")
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="y", alpha=0.3, zorder=1, color="grey")
- _ax.set_xlim(-0.6, _n - 0.4)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _data_range = max(abs(_global_top), abs(_global_bot), 1)
- _offset = _data_range * 0.05
- _y_min = _global_bot - _data_range * 0.35
- _y_max = _global_top + _data_range * 0.35
-
- for _ax, _pd in zip(_axes, _panel_data):
- if _pd is None:
- continue
- _labels, _d_total, _bar_top, _bar_bot, _x, _n = _pd
- for _i, _lib in enumerate(_labels):
- _sig = _sig_map.get(_lib, "")
- _y_pos = _bar_top[_i] + _offset if _d_total[_i] >= 0 else _bar_bot[_i] - _offset
- _va = "bottom" if _d_total[_i] >= 0 else "top"
- _ax.text(_i, _y_pos, _sig, ha="center", va=_va,
- fontsize=12, fontweight="bold", color="black")
- _ax.set_ylim(_y_min, _y_max)
-
- _axes[0].set_ylabel(
- "\u0394 % Recovery of Top-N Hits\n(TACTICS TT-TS \u2212 Legacy RWS)",
- fontsize=14, color="black")
-
- _legend_elements = [
- Patch(facecolor=_gmic_color, edgecolor="black", label="GMIC rotation"),
- Patch(facecolor=_ttts_color, edgecolor="black", label="TT-TS selection"),
- Line2D([0], [0], marker="D", color="black", linestyle="None",
- markersize=6, label="Net total (\u0394 total)"),
- ]
- _axes[1].legend(handles=_legend_elements, loc="upper right", fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black")
-
- _fig.suptitle("Gain Decomposition: GMIC Rotation vs TT-TS Selection (ROCS Libraries)\n"
- "Baseline: Legacy Enhanced-RWS",
- fontsize=15, fontweight="bold", color="black", y=1.02)
- _fig.tight_layout()
-
- mo.vstack([
- _fig,
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="gain_decomposition_rocs.pdf",
- mimetype="application/pdf",
- label="Download ROCS as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(
- Line2D,
- Patch,
- fig_to_pdf_bytes,
- gain_decomposition,
- mo,
- np,
- pl,
- plt,
- tukey_at_n,
-):
- """Gain decomposition: Docking (baseline = Legacy Greedy)."""
-
- _gmic_color = "#59A14F" # green
- _ttts_color = "#F28E2B" # orange
-
- # Significance: TT-TS vs Legacy Greedy for docking
- _sig_map_dock = {}
- for _lib_id in gain_decomposition["library_id"].to_list():
- _match = tukey_at_n.filter(
- (pl.col("library_id") == _lib_id)
- & (
- ((pl.col("group1") == "Legacy Standard-Greedy") & (pl.col("group2") == "TACTICS Enhanced-TT-TS (GMIC)"))
- | ((pl.col("group1") == "TACTICS Enhanced-TT-TS (GMIC)") & (pl.col("group2") == "Legacy Standard-Greedy"))
- )
- )
- if len(_match) > 0:
- _sig_map_dock[_lib_id] = _match["significance"].to_list()[0]
- else:
- _sig_map_dock[_lib_id] = "ns"
-
- _dock_gd = gain_decomposition.filter(pl.col("scoring_type") == "Docking")
-
- _groups = {
- 2: _dock_gd.filter(pl.col("n_components") == 2).sort("delta_total", descending=True),
- 3: _dock_gd.filter(pl.col("n_components") == 3).sort("delta_total", descending=True),
- }
-
- _fig, _axes = plt.subplots(1, 2, figsize=(18, 7), facecolor="white")
-
- for _ax, (_nc, _gdf) in zip(_axes, _groups.items()):
- _ax.set_facecolor("white")
- _n = len(_gdf)
- if _n == 0:
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14,
- fontweight="bold", color="black")
- _ax.text(0.5, 0.5, "No data", ha="center", va="center",
- fontsize=14, color="grey", transform=_ax.transAxes)
- _ax.set_xticks([])
- continue
-
- _labels = _gdf["library_id"].to_list()
- _d_gmic = _gdf["delta_gmic"].to_numpy().astype(float)
- _d_ttts = _gdf["delta_ttts"].to_numpy().astype(float)
- _d_total = _gdf["delta_total"].to_numpy().astype(float)
- _x = np.arange(_n)
-
- _bar_top = np.zeros(_n)
- _bar_bot = np.zeros(_n)
-
- for _i in range(_n):
- _pos_bottom = 0.0
- _neg_bottom = 0.0
-
- _dg = _d_gmic[_i]
- if _dg >= 0:
- _ax.bar(_i, _dg, bottom=_pos_bottom, color=_gmic_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _pos_bottom += _dg
- else:
- _ax.bar(_i, _dg, bottom=_neg_bottom, color=_gmic_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _neg_bottom += _dg
-
- _dt = _d_ttts[_i]
- if _dt >= 0:
- _ax.bar(_i, _dt, bottom=_pos_bottom, color=_ttts_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _pos_bottom += _dt
- else:
- _ax.bar(_i, _dt, bottom=_neg_bottom, color=_ttts_color,
- edgecolor="white", linewidth=0.5, width=0.7, zorder=3)
- _neg_bottom += _dt
-
- _bar_top[_i] = _pos_bottom
- _bar_bot[_i] = _neg_bottom
-
- _ax.scatter(_x, _d_total, marker="D", color="black", s=30, zorder=5)
- _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
- _ax.set_xticks(_x)
- _ax.set_xticklabels(_labels, rotation=45, ha="right", fontsize=12, color="black")
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14, fontweight="bold", color="black")
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="y", alpha=0.3, zorder=1, color="grey")
- _ax.set_xlim(-0.6, _n - 0.4)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- # Per-panel y-limits and significance stars
- _panel_range = max(abs(_bar_top.max()), abs(_bar_bot.min()), 1)
- _offset = _panel_range * 0.05
- _ax.set_ylim(
- _bar_bot.min() - _panel_range * 0.35,
- _bar_top.max() + _panel_range * 0.35,
- )
- for _i, _lib in enumerate(_labels):
- _sig = _sig_map_dock.get(_lib, "")
- _y_pos = _bar_top[_i] + _offset if _d_total[_i] >= 0 else _bar_bot[_i] - _offset
- _va = "bottom" if _d_total[_i] >= 0 else "top"
- _ax.text(_i, _y_pos, _sig, ha="center", va=_va,
- fontsize=12, fontweight="bold", color="black")
-
- _axes[0].set_ylabel(
- "\u0394 % Recovery of Top-N Hits\n(TACTICS TT-TS \u2212 Legacy Greedy)",
- fontsize=14, color="black")
-
- _legend_elements = [
- Patch(facecolor=_gmic_color, edgecolor="black", label="GMIC rotation"),
- Patch(facecolor=_ttts_color, edgecolor="black", label="TT-TS selection"),
- Line2D([0], [0], marker="D", color="black", linestyle="None",
- markersize=6, label="Net total (\u0394 total)"),
- ]
- _axes[1].legend(handles=_legend_elements, loc="upper right", fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black")
-
- _fig.suptitle("Gain Decomposition: GMIC Rotation vs TT-TS Selection (Docking Libraries)\n"
- "Baseline: Legacy Standard-Greedy",
- fontsize=15, fontweight="bold", color="black", y=1.02)
- _fig.tight_layout()
-
- mo.vstack([
- _fig,
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="gain_decomposition_docking.pdf",
- mimetype="application/pdf",
- label="Download Docking as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(df, mo, pl):
- """Per-query recovery for detailed analysis (e.g., scatter plots)."""
- query_method_recovery = (
- df.group_by(["library_id", "n_components", "query_id", "method"])
- .agg([
- pl.col("recovered").mean().round(1).alias("mean_recovery"),
- pl.col("recovered").std().round(1).alias("std_recovery"),
- ])
- )
-
- mo.vstack([
- mo.md("## Per-Query Recovery"),
- mo.ui.table(query_method_recovery.to_pandas()),
- ])
- return
-
-
-@app.cell
-def _(mo):
- """Available DataFrames reference."""
- mo.md(
- """
- ## Available DataFrames
-
- | Variable | Description |
- |---|---|
- | `df` | Raw benchmark results (109,090 rows, 22 libraries incl. thrombin-docking) |
- | `method_summary` | Overall method means/std/median |
- | `lib_method` | Per-library per-method mean recovery (long format) |
- | `lib_wide` | Per-library recovery, one column per method (wide format) |
- | `lib_method_stats` | Per-library per-method mean, std, SE |
- | `component_summary` | Recovery by 2-comp vs 3-comp |
- | `pairwise_df` | All pairwise significance tests per library |
- | `gain_decomposition` | Δ GMIC rotation vs Δ TT-TS per library; baseline = Legacy RWS (ROCS) or Legacy Greedy (docking) |
- | `query_method_recovery` | Per-query per-method mean recovery |
- """
- )
- return
-
-
-@app.cell
-def _(mo):
- mo.md(r"""
- # Sensitivity Analysis - Top N compounds
- In this section we explore the roubustness of CATS as the interval for the top N compounds increases, if CATS is interpreting the SAR landscape correctly and systematically able to uncover mid tier hits.
- """)
- return
-
-
-@app.cell
-def _(mo, recovery_all):
- """Summary of recovery data available for sensitivity analysis."""
- _n_rows = len(recovery_all)
- _n_libs = recovery_all["library_id"].n_unique()
- _thresholds = sorted(recovery_all["top_n"].unique().to_list())
- _methods = sorted(recovery_all["method"].unique().to_list())
-
- mo.vstack([
- mo.md(
- f"""
- ## Top-N Sensitivity Data (from Post-Hoc Analysis)
-
- - **Rows**: {_n_rows:,}
- - **Libraries**: {_n_libs}
- - **Top-N thresholds**: {_thresholds}
- - **Methods**: {', '.join(_methods)}
- """
- ),
- ])
- return
-
-
-@app.cell
-def _(fig_to_pdf_bytes, mo, pl, plt, recovery_all):
- """Overall % recovery vs top-N by component count (ROCS libraries, 10 thresholds)."""
-
- _DOCK_LIBS = {"thrombin", "adenine", "amide", "quinazoline"}
- _ROCS_NCOMP = {
- "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
- "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
- "amide-suzuki": 3, "betti": 3, "dobener": 3,
- "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
- "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
- }
-
- _rocs_all = (
- recovery_all
- .filter(~pl.col("library_id").is_in(_DOCK_LIBS))
- .filter(pl.col("library_id").is_in(set(_ROCS_NCOMP.keys())))
- .with_columns(
- pl.col("library_id").replace(_ROCS_NCOMP).cast(pl.Int64).alias("n_components")
- )
- )
-
- mo.stop(len(_rocs_all) == 0, mo.md("No ROCS library data available."))
-
- _method_colors = {
- "TACTICS Enhanced-TT-TS (GMIC)": "#E45756",
- "TACTICS Enhanced-RWS (GMIC)": "#4C78A8",
- "TACTICS Balanced-Greedy": "#F28E2B",
- "Legacy Standard-Greedy": "#76B7B2",
- "Legacy Enhanced-RWS": "#B07AA1",
- }
- _method_markers = {
- "TACTICS Enhanced-TT-TS (GMIC)": "D",
- "TACTICS Enhanced-RWS (GMIC)": "o",
- "TACTICS Balanced-Greedy": "P",
- "Legacy Standard-Greedy": "s",
- "Legacy Enhanced-RWS": "^",
- }
- _method_order = [
- "TACTICS Enhanced-TT-TS (GMIC)",
- "TACTICS Enhanced-RWS (GMIC)",
- "TACTICS Balanced-Greedy",
- "Legacy Standard-Greedy",
- "Legacy Enhanced-RWS",
- ]
-
- _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white",
- sharey=True)
-
- for _panel_idx, _nc in enumerate([2, 3]):
- _ax = _axes[_panel_idx]
- _ax.set_facecolor("white")
-
- _comp_data = _rocs_all.filter(pl.col("n_components") == _nc)
-
- _agg = (
- _comp_data.group_by(["method", "top_n"])
- .agg([
- pl.col("recovery_frac").mean().alias("mean_frac"),
- pl.col("recovery_frac").std().alias("std_frac"),
- pl.col("recovery_frac").count().alias("n"),
- ])
- .with_columns(
- (pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac")
- )
- .sort(["method", "top_n"])
- )
-
- for _method in _method_order:
- _mdata = _agg.filter(pl.col("method") == _method).sort("top_n")
- if len(_mdata) == 0:
- continue
- _x = _mdata["top_n"].to_numpy()
- _y = _mdata["mean_frac"].to_numpy() * 100
- _se = _mdata["se_frac"].to_numpy() * 100
- _color = _method_colors.get(_method, "#999999")
- _marker = _method_markers.get(_method, "o")
-
- _ax.plot(_x, _y, marker=_marker, color=_color, linewidth=2,
- markersize=7, label=_method, zorder=3)
- _ax.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se,
- alpha=0.15, color=_color, zorder=2)
-
- _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=15,
- fontweight="bold", color="black")
- _ax.set_xticks(sorted(_agg["top_n"].unique().to_list()))
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="both", alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _axes[0].set_ylabel("% Recovery", fontsize=14, color="black")
-
- # Legend outside the figure below the panels
- _handles, _labels = _axes[0].get_legend_handles_labels()
- _fig.legend(
- _handles, _labels,
- loc="lower center", ncol=3, fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black",
- bbox_to_anchor=(0.5, -0.08),
- )
-
- _fig.suptitle("Mean % Recovery vs Top-N Threshold (ROCS Libraries)",
- fontsize=15, fontweight="bold", color="black", y=1.01)
- _fig.tight_layout()
-
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="overall_recovery_vs_topn.pdf",
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(fig_to_pdf_bytes, mo, pl, plt, recovery_all):
- """Mean % recovery vs top-N for docking libraries (2-comp and 3-comp panels)."""
-
- _DOCK_LIBS = {"adenine", "amide", "quinazoline", "thrombin"}
- # n_components is 0 for thrombin/quinazoline in the data — fix with known values
- _DOCK_NCOMP = {"thrombin": 2, "amide": 2, "adenine": 3, "quinazoline": 3}
-
- _dock_all = (
- recovery_all
- .filter(pl.col("library_id").is_in(_DOCK_LIBS))
- .with_columns(
- pl.col("library_id").replace(_DOCK_NCOMP).cast(pl.Int64).alias("n_components")
- )
- )
-
- mo.stop(len(_dock_all) == 0, mo.md("No docking library data available."))
-
- _method_colors = {
- "TACTICS Enhanced-TT-TS (GMIC)": "#E45756",
- "TACTICS Enhanced-RWS (GMIC)": "#4C78A8",
- "TACTICS Balanced-Greedy": "#F28E2B",
- "Legacy Standard-Greedy": "#76B7B2",
- "Legacy Enhanced-RWS": "#B07AA1",
- }
- _method_markers = {
- "TACTICS Enhanced-TT-TS (GMIC)": "D",
- "TACTICS Enhanced-RWS (GMIC)": "o",
- "TACTICS Balanced-Greedy": "P",
- "Legacy Standard-Greedy": "s",
- "Legacy Enhanced-RWS": "^",
- }
- _method_order = [
- "TACTICS Enhanced-TT-TS (GMIC)",
- "TACTICS Enhanced-RWS (GMIC)",
- "TACTICS Balanced-Greedy",
- "Legacy Standard-Greedy",
- "Legacy Enhanced-RWS",
- ]
-
- _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white",
- sharey=True)
-
- for _panel_idx, _nc in enumerate([2, 3]):
- _ax = _axes[_panel_idx]
- _ax.set_facecolor("white")
-
- _comp_data = _dock_all.filter(pl.col("n_components") == _nc)
-
- # Aggregate: mean and SE of recovery fraction across libraries and replicates
- _agg = (
- _comp_data.group_by(["method", "top_n"])
- .agg([
- pl.col("recovery_frac").mean().alias("mean_frac"),
- pl.col("recovery_frac").std().alias("std_frac"),
- pl.col("recovery_frac").count().alias("n"),
- ])
- .with_columns(
- (pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac")
- )
- .sort(["method", "top_n"])
- )
-
- for _method in _method_order:
- _mdata = _agg.filter(pl.col("method") == _method).sort("top_n")
- if len(_mdata) == 0:
- continue
- _x = _mdata["top_n"].to_numpy()
- _y = _mdata["mean_frac"].to_numpy() * 100
- _se = _mdata["se_frac"].to_numpy() * 100
- _color = _method_colors.get(_method, "#999999")
- _marker = _method_markers.get(_method, "o")
-
- _ax.plot(_x, _y, marker=_marker, color=_color, linewidth=2,
- markersize=7, label=_method, zorder=3)
- _ax.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se,
- alpha=0.15, color=_color, zorder=2)
-
- _libs_in_panel = sorted(
- _comp_data["library_id"].unique().to_list()
- )
- _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
- _ax.set_title(
- f"{_nc}-Component Docking Libraries\n({', '.join(_libs_in_panel)})",
- fontsize=15, fontweight="bold", color="black",
- )
- _ax.set_xticks(sorted(_agg["top_n"].unique().to_list()))
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="both", alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _axes[0].set_ylabel("% Recovery", fontsize=14, color="black")
-
- # Legend outside the figure below the panels
- _handles, _labels = _axes[0].get_legend_handles_labels()
- _fig.legend(
- _handles, _labels,
- loc="lower center", ncol=3, fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black",
- bbox_to_anchor=(0.5, -0.08),
- )
-
- _fig.suptitle("Mean % Recovery vs Top-N Threshold (Docking Libraries)",
- fontsize=15, fontweight="bold", color="black", y=1.01)
- _fig.tight_layout()
-
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="docking_recovery_vs_topn.pdf",
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(mo, recovery_all):
- """Select library for top-N recovery curve (all libraries)."""
- _all_libs = sorted(recovery_all["library_id"].unique().to_list())
-
- topn_library_dropdown = mo.ui.dropdown(
- options=_all_libs,
- value=_all_libs[0],
- label="Library",
- )
- topn_library_dropdown
- return (topn_library_dropdown,)
-
-
-@app.cell
-def _(fig_to_pdf_bytes, mo, pl, plt, recovery_all, topn_library_dropdown):
- """Top-N recovery curve: % recovery vs top-N threshold per method."""
-
- _lib_id = topn_library_dropdown.value
- _lib_data = recovery_all.filter(pl.col("library_id") == _lib_id)
-
- # Aggregate: mean and SE of recovery_frac per (method, top_n)
- _agg = (
- _lib_data.group_by(["method", "top_n"])
- .agg([
- pl.col("recovery_frac").mean().alias("mean_frac"),
- pl.col("recovery_frac").std().alias("std_frac"),
- pl.col("recovery_frac").count().alias("n"),
- ])
- .with_columns(
- (pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac")
- )
- .sort(["method", "top_n"])
- )
-
- _methods = sorted(_agg["method"].unique().to_list())
- _method_colors = {
- "TACTICS Enhanced-TT-TS (GMIC)": "#E45756",
- "TACTICS Enhanced-RWS (GMIC)": "#4C78A8",
- "TACTICS Balanced-Greedy": "#F28E2B",
- "Legacy Standard-Greedy": "#76B7B2",
- "Legacy Enhanced-RWS": "#B07AA1",
- }
- _method_markers = {
- "TACTICS Enhanced-TT-TS (GMIC)": "D",
- "TACTICS Enhanced-RWS (GMIC)": "o",
- "TACTICS Balanced-Greedy": "P",
- "Legacy Standard-Greedy": "s",
- "Legacy Enhanced-RWS": "^",
- }
-
- _fig, _ax = plt.subplots(figsize=(10, 6), facecolor="white")
- _ax.set_facecolor("white")
-
- for _method in _methods:
- _mdata = _agg.filter(pl.col("method") == _method).sort("top_n")
- _x = _mdata["top_n"].to_numpy()
- _y = _mdata["mean_frac"].to_numpy() * 100
- _se = _mdata["se_frac"].to_numpy() * 100
- _color = _method_colors.get(_method, "#999999")
- _marker = _method_markers.get(_method, "o")
-
- _ax.plot(_x, _y, marker=_marker, color=_color, linewidth=2,
- markersize=7, label=_method, zorder=3)
- _ax.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se,
- alpha=0.15, color=_color, zorder=2)
-
- _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
- _ax.set_ylabel("% Recovery", fontsize=14, color="black")
- _ax.set_title(
- f"% Recovery vs Top-N Threshold: {_lib_id}",
- fontsize=15, fontweight="bold", color="black",
- )
- _ax.set_xticks(sorted(_agg["top_n"].unique().to_list()))
- _ax.tick_params(colors="black", labelsize=12)
- _ax.legend(fontsize=12, facecolor="white", edgecolor="black", labelcolor="black")
- _ax.grid(axis="both", alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
- _fig.tight_layout()
-
- _pdf_name = f"recovery_vs_topn_{_lib_id}.pdf".replace(" ", "_")
- mo.vstack([
- plt.gca(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=_pdf_name,
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(mo, recovery_all):
- """Select baseline and challenger methods for top-N difference plot."""
- _all_methods = sorted(recovery_all["method"].unique().to_list())
-
- _default_baseline = "Legacy Enhanced-RWS" if "Legacy Enhanced-RWS" in _all_methods else _all_methods[0]
- _default_challenger = "TACTICS Enhanced-TT-TS (GMIC)" if "TACTICS Enhanced-TT-TS (GMIC)" in _all_methods else _all_methods[-1]
-
- topn_baseline_dropdown = mo.ui.dropdown(
- options=_all_methods,
- value=_default_baseline,
- label="Baseline method",
- )
- topn_challenger_dropdown = mo.ui.dropdown(
- options=_all_methods,
- value=_default_challenger,
- label="Challenger method",
- )
-
- mo.hstack(
- [topn_baseline_dropdown, topn_challenger_dropdown],
- justify="start",
- gap=2,
- )
- return topn_baseline_dropdown, topn_challenger_dropdown
-
-
-@app.cell
-def _(
- fig_to_pdf_bytes,
- mo,
- np,
- pl,
- plt,
- recovery_all,
- stats,
- topn_baseline_dropdown,
- topn_challenger_dropdown,
- topn_library_dropdown,
-):
- """Bars showing Δ % recovery at each top-N threshold with significance stars."""
-
- _lib_id = topn_library_dropdown.value
- _baseline = topn_baseline_dropdown.value
- _challenger = topn_challenger_dropdown.value
-
- _lib_data = recovery_all.filter(pl.col("library_id") == _lib_id)
-
- _available_methods = set(_lib_data["method"].unique().to_list())
- mo.stop(
- _baseline not in _available_methods or _challenger not in _available_methods,
- mo.md(
- f"Methods **{_baseline}** and/or **{_challenger}** not found in `{_lib_id}`. "
- f"Available: {sorted(_available_methods)}"
- ),
- )
-
- _thresholds = sorted(_lib_data["top_n"].unique().to_list())
-
- _topn_vals = []
- _mean_diffs = []
- _ci_lows = []
- _ci_highs = []
- _sigs = []
-
- for _tn in _thresholds:
- _tn_data = _lib_data.filter(pl.col("top_n") == _tn)
- _a = (
- _tn_data.filter(pl.col("method") == _baseline)
- .sort(["query_id", "replicate"])["recovery_frac"]
- .to_numpy()
- )
- _b = (
- _tn_data.filter(pl.col("method") == _challenger)
- .sort(["query_id", "replicate"])["recovery_frac"]
- .to_numpy()
- )
- _n = min(len(_a), len(_b))
- if _n < 2:
- continue
- _a, _b = _a[:_n], _b[:_n]
- _diffs = (_b - _a) * 100
- _mean_d = float(np.mean(_diffs))
- _se_d = float(np.std(_diffs, ddof=1) / np.sqrt(_n))
- _t_crit = float(stats.t.ppf(0.975, df=_n - 1))
- _, _p = stats.ttest_rel(_a, _b)
- _sig = (
- "****" if _p <= 0.0001
- else "***" if _p <= 0.001
- else "**" if _p <= 0.01
- else "*" if _p <= 0.05
- else "ns"
- )
-
- _topn_vals.append(_tn)
- _mean_diffs.append(_mean_d)
- _ci_lows.append(_mean_d - _t_crit * _se_d)
- _ci_highs.append(_mean_d + _t_crit * _se_d)
- _sigs.append(_sig)
-
- _x_vals = np.array(_topn_vals)
- _y = np.array(_mean_diffs)
- _lo = np.array(_ci_lows)
- _hi = np.array(_ci_highs)
-
- _x = np.arange(len(_x_vals))
- _bar_width = 0.6
- _color = "#4C78A8"
-
- _fig, _ax = plt.subplots(figsize=(10, 6), facecolor="white")
- _ax.set_facecolor("white")
-
- # Bars with 95% CI error bars
- _yerr = np.array([_y - _lo, _hi - _y])
- _ax.bar(_x, _y, width=_bar_width, color=_color, alpha=0.75,
- edgecolor="white", linewidth=0.5, zorder=3,
- yerr=_yerr, capsize=4,
- error_kw={"linewidth": 1.2, "color": "black"})
-
- _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
-
- # Significance stars
- _data_range = max(abs(_hi.max()), abs(_lo.min()), 1)
- _offset = _data_range * 0.04
- for _i in range(len(_x)):
- _y_pos = _hi[_i] + _offset if _y[_i] >= 0 else _lo[_i] - _offset
- _va = "bottom" if _y[_i] >= 0 else "top"
- _ax.text(_x[_i], _y_pos, _sigs[_i], ha="center", va=_va,
- fontsize=12, fontweight="bold", color="black", rotation=90)
-
- # Expand y-limits for annotations
- _y_min = min(_lo.min(), _y.min()) - _data_range * 0.15
- _y_max = max(_hi.max(), _y.max()) + _data_range * 0.15
- _ax.set_ylim(_y_min, _y_max)
-
- _ax.set_xticks(_x)
- _ax.set_xticklabels([str(v) for v in _x_vals], fontsize=12, color="black")
- _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
- _ax.set_ylabel("\u0394 % Recovery", fontsize=14, color="black")
- _ax.set_title(
- f"Recovery Difference vs Top-N: {_challenger} \u2212 {_baseline} ({_lib_id})",
- fontsize=14, fontweight="bold", color="black",
- )
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="y", alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _fig.tight_layout()
-
- _pdf_name = f"recovery_diff_vs_topn_{_challenger}_{_baseline}_{_lib_id}.pdf".replace(" ", "_")
- mo.vstack([
- plt.gca(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=_pdf_name,
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(
- fig_to_pdf_bytes,
- mo,
- np,
- pl,
- plt,
- recovery_all,
- stats,
- topn_baseline_dropdown,
- topn_challenger_dropdown,
-):
- """Component-aggregated Δ % recovery vs top-N (2-comp and 3-comp panels, ROCS libraries)."""
-
- _baseline = topn_baseline_dropdown.value
- _challenger = topn_challenger_dropdown.value
-
- _ROCS_NCOMP = {
- "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
- "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
- "amide-suzuki": 3, "betti": 3, "dobener": 3,
- "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
- "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
- }
-
- _rocs_data = (
- recovery_all
- .filter(pl.col("library_id").is_in(set(_ROCS_NCOMP.keys())))
- .with_columns(
- pl.col("library_id").replace(_ROCS_NCOMP).cast(pl.Int64).alias("n_components")
- )
- )
-
- _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white",
- sharey=True)
-
- _global_ymin = 0.0
- _global_ymax = 0.0
- _panel_annots = [] # store (sigs, y, hi, lo) per panel for second pass
-
- for _panel_idx, _nc in enumerate([2, 3]):
- _ax = _axes[_panel_idx]
- _ax.set_facecolor("white")
-
- _comp_data = _rocs_data.filter(pl.col("n_components") == _nc)
- _thresholds = sorted(_comp_data["top_n"].unique().to_list())
-
- _topn_vals = []
- _mean_diffs = []
- _ci_lows = []
- _ci_highs = []
- _sigs = []
-
- for _tn in _thresholds:
- _tn_data = _comp_data.filter(pl.col("top_n") == _tn)
- # Pool all libraries: pair by (library_id, query_id, replicate)
- _a = (
- _tn_data.filter(pl.col("method") == _baseline)
-
- .sort(["library_id", "query_id", "replicate"])["recovery_frac"]
- .to_numpy()
- )
- _b = (
- _tn_data.filter(pl.col("method") == _challenger)
- .sort(["library_id", "query_id", "replicate"])["recovery_frac"]
- .to_numpy()
- )
- _n = min(len(_a), len(_b))
- if _n < 2:
- continue
- _a, _b = _a[:_n], _b[:_n]
- _diffs = (_b - _a) * 100
- _mean_d = float(np.mean(_diffs))
- _se_d = float(np.std(_diffs, ddof=1) / np.sqrt(_n))
- _t_crit = float(stats.t.ppf(0.975, df=_n - 1))
- _, _p = stats.ttest_rel(_a, _b)
- _sig = (
- "****" if _p <= 0.0001
- else "***" if _p <= 0.001
- else "**" if _p <= 0.01
- else "*" if _p <= 0.05
- else "ns"
- )
-
- _topn_vals.append(_tn)
- _mean_diffs.append(_mean_d)
- _ci_lows.append(_mean_d - _t_crit * _se_d)
- _ci_highs.append(_mean_d + _t_crit * _se_d)
- _sigs.append(_sig)
-
- _y = np.array(_mean_diffs)
- _lo = np.array(_ci_lows)
- _hi = np.array(_ci_highs)
- _x = np.arange(len(_topn_vals))
-
- _color = "#4C78A8" if _nc == 2 else "#E45756"
- _yerr = np.array([_y - _lo, _hi - _y])
- _ax.bar(_x, _y, width=0.6, color=_color, alpha=0.75,
- edgecolor="white", linewidth=0.5, zorder=3,
- yerr=_yerr, capsize=4,
- error_kw={"linewidth": 1.2, "color": "black"})
-
- _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
-
- _ax.set_xticks(_x)
- _ax.set_xticklabels([str(v) for v in _topn_vals], fontsize=12, color="black")
- _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
- _ax.set_title(f"{_nc}-Component Libraries", fontsize=14,
- fontweight="bold", color="black")
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="y", alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _global_ymin = min(_global_ymin, _lo.min() if len(_lo) else 0)
- _global_ymax = max(_global_ymax, _hi.max() if len(_hi) else 0)
-
- _panel_annots.append((_sigs, _y, _hi, _lo))
-
- # Second pass: uniform y-limits and significance stars
- _data_range = max(abs(_global_ymax), abs(_global_ymin), 1)
- _offset = _data_range * 0.04
- _y_min = _global_ymin - _data_range * 0.20
- _y_max = _global_ymax + _data_range * 0.20
-
- for _panel_idx in range(2):
- _ax = _axes[_panel_idx]
- _ax.set_ylim(_y_min, _y_max)
- _sigs, _y_arr, _hi_arr, _lo_arr = _panel_annots[_panel_idx]
- for _i in range(len(_sigs)):
- _y_pos = _hi_arr[_i] + _offset if _y_arr[_i] >= 0 else _lo_arr[_i] - _offset
- _va = "bottom" if _y_arr[_i] >= 0 else "top"
- _ax.text(_i, _y_pos, _sigs[_i], ha="center", va=_va,
- fontsize=12, fontweight="bold", color="black", rotation=90)
-
- _axes[0].set_ylabel("\u0394 % Recovery", fontsize=14, color="black")
-
- _fig.suptitle(
- f"Mean Recovery Difference vs Top-N (ROCS): {_challenger} \u2212 {_baseline}",
- fontsize=15, fontweight="bold", color="black", y=1.01,
- )
- _fig.tight_layout()
-
- _pdf_name = f"component_agg_diff_vs_topn_rocs_{_challenger}_{_baseline}.pdf".replace(" ", "_")
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=_pdf_name,
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(
- fig_to_pdf_bytes,
- mo,
- np,
- pl,
- plt,
- recovery_all,
- stats,
- topn_challenger_dropdown,
-):
- """Component-aggregated Δ % recovery vs top-N (2-comp and 3-comp panels, docking libraries)."""
-
- # Docking baseline is always Legacy Greedy (Legacy RWS is broken on docking)
- _baseline = "Legacy Standard-Greedy"
- _challenger = topn_challenger_dropdown.value
-
- _DOCK_NCOMP = {"thrombin": 2, "amide": 2, "adenine": 3, "quinazoline": 3}
-
- _dock_data = (
- recovery_all
- .filter(pl.col("library_id").is_in(set(_DOCK_NCOMP.keys())))
- .with_columns(
- pl.col("library_id").replace(_DOCK_NCOMP).cast(pl.Int64).alias("n_components")
- )
- )
-
- _available_methods = set(_dock_data["method"].unique().to_list())
- mo.stop(
- _baseline not in _available_methods or _challenger not in _available_methods,
- mo.md(
- f"Methods **{_baseline}** and/or **{_challenger}** not found in docking data. "
- f"Available: {sorted(_available_methods)}"
- ),
- )
-
- _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white",
- sharey=True)
-
- _global_ymin = 0.0
- _global_ymax = 0.0
- _panel_annots = []
-
- for _panel_idx, _nc in enumerate([2, 3]):
- _ax = _axes[_panel_idx]
- _ax.set_facecolor("white")
-
- _comp_data = _dock_data.filter(pl.col("n_components") == _nc)
- _thresholds = sorted(_comp_data["top_n"].unique().to_list())
-
- _libs_in_panel = sorted(_comp_data["library_id"].unique().to_list())
-
- _topn_vals = []
- _mean_diffs = []
- _ci_lows = []
- _ci_highs = []
- _sigs = []
-
- for _tn in _thresholds:
- _tn_data = _comp_data.filter(pl.col("top_n") == _tn)
- _a = (
- _tn_data.filter(pl.col("method") == _baseline)
- .sort(["library_id", "query_id", "replicate"])["recovery_frac"]
- .to_numpy()
- )
- _b = (
- _tn_data.filter(pl.col("method") == _challenger)
- .sort(["library_id", "query_id", "replicate"])["recovery_frac"]
- .to_numpy()
- )
- _n = min(len(_a), len(_b))
- if _n < 2:
- continue
- _a, _b = _a[:_n], _b[:_n]
- _diffs = (_b - _a) * 100
- _mean_d = float(np.mean(_diffs))
- _se_d = float(np.std(_diffs, ddof=1) / np.sqrt(_n))
- _t_crit = float(stats.t.ppf(0.975, df=_n - 1))
- _, _p = stats.ttest_rel(_a, _b)
- _sig = (
- "****" if _p <= 0.0001
- else "***" if _p <= 0.001
- else "**" if _p <= 0.01
- else "*" if _p <= 0.05
- else "ns"
- )
-
- _topn_vals.append(_tn)
- _mean_diffs.append(_mean_d)
- _ci_lows.append(_mean_d - _t_crit * _se_d)
- _ci_highs.append(_mean_d + _t_crit * _se_d)
- _sigs.append(_sig)
-
- _y = np.array(_mean_diffs)
- _lo = np.array(_ci_lows)
- _hi = np.array(_ci_highs)
- _x = np.arange(len(_topn_vals))
-
- _color = "#4C78A8" if _nc == 2 else "#E45756"
- _yerr = np.array([_y - _lo, _hi - _y])
- _ax.bar(_x, _y, width=0.6, color=_color, alpha=0.75,
- edgecolor="white", linewidth=0.5, zorder=3,
- yerr=_yerr, capsize=4,
- error_kw={"linewidth": 1.2, "color": "black"})
-
- _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
-
- _ax.set_xticks(_x)
- _ax.set_xticklabels([str(v) for v in _topn_vals], fontsize=12, color="black")
- _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
- _ax.set_title(
- f"{_nc}-Component Docking\n({', '.join(_libs_in_panel)})",
- fontsize=15, fontweight="bold", color="black",
- )
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(axis="y", alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _global_ymin = min(_global_ymin, _lo.min() if len(_lo) else 0)
- _global_ymax = max(_global_ymax, _hi.max() if len(_hi) else 0)
-
- _panel_annots.append((_sigs, _y, _hi, _lo))
-
- # Second pass: uniform y-limits and significance stars
- _data_range = max(abs(_global_ymax), abs(_global_ymin), 1)
- _offset = _data_range * 0.04
- _y_min = _global_ymin - _data_range * 0.20
- _y_max = _global_ymax + _data_range * 0.20
-
- for _panel_idx in range(2):
- _ax = _axes[_panel_idx]
- _ax.set_ylim(_y_min, _y_max)
- _sigs, _y_arr, _hi_arr, _lo_arr = _panel_annots[_panel_idx]
- for _i in range(len(_sigs)):
- _y_pos = _hi_arr[_i] + _offset if _y_arr[_i] >= 0 else _lo_arr[_i] - _offset
- _va = "bottom" if _y_arr[_i] >= 0 else "top"
- _ax.text(_i, _y_pos, _sigs[_i], ha="center", va=_va,
- fontsize=12, fontweight="bold", color="black", rotation=90)
-
- _axes[0].set_ylabel("\u0394 % Recovery", fontsize=14, color="black")
-
- _fig.suptitle(
- f"Mean Recovery Difference vs Top-N (Docking): {_challenger} \u2212 {_baseline}",
- fontsize=15, fontweight="bold", color="black", y=1.01,
- )
- _fig.tight_layout()
-
- _pdf_name = f"component_agg_diff_vs_topn_docking_{_challenger}_{_baseline}.pdf".replace(" ", "_")
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=_pdf_name,
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(mo):
- mo.md(r"""
- ## Simultaneous Confidence Interval Plots (Tukey HSD)
-
- Per-library: select a library to view all pairwise method comparisons.
- Overall: aggregated across ROCS libraries and docking libraries separately.
- """)
- return
-
-
-@app.cell
-def _(df, mo):
- """Library selector for per-library Tukey HSD CI plot."""
- _libraries = sorted(df["library_id"].unique().to_list())
- library_dropdown = mo.ui.dropdown(
- options=_libraries,
- value=_libraries[0] if _libraries else "",
- label="Library",
- )
- library_dropdown
- return (library_dropdown,)
-
-
-@app.cell
-def _(
- Line2D,
- TOP_N,
- fig_to_pdf_bytes,
- library_dropdown,
- mo,
- np,
- pl,
- plt,
- tukey_at_n,
-):
- """Option 4 (per-library): Simultaneous CI plot for one library."""
- _lib_id = library_dropdown.value
- _lib_tukey = tukey_at_n.filter(pl.col("library_id") == _lib_id)
-
- mo.stop(len(_lib_tukey) == 0, mo.md(f"No Tukey HSD results for {_lib_id} at this top-N."))
-
- _n_comparisons = len(_lib_tukey)
-
- _left_labels = []
- _right_labels = []
- _means = []
- _ci_lows = []
- _ci_highs = []
- _rejects = []
-
- # Normalize: mean_diff = group2 - group1 from Tukey HSD.
- # Ensure the better method (higher recovery) is always the challenger (right).
- # If mean_diff > 0, group2 is better → group1=baseline(left), group2=challenger(right).
- # If mean_diff < 0, group1 is better → flip: group2=baseline(left), group1=challenger(right).
- for _row in _lib_tukey.iter_rows(named=True):
- if _row["mean_diff"] >= 0:
- _left_labels.append(_row["group1"])
- _right_labels.append(_row["group2"])
- _means.append(_row["mean_diff"])
- _ci_lows.append(_row["ci_low"])
- _ci_highs.append(_row["ci_high"])
- else:
- _left_labels.append(_row["group2"])
- _right_labels.append(_row["group1"])
- _means.append(-_row["mean_diff"])
- _ci_lows.append(-_row["ci_high"])
- _ci_highs.append(-_row["ci_low"])
- _rejects.append(_row["reject"])
-
- # Sort by mean_diff descending after normalization
- _order = np.argsort(_means)[::-1]
- _left_labels = [_left_labels[i] for i in _order]
- _right_labels = [_right_labels[i] for i in _order]
- _rejects = [_rejects[i] for i in _order]
- _means = np.array([_means[i] for i in _order])
- _ci_lows = np.array([_ci_lows[i] for i in _order])
- _ci_highs = np.array([_ci_highs[i] for i in _order])
-
- _means = np.array(_means)
- _ci_lows = np.array(_ci_lows)
- _ci_highs = np.array(_ci_highs)
-
- # Extract p-values for significance stars
- _p_values = []
- for _row in _lib_tukey.iter_rows(named=True):
- _p_values.append(_row["p_adj"])
- _p_values = np.array([_p_values[i] for i in _order])
-
- _fig, _ax = plt.subplots(
- figsize=(12, max(4, _n_comparisons * 0.55)),
- facecolor="white",
- )
- _ax.set_facecolor("white")
- _y = np.arange(_n_comparisons)
-
- _colors = ["#4C78A8" if r else "#999999" for r in _rejects]
-
- for _i in range(_n_comparisons):
- _ax.plot(
- [_ci_lows[_i], _ci_highs[_i]], [_i, _i],
- color=_colors[_i], linewidth=2.5, solid_capstyle="round", zorder=3,
- )
- _ax.plot(_means[_i], _i, "o", color=_colors[_i], markersize=8, zorder=4)
- # Significance stars
- _p = _p_values[_i]
- _sig_str = (
- "****" if _p <= 0.0001
- else "***" if _p <= 0.001
- else "**" if _p <= 0.01
- else "*" if _p <= 0.05
- else "ns"
- )
- _ax.text(
- _ci_highs[_i] + 0.15, _i, _sig_str,
- ha="left", va="center",
- fontsize=12, fontweight="bold", color="black",
- )
-
- _ax.axvline(0, color="black", linewidth=0.8, linestyle="-", zorder=2)
-
- # Expand x-limits so stars fit inside the axes
- _x_pad = max(_ci_highs.max(), abs(_ci_lows.min())) * 0.18
- _ax.set_xlim(
- min(_ci_lows.min(), 0) - _x_pad * 0.5,
- _ci_highs.max() + _x_pad,
- )
-
- # Define key comparisons
- _legacy_methods = {"Legacy Enhanced-RWS", "Legacy Standard-Greedy"}
- _tactics_adaptive = {"TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"}
-
- def _is_tactics_vs_legacy(_left, _right):
- return (
- (_left in _legacy_methods and _right in _tactics_adaptive)
- or (_left in _tactics_adaptive and _right in _legacy_methods)
- )
-
- _COLOR_KEY_TICK = "#4C78A8"
-
- # Left y-axis: challenger (better method)
- _ax.set_yticks(_y)
- _ax.set_yticklabels(_right_labels, fontsize=12, color="black")
- _ax.set_ylabel("Challenger", fontsize=14, color="black")
-
- # Pad y-limits so effect size labels above top row aren't clipped
- _ax.set_ylim(-0.6, _n_comparisons - 0.4)
-
- # Invert y so largest diff is at top (do this BEFORE twinx so both share orientation)
- _ax.invert_yaxis()
-
- # Right y-axis: baseline (worse method)
- _ax_right = _ax.twinx()
- _ax_right.set_ylim(_ax.get_ylim())
- _ax_right.set_yticks(_y)
- _ax_right.set_yticklabels(_left_labels, fontsize=12, color="black")
- _ax_right.set_ylabel("Baseline", fontsize=14, color="black")
- _ax_right.tick_params(colors="black", labelsize=12)
- for _spine in _ax_right.spines.values():
- _spine.set_color("black")
-
- # Per-tick styling: challenger red+bold, baseline bold for key comparisons
- for _i in range(_n_comparisons):
- if _is_tactics_vs_legacy(_left_labels[_i], _right_labels[_i]):
- _ax.get_yticklabels()[_i].set_color(_COLOR_KEY_TICK)
- _ax.get_yticklabels()[_i].set_fontweight("bold")
- _ax_right.get_yticklabels()[_i].set_fontweight("bold")
-
- _ax.set_xlabel(
- f"\u0394 Recovery: Challenger \u2212 Baseline (top-{TOP_N})",
- fontsize=14, color="black",
- )
- _ax.set_title(
- f"Tukey HSD Pairwise CIs: {_lib_id} (top-{TOP_N})",
- fontsize=15, fontweight="bold", color="black",
- )
- _ax.grid(axis="x", alpha=0.3, zorder=1, color="grey")
- _ax.tick_params(axis="x", colors="black", labelsize=12)
- _ax.tick_params(axis="y", color="black", labelsize=12)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _legend_elements = [
- Line2D([0], [0], color="#4C78A8", linewidth=2.5, marker="o",
- markersize=8, label="Significant (p < 0.05)"),
- Line2D([0], [0], color="#999999", linewidth=2.5, marker="o",
- markersize=8, label="Not significant"),
- ]
- _ax.legend(handles=_legend_elements, loc="lower right", fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black")
-
- _fig.tight_layout()
-
- _pdf_name = f"tukey_ci_{_lib_id}_top{TOP_N}.pdf".replace(" ", "_")
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=_pdf_name,
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(Line2D, TOP_N, df, fig_to_pdf_bytes, mo, np, pl, plt):
- """Across-library Tukey HSD pairwise CI plot (ROCS libraries only).
-
- Excludes docking libraries (thrombin, adenine, amide, quinazoline) which
- have different score distributions and only 1 query each. Docking results
- are shown in the separate docking Tukey HSD CI plot.
-
- Averages replicates within each (library, query) to get one independent
- observation per query, then runs statsmodels pairwise_tukeyhsd for
- FWER-controlled simultaneous CIs and adjusted p-values.
-
- Layout matches the per-library Tukey CI plot: positive values always mean
- the challenger (right y-axis) outperforms the baseline (left y-axis).
- """
- from statsmodels.stats.multicomp import pairwise_tukeyhsd
-
- # Exclude docking libraries — different score distributions, 1 query each
- _DOCKING_LIBS = {"thrombin", "adenine", "amide", "quinazoline"}
- _rocs_df = df.filter(~pl.col("library_id").is_in(_DOCKING_LIBS))
-
- # Average replicates: one observation per (library, query, method)
- _query_means = (
- _rocs_df.group_by(["library_id", "query_id", "method"])
- .agg(pl.col("recovered").mean().alias("mean_recovered"))
- )
-
- _values = _query_means["mean_recovered"].to_numpy()
- _groups = _query_means["method"].to_list()
-
- _tukey = pairwise_tukeyhsd(_values, _groups, alpha=0.05)
- _summary = _tukey.summary()
-
- # Parse Tukey HSD results into lists
- _left_labels = [] # baseline (worse method)
- _right_labels = [] # challenger (better method)
- _grand_means = []
- _ci_lows = []
- _ci_highs = []
- _p_values = []
- _rejects = []
-
- for _row in _summary.data[1:]: # skip header row
- _g1, _g2, _meandiff, _p_adj, _lower, _upper, _reject = _row
- _meandiff = float(_meandiff)
- _p_adj = float(_p_adj)
- _lower = float(_lower)
- _upper = float(_upper)
-
- # Normalize direction: positive = challenger (right) is better.
- if _meandiff >= 0:
- _left_labels.append(_g1)
- _right_labels.append(_g2)
- _grand_means.append(_meandiff)
- _ci_lows.append(_lower)
- _ci_highs.append(_upper)
- else:
- _left_labels.append(_g2)
- _right_labels.append(_g1)
- _grand_means.append(-_meandiff)
- _ci_lows.append(-_upper)
- _ci_highs.append(-_lower)
-
- _p_values.append(_p_adj)
- _rejects.append(bool(_reject))
-
- _grand_means = np.array(_grand_means)
- _ci_lows = np.array(_ci_lows)
- _ci_highs = np.array(_ci_highs)
- _p_values = np.array(_p_values)
-
- # Sort by grand mean descending (largest positive diff at top)
- _order = np.argsort(_grand_means)[::-1]
- _left_labels = [_left_labels[i] for i in _order]
- _right_labels = [_right_labels[i] for i in _order]
- _rejects = [_rejects[i] for i in _order]
- _grand_means = _grand_means[_order]
- _ci_lows = _ci_lows[_order]
- _ci_highs = _ci_highs[_order]
- _p_values = _p_values[_order]
-
- _n_comparisons = len(_left_labels)
- _y = np.arange(_n_comparisons)
-
- _fig, _ax = plt.subplots(
- figsize=(12, max(4, _n_comparisons * 0.55)),
- facecolor="white",
- )
- _ax.set_facecolor("white")
-
- # Define the key TACTICS-vs-Legacy comparisons to highlight
- _legacy_methods = {"Legacy Enhanced-RWS", "Legacy Standard-Greedy"}
- _tactics_adaptive = {"TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"}
-
- def _is_tactics_vs_legacy(_left, _right):
- """True if one side is a legacy method and the other is a TACTICS adaptive method."""
- return (
- (_left in _legacy_methods and _right in _tactics_adaptive)
- or (_left in _tactics_adaptive and _right in _legacy_methods)
- )
-
- _COLOR_SIG = "#4C78A8" # blue — significant pairs
- _COLOR_NS = "#999999" # grey — not significant
- _COLOR_KEY_TICK = "#4C78A8" # blue — TACTICS vs Legacy tick labels
-
- for _i in range(_n_comparisons):
- _color = _COLOR_SIG if _rejects[_i] else _COLOR_NS
-
- # CI line
- _ax.plot(
- [_ci_lows[_i], _ci_highs[_i]], [_i, _i],
- color=_color, linewidth=2.5, solid_capstyle="round", zorder=3,
- )
- # Point estimate
- _ax.plot(
- _grand_means[_i], _i, "o",
- color=_color, markersize=8, zorder=4,
- )
- # Significance stars (Tukey HSD adjusted p-values, FWER=0.05)
- _p = _p_values[_i]
- _sig_str = (
- "****" if _p <= 0.0001
- else "***" if _p <= 0.001
- else "**" if _p <= 0.01
- else "*" if _p <= 0.05
- else "ns"
- )
- _ax.text(
- _ci_highs[_i] + 0.15, _i, _sig_str,
- ha="left", va="center",
- fontsize=12, fontweight="bold", color="black",
- )
-
- _ax.axvline(0, color="black", linewidth=0.8, linestyle="-", zorder=2)
-
- # Expand x-limits so stars and CI lines fit inside the axes
- _x_pad = max(_ci_highs.max(), abs(_ci_lows.min())) * 0.18
- _ax.set_xlim(
- min(_ci_lows.min(), 0) - _x_pad * 0.5,
- _ci_highs.max() + _x_pad,
- )
-
- # Left y-axis: challenger (better method)
- _ax.set_yticks(_y)
- _ax.set_yticklabels(_right_labels, fontsize=12, color="black")
- _ax.set_ylabel("Challenger", fontsize=14, color="black")
-
- # Pad y-limits so effect size labels above top/bottom rows aren't clipped
- _ax.set_ylim(-0.6, _n_comparisons - 0.4)
-
- # Invert y so largest diff is at top (before twinx so both share orientation)
- _ax.invert_yaxis()
-
- # Right y-axis: baseline (worse method)
- _ax_right = _ax.twinx()
- _ax_right.set_ylim(_ax.get_ylim())
- _ax_right.set_yticks(_y)
- _ax_right.set_yticklabels(_left_labels, fontsize=12, color="black")
- _ax_right.set_ylabel("Baseline", fontsize=14, color="black")
- _ax_right.tick_params(colors="black", labelsize=12)
- for _spine in _ax_right.spines.values():
- _spine.set_color("black")
-
- # Per-tick styling: challenger red+bold, baseline bold for key comparisons
- # Applied after both axes are fully configured so nothing resets them
- for _i in range(_n_comparisons):
- if _is_tactics_vs_legacy(_left_labels[_i], _right_labels[_i]):
- _ax.get_yticklabels()[_i].set_color(_COLOR_KEY_TICK)
- _ax.get_yticklabels()[_i].set_fontweight("bold")
- _ax_right.get_yticklabels()[_i].set_fontweight("bold")
-
- _ax.set_xlabel(
- f"\u0394 Recovery: Challenger \u2212 Baseline (top-{TOP_N})",
- fontsize=14, color="black",
- )
- _n_rocs_libs = _rocs_df["library_id"].n_unique()
- _ax.set_title(
- f"Tukey HSD Pairwise CIs \u2014 {_n_rocs_libs} ROCS Libraries (top-{TOP_N}, FWER=0.05)",
- fontsize=15, fontweight="bold", color="black",
- )
- _ax.grid(axis="x", alpha=0.3, zorder=1, color="grey")
- _ax.tick_params(axis="x", colors="black", labelsize=12)
- _ax.tick_params(axis="y", color="black", labelsize=12)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- _legend_elements = [
- Line2D([0], [0], color=_COLOR_SIG, linewidth=2.5, marker="o",
- markersize=8, label="Significant (p < 0.05)"),
- Line2D([0], [0], color=_COLOR_NS, linewidth=2.5, marker="o",
- markersize=8, label="Not significant"),
- ]
- _ax.legend(handles=_legend_elements, loc="lower right", fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black")
-
- _fig.tight_layout()
-
- _pdf_name = f"tukey_hsd_ci_across_libraries_top{TOP_N}.pdf"
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=_pdf_name,
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(Line2D, TOP_N, df, fig_to_pdf_bytes, mo, np, pl, plt):
- """Across-library Tukey HSD pairwise CI plot (docking libraries only).
-
- Docking libraries have 1 query each with 20 replicates.
- Each replicate is an independent algorithm run, so replicate is the unit of
- observation. Runs statsmodels pairwise_tukeyhsd for FWER-controlled CIs.
- """
- from statsmodels.stats.multicomp import pairwise_tukeyhsd as _pairwise_tukeyhsd
-
- _DOCK_LIBS = {"thrombin", "adenine", "amide", "quinazoline"}
- dock_df = df.filter(pl.col("library_id").is_in(_DOCK_LIBS))
-
- mo.stop(
- len(dock_df) == 0,
- mo.md("No docking library data at this top-N."),
- )
-
- dock_values = dock_df["recovered"].to_numpy()
- dock_groups = dock_df["method"].to_list()
-
- dock_tukey = _pairwise_tukeyhsd(dock_values, dock_groups, alpha=0.05)
- dock_summary = dock_tukey.summary()
-
- # Parse results
- dock_query_means = (
- dock_df.group_by(["library_id", "query_id", "method"])
- .agg(pl.col("recovered").mean().alias("mean_recovered"))
- )
-
- dock_left = []
- dock_right = []
- dock_means = []
- dock_ci_lo = []
- dock_ci_hi = []
- dock_pvals = []
- dock_reject = []
-
- for dock_row in dock_summary.data[1:]:
- dg1, dg2, d_meandiff, d_padj, d_lower, d_upper, d_reject = dock_row
- d_meandiff = float(d_meandiff)
- d_padj = float(d_padj)
- d_lower = float(d_lower)
- d_upper = float(d_upper)
-
- if d_meandiff >= 0:
- dock_left.append(dg1)
- dock_right.append(dg2)
- dock_means.append(d_meandiff)
- dock_ci_lo.append(d_lower)
- dock_ci_hi.append(d_upper)
- else:
- dock_left.append(dg2)
- dock_right.append(dg1)
- dock_means.append(-d_meandiff)
- dock_ci_lo.append(-d_upper)
- dock_ci_hi.append(-d_lower)
-
- dock_pvals.append(d_padj)
- dock_reject.append(bool(d_reject))
-
- dock_means = np.array(dock_means)
- dock_ci_lo = np.array(dock_ci_lo)
- dock_ci_hi = np.array(dock_ci_hi)
- dock_pvals = np.array(dock_pvals)
-
- dock_order = np.argsort(dock_means)[::-1]
- dock_left = [dock_left[i] for i in dock_order]
- dock_right = [dock_right[i] for i in dock_order]
- dock_reject = [dock_reject[i] for i in dock_order]
- dock_means = dock_means[dock_order]
- dock_ci_lo = dock_ci_lo[dock_order]
- dock_ci_hi = dock_ci_hi[dock_order]
- dock_pvals = dock_pvals[dock_order]
-
- dock_n_comp = len(dock_left)
- dock_y = np.arange(dock_n_comp)
-
- dock_fig, dock_ax = plt.subplots(
- figsize=(12, max(4, dock_n_comp * 0.55)),
- facecolor="white",
- )
- dock_ax.set_facecolor("white")
-
- _LEGACY = {"Legacy Enhanced-RWS", "Legacy Standard-Greedy"}
- _TACTICS = {"TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"}
- _CLR_SIG = "#4C78A8"
- _CLR_NS = "#999999"
- _CLR_KEY_TICK = "#4C78A8"
-
- def _is_tvl(left, right):
- return (
- (left in _LEGACY and right in _TACTICS)
- or (left in _TACTICS and right in _LEGACY)
- )
-
- for di in range(dock_n_comp):
- clr = _CLR_SIG if dock_reject[di] else _CLR_NS
-
- dock_ax.plot(
- [dock_ci_lo[di], dock_ci_hi[di]], [di, di],
- color=clr, linewidth=2.5, solid_capstyle="round", zorder=3,
- )
- dock_ax.plot(
- dock_means[di], di, "o",
- color=clr, markersize=8, zorder=4,
- )
- dp = dock_pvals[di]
- d_sig = (
- "****" if dp <= 0.0001
- else "***" if dp <= 0.001
- else "**" if dp <= 0.01
- else "*" if dp <= 0.05
- else "ns"
- )
- dock_ax.text(
- dock_ci_hi[di] + 0.5, di, d_sig,
- ha="left", va="center",
- fontsize=12, fontweight="bold", color="black",
- )
-
- dock_ax.axvline(0, color="black", linewidth=0.8, linestyle="-", zorder=2)
-
- dock_x_pad = max(dock_ci_hi.max(), abs(dock_ci_lo.min())) * 0.18
- dock_ax.set_xlim(
- min(dock_ci_lo.min(), 0) - dock_x_pad * 0.5,
- dock_ci_hi.max() + dock_x_pad,
- )
-
- dock_ax.set_yticks(dock_y)
- dock_ax.set_yticklabels(dock_right, fontsize=12, color="black")
- dock_ax.set_ylabel("Challenger", fontsize=14, color="black")
-
- dock_ax.set_ylim(-0.6, dock_n_comp - 0.4)
- dock_ax.invert_yaxis()
-
- dock_ax_right = dock_ax.twinx()
- dock_ax_right.set_ylim(dock_ax.get_ylim())
- dock_ax_right.set_yticks(dock_y)
- dock_ax_right.set_yticklabels(dock_left, fontsize=12, color="black")
- dock_ax_right.set_ylabel("Baseline", fontsize=14, color="black")
- dock_ax_right.tick_params(colors="black", labelsize=12)
- for sp in dock_ax_right.spines.values():
- sp.set_color("black")
-
- # Per-tick styling applied last so nothing resets them
- for di in range(dock_n_comp):
- if _is_tvl(dock_left[di], dock_right[di]):
- dock_ax.get_yticklabels()[di].set_color(_CLR_KEY_TICK)
- dock_ax.get_yticklabels()[di].set_fontweight("bold")
- dock_ax_right.get_yticklabels()[di].set_fontweight("bold")
-
- dock_ax.set_xlabel(
- f"\u0394 Recovery: Challenger \u2212 Baseline (top-{TOP_N})",
- fontsize=14, color="black",
- )
- dock_ax.set_title(
- f"Tukey HSD Pairwise CIs \u2014 4 Docking Datasets (top-{TOP_N}, FWER=0.05)",
- fontsize=15, fontweight="bold", color="black",
- )
- dock_ax.grid(axis="x", alpha=0.3, zorder=1, color="grey")
- dock_ax.tick_params(axis="x", colors="black", labelsize=12)
- dock_ax.tick_params(axis="y", color="black", labelsize=12)
- for sp in dock_ax.spines.values():
- sp.set_color("black")
-
- dock_legend = [
- Line2D([0], [0], color=_CLR_SIG, linewidth=2.5, marker="o",
- markersize=8, label="Significant (p < 0.05)"),
- Line2D([0], [0], color=_CLR_NS, linewidth=2.5, marker="o",
- markersize=8, label="Not significant"),
- ]
- dock_ax.legend(handles=dock_legend, loc="lower right", fontsize=12,
- facecolor="white", edgecolor="black", labelcolor="black")
-
- dock_fig.tight_layout()
-
- dock_pdf_name = f"tukey_hsd_ci_docking_top{TOP_N}.pdf"
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(dock_fig),
- filename=dock_pdf_name,
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(mo):
- mo.md(r"""
- ## SAR Landscape Characterization — Reagent Posterior Separation
-
- Riley (2024) showed that for ROCS, reagents producing top compounds have high means
- and low variance (they "stand out from the crowd"), while for docking, top-hit reagents
- are statistically indistinguishable from mediocre ones. This plot reproduces that
- analysis for adenine and quinazoline docking libraries.
- """)
- return
-
-
-@app.cell
-def _(fig_to_pdf_bytes, mo, np, pl, plt, project_root):
- """Reagent posterior separation plot (Riley-style) for docking libraries.
-
- For each reagent, computes (mean score, std score) across all products containing
- that reagent. Colors reagents by how many times they appear in top-100 compounds
- (continuous colormap), revealing hit concentration patterns.
- """
- import re as _re
- from matplotlib.colors import Normalize as _Normalize
- from matplotlib import cm as _cm
-
- _SCORES_DIR = project_root / "data" / "scores"
- _TOP_N = 100
-
- # Library configs: (library_id, score_path, component_names, parse_function)
- def _parse_adenine(code):
- m = _re.match(r"(amidine_\d+)_(isocyanide_db_\d+)_(aldehyde_\d+)", code)
- return m.groups() if m else (None, None, None)
-
- def _parse_quinazoline(code):
- parts = code.split("_")
- return tuple(parts) if len(parts) == 3 else (None, None, None)
-
- _libraries = [
- {
- "lib_id": "adenine",
- "components": ["Amidines", "Isocyanides", "Aldehydes"],
- "parse": _parse_adenine,
- },
- {
- "lib_id": "quinazoline",
- "components": ["Aminobenzoics", "Amines", "Acids"],
- "parse": _parse_quinazoline,
- },
- ]
-
- # Use gridspec to reserve a narrow column on the right for the colorbar
- _gs = plt.GridSpec(2, 4, width_ratios=[1, 1, 1, 0.05], wspace=0.35)
- _fig = plt.figure(figsize=(20, 11), facecolor="white")
- _axes = [[_fig.add_subplot(_gs[r, c]) for c in range(3)] for r in range(2)]
-
- # Shared colormap for top-N count
- _cmap = _cm.YlOrRd
-
- # First pass: compute per-reagent stats for all libraries/components,
- # track global max count for consistent colorbar scaling
- _global_max_count = 0
- _plot_data = [] # list of (row_idx, comp_idx, means, stds, counts, unique_reagents)
-
- for _row_idx, _lib_cfg in enumerate(_libraries):
- _lib_id = _lib_cfg["lib_id"]
- _parse_fn = _lib_cfg["parse"]
- _comp_names = _lib_cfg["components"]
-
- _score_df = pl.read_parquet(_SCORES_DIR / f"{_lib_id}.parquet")
- _score_df = _score_df.filter(pl.col("Scores").is_not_nan())
-
- _top_n_codes = (
- _score_df.sort("Scores", descending=False)
- .head(_TOP_N)["Product_Code"]
- .to_list()
- )
-
- _codes = _score_df["Product_Code"].to_list()
- _scores = _score_df["Scores"].to_numpy()
-
- _parsed = [_parse_fn(c) for c in _codes]
- _parsed_top = [_parse_fn(c) for c in _top_n_codes]
- _n_comp = len(_comp_names)
-
- for _comp_idx in range(_n_comp):
- _valid_mask = [p[_comp_idx] is not None for p in _parsed]
- _reagent_ids = [p[_comp_idx] for p, v in zip(_parsed, _valid_mask) if v]
- _reagent_scores = _scores[[i for i, v in enumerate(_valid_mask) if v]]
-
- _top_counts = {}
- for _p in _parsed_top:
- if _p[_comp_idx] is not None:
- _rid = _p[_comp_idx]
- _top_counts[_rid] = _top_counts.get(_rid, 0) + 1
-
- _reagent_to_idx = {}
- for _i, _rid in enumerate(_reagent_ids):
- _reagent_to_idx.setdefault(_rid, []).append(_i)
-
- _unique_reagents = sorted(set(_reagent_ids))
-
- _means = []
- _stds = []
- _counts = []
-
- for _rid in _unique_reagents:
- _idxs = _reagent_to_idx[_rid]
- _s = _reagent_scores[_idxs]
- _means.append(float(np.mean(_s)))
- _stds.append(float(np.std(_s)))
- _counts.append(_top_counts.get(_rid, 0))
-
- _means = np.array(_means)
- _stds = np.array(_stds)
- _counts = np.array(_counts)
-
- _hit_counts = _counts[_counts > 0]
- if len(_hit_counts) > 0:
- _global_max_count = max(_global_max_count, int(_hit_counts.max()))
-
- _plot_data.append((_row_idx, _comp_idx, _means, _stds, _counts, _unique_reagents))
-
- # Shared norm: 1 to observed max count
- _norm = _Normalize(vmin=1, vmax=max(_global_max_count, 2))
-
- # Second pass: z-score normalize and plot
- for _row_idx, _comp_idx, _means, _stds, _counts, _unique_reagents in _plot_data:
- _ax = _axes[_row_idx][_comp_idx]
- _ax.set_facecolor("white")
-
- # Z-score normalize means and stds so all subplots share the same scale
- _mean_mu, _mean_sigma = np.mean(_means), np.std(_means)
- _std_mu, _std_sigma = np.mean(_stds), np.std(_stds)
- _z_means = (_means - _mean_mu) / _mean_sigma if _mean_sigma > 0 else _means * 0
- _z_stds = (_stds - _std_mu) / _std_sigma if _std_sigma > 0 else _stds * 0
-
- _zero_mask = _counts == 0
- _ax.scatter(
- _z_means[_zero_mask], _z_stds[_zero_mask],
- c="#CCCCCC", s=40, alpha=0.6, edgecolors="none",
- label="Not in top-100", zorder=2,
- )
-
- _hit_mask = _counts > 0
- if _hit_mask.any():
- _ax.scatter(
- _z_means[_hit_mask], _z_stds[_hit_mask],
- c=_counts[_hit_mask], cmap=_cmap, norm=_norm,
- s=60, alpha=0.9, edgecolors="black", linewidths=0.5,
- zorder=3,
- )
-
- _lib_cfg = _libraries[_row_idx]
- _comp_names = _lib_cfg["components"]
- _ax.set_xlabel("Reagent Mean Score (z)", fontsize=14, color="black")
- if _comp_idx == 0:
- _ax.set_ylabel("Reagent Stdev (z)", fontsize=14, color="black")
- _n_in_top = int((_counts > 0).sum())
- _ax.set_title(
- f"{_comp_names[_comp_idx]} ({_n_in_top}/{len(_unique_reagents)} in top-100)",
- fontsize=14, fontweight="bold", color="black",
- )
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- # Row labels
- for _row_idx, _lib_cfg in enumerate(_libraries):
- _axes[_row_idx][0].annotate(
- _lib_cfg["lib_id"].capitalize(),
- xy=(-0.35, 0.5), xycoords="axes fraction",
- fontsize=16, fontweight="bold", color="black",
- rotation=90, va="center", ha="center",
- )
-
- # Colorbar in dedicated gridspec column (right side, spanning both rows)
- _cbar_ax = _fig.add_subplot(_gs[:, 3])
- _cbar = _fig.colorbar(
- _cm.ScalarMappable(norm=_norm, cmap=_cmap),
- cax=_cbar_ax,
- )
- _cbar.set_label(f"Count in top-{_TOP_N}", fontsize=14, color="black")
- _cbar.ax.tick_params(colors="black", labelsize=12)
-
- _fig.suptitle(
- f"SAR Landscape: Reagent Score Distributions vs Top-{_TOP_N} Hit Count (Docking)\n"
- f"3-Component Docking Libraries",
- fontsize=16, fontweight="bold", color="black", y=1.01,
- )
- _fig.tight_layout()
-
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="reagent_posterior_separation_docking.pdf",
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(fig_to_pdf_bytes, mo, np, pl, plt, project_root):
- """Reagent posterior separation plot for 2-component docking libraries (thrombin, amide)."""
- from matplotlib.colors import Normalize as _Normalize2
- from matplotlib import cm as _cm2
-
- _SCORES_DIR = project_root / "data" / "scores"
- _TOP_N = 100
-
- def _parse_thrombin(code):
- parts = code.split("_")
- if len(parts) == 3:
- return (parts[0], parts[1] + "_" + parts[2])
- return (None, None)
-
- def _parse_amide(code):
- parts = code.split("_")
- return tuple(parts) if len(parts) == 2 else (None, None)
-
- _libraries = [
- {
- "lib_id": "thrombin",
- "components": ["Acids (130)", "Dipeptides (3,844)"],
- "parse": _parse_thrombin,
- },
- {
- "lib_id": "amide",
- "components": ["Acids (10,000)", "Amines (1,000)"],
- "parse": _parse_amide,
- },
- ]
-
- # Use gridspec to reserve a narrow column on the right for the colorbar
- _gs = plt.GridSpec(2, 3, width_ratios=[1, 1, 0.05], wspace=0.35)
- _fig = plt.figure(figsize=(16, 11), facecolor="white")
- _axes = [[_fig.add_subplot(_gs[r, c]) for c in range(2)] for r in range(2)]
-
- _cmap = _cm2.YlOrRd
-
- # First pass: compute per-reagent stats, track global max count
- _global_max_count = 0
- _plot_data = []
-
- for _row_idx, _lib_cfg in enumerate(_libraries):
- _parse_fn = _lib_cfg["parse"]
- _comp_names = _lib_cfg["components"]
- _n_comp = len(_comp_names)
-
- _score_df = pl.read_parquet(_SCORES_DIR / f"{_lib_cfg['lib_id']}.parquet")
- _score_df = _score_df.filter(pl.col("Scores").is_not_nan())
-
- _top_n_codes = (
- _score_df.sort("Scores", descending=False)
- .head(_TOP_N)["Product_Code"]
- .to_list()
- )
-
- _codes = _score_df["Product_Code"].to_list()
- _scores = _score_df["Scores"].to_numpy()
-
- _parsed = [_parse_fn(c) for c in _codes]
- _parsed_top = [_parse_fn(c) for c in _top_n_codes]
-
- for _comp_idx in range(_n_comp):
- _valid_mask = [p[_comp_idx] is not None for p in _parsed]
- _reagent_ids = [p[_comp_idx] for p, v in zip(_parsed, _valid_mask) if v]
- _reagent_scores = _scores[[i for i, v in enumerate(_valid_mask) if v]]
-
- _top_counts = {}
- for _p in _parsed_top:
- if _p[_comp_idx] is not None:
- _rid = _p[_comp_idx]
- _top_counts[_rid] = _top_counts.get(_rid, 0) + 1
-
- _reagent_to_idx = {}
- for _i, _rid in enumerate(_reagent_ids):
- _reagent_to_idx.setdefault(_rid, []).append(_i)
-
- _unique_reagents = sorted(set(_reagent_ids))
-
- _means = []
- _stds = []
- _counts = []
-
- for _rid in _unique_reagents:
- _idxs = _reagent_to_idx[_rid]
- _s = _reagent_scores[_idxs]
- _means.append(float(np.mean(_s)))
- _stds.append(float(np.std(_s)))
- _counts.append(_top_counts.get(_rid, 0))
-
- _means = np.array(_means)
- _stds = np.array(_stds)
- _counts = np.array(_counts)
-
- _hit_counts = _counts[_counts > 0]
- if len(_hit_counts) > 0:
- _global_max_count = max(_global_max_count, int(_hit_counts.max()))
-
- _plot_data.append((_row_idx, _comp_idx, _means, _stds, _counts, _unique_reagents))
-
- # Shared norm: 1 to observed max count
- _norm = _Normalize2(vmin=1, vmax=max(_global_max_count, 2))
-
- # Second pass: z-score normalize and plot
- for _row_idx, _comp_idx, _means, _stds, _counts, _unique_reagents in _plot_data:
- _ax = _axes[_row_idx][_comp_idx]
- _ax.set_facecolor("white")
-
- # Z-score normalize means and stds so all subplots share the same scale
- _mean_mu, _mean_sigma = np.mean(_means), np.std(_means)
- _std_mu, _std_sigma = np.mean(_stds), np.std(_stds)
- _z_means = (_means - _mean_mu) / _mean_sigma if _mean_sigma > 0 else _means * 0
- _z_stds = (_stds - _std_mu) / _std_sigma if _std_sigma > 0 else _stds * 0
-
- _zero_mask = _counts == 0
- _ax.scatter(
- _z_means[_zero_mask], _z_stds[_zero_mask],
- c="#CCCCCC", s=40, alpha=0.6, edgecolors="none",
- label="Not in top-100", zorder=2,
- )
-
- _hit_mask = _counts > 0
- if _hit_mask.any():
- _ax.scatter(
- _z_means[_hit_mask], _z_stds[_hit_mask],
- c=_counts[_hit_mask], cmap=_cmap, norm=_norm,
- s=60, alpha=0.9, edgecolors="black", linewidths=0.5,
- zorder=3,
- )
-
- _lib_cfg = _libraries[_row_idx]
- _comp_names = _lib_cfg["components"]
- _ax.set_xlabel("Reagent Mean Score (z)", fontsize=14, color="black")
- if _comp_idx == 0:
- _ax.set_ylabel("Reagent Stdev (z)", fontsize=14, color="black")
- _n_in_top = int((_counts > 0).sum())
- _ax.set_title(
- f"{_comp_names[_comp_idx]} ({_n_in_top}/{len(_unique_reagents)} in top-100)",
- fontsize=14, fontweight="bold", color="black",
- )
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(alpha=0.3, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- # Row labels
- for _row_idx, _lib_cfg in enumerate(_libraries):
- _axes[_row_idx][0].annotate(
- _lib_cfg["lib_id"].capitalize(),
- xy=(-0.35, 0.5), xycoords="axes fraction",
- fontsize=16, fontweight="bold", color="black",
- rotation=90, va="center", ha="center",
- )
-
- # Colorbar in dedicated gridspec column (right side, spanning both rows)
- _cbar_ax = _fig.add_subplot(_gs[:, 2])
- _cbar = _fig.colorbar(
- _cm2.ScalarMappable(norm=_norm, cmap=_cmap),
- cax=_cbar_ax,
- )
- _cbar.set_label(f"Count in top-{_TOP_N}", fontsize=14, color="black")
- _cbar.ax.tick_params(colors="black", labelsize=12)
-
- _fig.suptitle(
- f"SAR Landscape: Reagent Score Distributions vs Top-{_TOP_N} Hit Count\n"
- f"2-Component Docking Libraries",
- fontsize=16, fontweight="bold", color="black", y=1.02,
- )
- _fig.tight_layout()
-
- mo.vstack([
- plt.gcf(),
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename="reagent_posterior_separation_2comp_docking.pdf",
- mimetype="application/pdf",
- label="Download as PDF",
- ),
- ])
- return
-
-
-@app.cell
-def _(mo):
- mo.md(r"""
- ## SAR Landscape vs Top-N Threshold (Riley-style multi-panel)
-
- These plots show how reagent posterior separation changes as the top-N threshold
- widens. Each row is a top-N value (25–250), each column is a reaction component.
- As top-N increases, more reagents enter the hit set and the colored points drift
- toward z=0 — the SAR becomes less distinctive. Libraries where hit reagents are
- barely separated at narrow top-N (e.g., adenine) are the hardest for greedy methods.
- """)
- return
-
-
-@app.cell
-def _(fig_to_pdf_bytes, mo, np, pl, plt, project_root):
- """Multi-top-N posterior separation grid for all 4 docking libraries."""
- import re as _re3
- from matplotlib.colors import Normalize as _Normalize3
- from matplotlib import cm as _cm3
-
- _SCORES_DIR = project_root / "data" / "scores"
- _TOP_N_VALUES = [25, 50, 100, 150, 200, 250]
-
- # --- Library definitions ---
- def _parse_adenine3(code):
- _m = _re3.match(r"(amidine_\d+)_(isocyanide_db_\d+)_(aldehyde_\d+)", code)
- return _m.groups() if _m else None
-
- def _parse_quinazoline3(code):
- parts = code.split("_")
- return tuple(parts) if len(parts) == 3 else None
-
- def _parse_thrombin3(code):
- parts = code.split("_")
- if len(parts) == 3:
- return (parts[0], parts[1] + "_" + parts[2])
- return None
-
- def _parse_amide3(code):
- parts = code.split("_")
- return tuple(parts) if len(parts) == 2 else None
-
- _libraries = [
- {
- "lib_id": "adenine",
- "components": ["Amidines", "Isocyanides", "Aldehydes"],
- "parse": _parse_adenine3,
- },
- {
- "lib_id": "quinazoline",
- "components": ["Aminobenzoics", "Amines", "Acids"],
- "parse": _parse_quinazoline3,
- },
- {
- "lib_id": "thrombin",
- "components": ["Acids", "Dipeptides"],
- "parse": _parse_thrombin3,
- },
- {
- "lib_id": "amide",
- "components": ["Acids", "Amines"],
- "parse": _parse_amide3,
- },
- ]
-
- _cmap = _cm3.YlOrRd
- _figs = []
-
- for _lib_cfg in _libraries:
- _lib_id = _lib_cfg["lib_id"]
- _parse_fn = _lib_cfg["parse"]
- _comp_names = _lib_cfg["components"]
- _n_comp = len(_comp_names)
-
- _score_df = pl.read_parquet(_SCORES_DIR / f"{_lib_id}.parquet")
- _score_df = _score_df.filter(pl.col("Scores").is_not_nan())
-
- _codes = _score_df["Product_Code"].to_list()
- _scores = _score_df["Scores"].to_numpy()
- _sorted_idx = np.argsort(_scores) # minimize for docking
-
- _parsed = [_parse_fn(c) for c in _codes]
-
- # Pre-compute per-reagent stats (shared across top-N rows)
- _reagent_data = [] # one per component
- for _comp_idx in range(_n_comp):
- _valid = [(i, p[_comp_idx]) for i, p in enumerate(_parsed) if p is not None]
- _reagent_to_idx = {}
- for _i, _rid in _valid:
- _reagent_to_idx.setdefault(_rid, []).append(_i)
-
- _unique_reagents = sorted(_reagent_to_idx.keys())
- _means = np.array([float(np.mean(_scores[_reagent_to_idx[r]])) for r in _unique_reagents])
- _stds = np.array([float(np.std(_scores[_reagent_to_idx[r]])) for r in _unique_reagents])
-
- # Z-score
- _mean_mu, _mean_sigma = np.mean(_means), np.std(_means)
- _std_mu, _std_sigma = np.mean(_stds), np.std(_stds)
- _z_means = (_means - _mean_mu) / _mean_sigma if _mean_sigma > 0 else _means * 0
- _z_stds = (_stds - _std_mu) / _std_sigma if _std_sigma > 0 else _stds * 0
-
- _reagent_data.append({
- "unique_reagents": _unique_reagents,
- "z_means": _z_means,
- "z_stds": _z_stds,
- "reagent_to_idx": _reagent_to_idx,
- })
-
- # Find global max count across all top-N panels for this library
- _global_max = 0
- for _top_n in _TOP_N_VALUES:
- _top_codes = [_codes[i] for i in _sorted_idx[:_top_n]]
- _top_parsed = [_parse_fn(c) for c in _top_codes]
- for _comp_idx in range(_n_comp):
- _counts = {}
- for _p in _top_parsed:
- if _p is not None and _p[_comp_idx] is not None:
- _rid = _p[_comp_idx]
- _counts[_rid] = _counts.get(_rid, 0) + 1
- if _counts:
- _global_max = max(_global_max, max(_counts.values()))
-
- _norm = _Normalize3(vmin=1, vmax=max(_global_max, 2))
-
- # Build grid: rows = top-N thresholds, cols = components + colorbar
- _n_rows = len(_TOP_N_VALUES)
- _width_ratios = [1] * _n_comp + [0.05]
- _gs = plt.GridSpec(_n_rows, _n_comp + 1, width_ratios=_width_ratios, wspace=0.3, hspace=0.4)
- _fig = plt.figure(figsize=(6 * _n_comp + 1, 4 * _n_rows), facecolor="white")
-
- for _row_idx, _top_n in enumerate(_TOP_N_VALUES):
- _top_codes = [_codes[i] for i in _sorted_idx[:_top_n]]
- _top_parsed = [_parse_fn(c) for c in _top_codes]
-
- for _comp_idx in range(_n_comp):
- _ax = _fig.add_subplot(_gs[_row_idx, _comp_idx])
- _ax.set_facecolor("white")
-
- _rd = _reagent_data[_comp_idx]
- _z_means = _rd["z_means"]
- _z_stds = _rd["z_stds"]
- _unique_reagents = _rd["unique_reagents"]
-
- # Count per-reagent appearances in this top-N
- _top_counts = {}
- for _p in _top_parsed:
- if _p is not None and _p[_comp_idx] is not None:
- _rid = _p[_comp_idx]
- _top_counts[_rid] = _top_counts.get(_rid, 0) + 1
-
- _counts = np.array([_top_counts.get(r, 0) for r in _unique_reagents])
-
- # Grey background points
- _zero_mask = _counts == 0
- _ax.scatter(
- _z_means[_zero_mask], _z_stds[_zero_mask],
- c="#CCCCCC", s=20, alpha=0.4, edgecolors="none", zorder=2,
- )
-
- # Colored hit points
- _hit_mask = _counts > 0
- if _hit_mask.any():
- _ax.scatter(
- _z_means[_hit_mask], _z_stds[_hit_mask],
- c=_counts[_hit_mask], cmap=_cmap, norm=_norm,
- s=30, alpha=0.9, edgecolors="black", linewidths=0.3,
- zorder=3,
- )
-
- # Axis labels only on edges
- if _row_idx == _n_rows - 1:
- _ax.set_xlabel("Reagent Mean Score (z)", fontsize=12, color="black")
- if _comp_idx == 0:
- _ax.set_ylabel("Reagent Stdev (z)", fontsize=12, color="black")
-
- # Column titles on first row only
- if _row_idx == 0:
- _n_total = len(_unique_reagents)
- _ax.set_title(f"{_comp_names[_comp_idx]} (n={_n_total})",
- fontsize=13, fontweight="bold", color="black")
-
- # Row label on rightmost component column
- if _comp_idx == _n_comp - 1:
- _n_hit = int((_counts > 0).sum())
- _ax.annotate(
- f"Top-{_top_n}\n({_n_hit} reagents)",
- xy=(1.08, 0.5), xycoords="axes fraction",
- fontsize=12, fontweight="bold", color="black",
- rotation=-90, va="center", ha="left",
- )
-
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(alpha=0.2, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- # Shared colorbar
- _cbar_ax = _fig.add_subplot(_gs[:, _n_comp])
- _cbar = _fig.colorbar(
- _cm3.ScalarMappable(norm=_norm, cmap=_cmap),
- cax=_cbar_ax,
- )
- _cbar.set_label(f"Count in top-N", fontsize=13, color="black")
- _cbar.ax.tick_params(colors="black", labelsize=12)
-
- _fig.suptitle(
- f"SAR Landscape vs Top-N Threshold — {_lib_id.capitalize()} (Docking)",
- fontsize=16, fontweight="bold", color="black", y=1.0,
- )
-
- _figs.append((_lib_id, _fig))
-
- _display_items = []
- for _lib_id, _fig in _figs:
- _display_items.append(_fig)
- _display_items.append(
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=f"reagent_separation_topn_{_lib_id}.pdf",
- mimetype="application/pdf",
- label=f"Download {_lib_id.capitalize()} as PDF",
- )
- )
- mo.vstack(_display_items)
- return
-
-
-@app.cell
-def _(mo):
- mo.md(r"""
- ## SAR Landscape vs Top-N Threshold — ROCS Libraries
-
- Same analysis as the docking multi-panel plots, but for ROCS libraries.
- ROCS shows the opposite pattern: hit reagents start well-separated at narrow top-N
- (additive SAR, greedy-reachable) but separation erodes at wider top-N as new SAR
- series are needed. TACTICS advantage grows at wider top-N because greedy gets stuck
- exploiting the first SAR series while TACTICS explores second-tier reagents.
-
- One representative query per library selected to demonstrate the erosion pattern.
- """)
- return
-
-
-@app.cell
-def _(fig_to_pdf_bytes, mo, np, pl, plt, project_root):
- """Multi-top-N posterior separation grid for selected ROCS libraries."""
- from matplotlib.colors import Normalize as _Normalize4
- from matplotlib import cm as _cm4
-
- _SCORES_DIR = project_root / "data" / "scores"
- _TOP_N_VALUES = [25, 50, 100, 150, 200, 250]
-
- _libraries = [
- {
- "lib_id": "rxn206",
- "query": "query_066",
- "n_comp": 2,
- "components": ["Component 0 (980)", "Component 1 (993)"],
- "label": "rxn206 — 2-comp (980 × 993)",
- },
- {
- "lib_id": "rxn101",
- "query": "query_042",
- "n_comp": 2,
- "components": ["Component 0 (989)", "Component 1 (973)"],
- "label": "rxn101 — 2-comp (989 × 973)",
- },
- {
- "lib_id": "petasis",
- "query": "query_086",
- "n_comp": 3,
- "components": ["Component 0 (97)", "Component 1 (98)", "Component 2 (88)"],
- "label": "Petasis — 3-comp (97 × 98 × 88)",
- },
- {
- "lib_id": "poparov",
- "query": "query_077",
- "n_comp": 3,
- "components": ["Component 0 (100)", "Component 1 (91)", "Component 2 (100)"],
- "label": "Poparov — 3-comp (100 × 91 × 100)",
- },
- ]
-
- _cmap = _cm4.YlOrRd
- _figs = []
-
- for _lib_cfg in _libraries:
- _lib_id = _lib_cfg["lib_id"]
- _qname = _lib_cfg["query"]
- _n_comp = _lib_cfg["n_comp"]
- _comp_names = _lib_cfg["components"]
-
- _df = pl.read_parquet(_SCORES_DIR / f"{_lib_id}.parquet")
- _all_codes = _df["Name"].to_list()
- _all_scores = _df[_qname].to_numpy()
-
- # Filter NaN
- _valid = ~np.isnan(_all_scores)
- _scores = _all_scores[_valid]
- _codes = [c for c, v in zip(_all_codes, _valid) if v]
-
- def _parse(code, nc=_n_comp):
- parts = code.split("_")
- return tuple(parts) if len(parts) == nc else None
-
- _parsed = [_parse(c) for c in _codes]
- _sorted_idx = np.argsort(-_scores) # maximize for ROCS
-
- # Pre-compute per-reagent z-scored stats
- _reagent_data = []
- for _comp_idx in range(_n_comp):
- _r2i = {}
- for _i, _p in enumerate(_parsed):
- if _p is not None:
- _r2i.setdefault(_p[_comp_idx], []).append(_i)
-
- _unique = sorted(_r2i.keys())
- _means = np.array([float(np.mean(_scores[_r2i[r]])) for r in _unique])
- _stds = np.array([float(np.std(_scores[_r2i[r]])) for r in _unique])
-
- _mean_mu, _mean_sigma = np.mean(_means), np.std(_means)
- _std_mu, _std_sigma = np.mean(_stds), np.std(_stds)
- _z_means = (_means - _mean_mu) / _mean_sigma if _mean_sigma > 0 else _means * 0
- _z_stds = (_stds - _std_mu) / _std_sigma if _std_sigma > 0 else _stds * 0
-
- _reagent_data.append({
- "unique": _unique, "z_means": _z_means, "z_stds": _z_stds,
- })
-
- # Find global max count across all top-N panels
- _global_max = 0
- for _top_n in _TOP_N_VALUES:
- _top_codes = [_codes[i] for i in _sorted_idx[:_top_n]]
- _top_parsed = [_parse(c) for c in _top_codes]
- for _comp_idx in range(_n_comp):
- _tc = {}
- for _p in _top_parsed:
- if _p is not None:
- _rid = _p[_comp_idx]
- _tc[_rid] = _tc.get(_rid, 0) + 1
- if _tc:
- _global_max = max(_global_max, max(_tc.values()))
-
- _norm = _Normalize4(vmin=1, vmax=max(_global_max, 2))
-
- # Build grid
- _n_rows = len(_TOP_N_VALUES)
- _width_ratios = [1] * _n_comp + [0.05]
- _gs = plt.GridSpec(_n_rows, _n_comp + 1, width_ratios=_width_ratios, wspace=0.3, hspace=0.4)
- _fig = plt.figure(figsize=(6 * _n_comp + 1, 4 * _n_rows), facecolor="white")
-
- for _row_idx, _top_n in enumerate(_TOP_N_VALUES):
- _top_codes = [_codes[i] for i in _sorted_idx[:_top_n]]
- _top_parsed = [_parse(c) for c in _top_codes]
-
- for _comp_idx in range(_n_comp):
- _ax = _fig.add_subplot(_gs[_row_idx, _comp_idx])
- _ax.set_facecolor("white")
-
- _rd = _reagent_data[_comp_idx]
- _z_means = _rd["z_means"]
- _z_stds = _rd["z_stds"]
- _unique = _rd["unique"]
-
- _tc = {}
- for _p in _top_parsed:
- if _p is not None:
- _rid = _p[_comp_idx]
- _tc[_rid] = _tc.get(_rid, 0) + 1
-
- _counts = np.array([_tc.get(r, 0) for r in _unique])
-
- _zero_mask = _counts == 0
- _ax.scatter(
- _z_means[_zero_mask], _z_stds[_zero_mask],
- c="#CCCCCC", s=20, alpha=0.4, edgecolors="none", zorder=2,
- )
-
- _hit_mask = _counts > 0
- if _hit_mask.any():
- _ax.scatter(
- _z_means[_hit_mask], _z_stds[_hit_mask],
- c=_counts[_hit_mask], cmap=_cmap, norm=_norm,
- s=30, alpha=0.9, edgecolors="black", linewidths=0.3,
- zorder=3,
- )
-
- if _row_idx == _n_rows - 1:
- _ax.set_xlabel("Reagent Mean Score (z)", fontsize=12, color="black")
- if _comp_idx == 0:
- _ax.set_ylabel("Reagent Stdev (z)", fontsize=12, color="black")
-
- if _row_idx == 0:
- _ax.set_title(_comp_names[_comp_idx],
- fontsize=13, fontweight="bold", color="black")
-
- if _comp_idx == _n_comp - 1:
- _n_hit = int((_counts > 0).sum())
- _ax.annotate(
- f"Top-{_top_n}\n({_n_hit} reagents)",
- xy=(1.08, 0.5), xycoords="axes fraction",
- fontsize=12, fontweight="bold", color="black",
- rotation=-90, va="center", ha="left",
- )
-
- _ax.tick_params(colors="black", labelsize=12)
- _ax.grid(alpha=0.2, color="grey", zorder=1)
- for _spine in _ax.spines.values():
- _spine.set_color("black")
-
- # Shared colorbar
- _cbar_ax = _fig.add_subplot(_gs[:, _n_comp])
- _cbar = _fig.colorbar(
- _cm4.ScalarMappable(norm=_norm, cmap=_cmap),
- cax=_cbar_ax,
- )
- _cbar.set_label("Count in top-N", fontsize=13, color="black")
- _cbar.ax.tick_params(colors="black", labelsize=12)
-
- _fig.suptitle(
- f"SAR Landscape vs Top-N Threshold — {_lib_cfg['label']}\n"
- f"ROCS {_qname}",
- fontsize=16, fontweight="bold", color="black", y=1.0,
- )
-
- _figs.append((_lib_id, _fig))
-
- _display_items = []
- for _lib_id, _fig in _figs:
- _display_items.append(_fig)
- _display_items.append(
- mo.download(
- data=fig_to_pdf_bytes(_fig),
- filename=f"reagent_separation_topn_rocs_{_lib_id}.pdf",
- mimetype="application/pdf",
- label=f"Download {_lib_id} as PDF",
- )
- )
- mo.vstack(_display_items)
- return
-
-
-if __name__ == "__main__":
- app.run()
diff --git a/tutorials/manuscript_plots_ROCS.py b/tutorials/manuscript_plots_ROCS.py
new file mode 100644
index 0000000..4a99be7
--- /dev/null
+++ b/tutorials/manuscript_plots_ROCS.py
@@ -0,0 +1,1934 @@
+"""
+TACTICS Manuscript Plots — ROCS Libraries
+
+Loads the canonical 28-library recovery summary, filters to ROCS libraries
+(20 similarity libraries × 109 queries × 20 replicates), and renders all
+ROCS-related manuscript figures: aggregate Top-N recovery, Zhao Fig 5
+extensions, Tukey HSD pairwise CIs, per-library breakdowns, and the ROCS
+budget-sensitivity panel. The parallel docking figures live in
+`tutorials/manuscript_plots.py`.
+
+Run as app: marimo run tutorials/manuscript_plots_ROCS.py
+Edit mode: marimo edit tutorials/manuscript_plots_ROCS.py
+"""
+
+import marimo
+
+__generated_with = "0.21.1"
+app = marimo.App(
+ width="full",
+ app_title="TACTICS Manuscript Plots — ROCS Libraries",
+)
+
+
+@app.cell
+def _():
+ """Imports."""
+ import io
+ import marimo as mo
+ import sys
+ from pathlib import Path
+
+ import matplotlib.pyplot as plt
+ from matplotlib.lines import Line2D
+ from matplotlib.patches import Patch
+ import numpy as np
+ import polars as pl
+ from scipy import stats
+
+ def fig_to_pdf_bytes(fig):
+ """Convert a matplotlib figure to PDF bytes in memory."""
+ buf = io.BytesIO()
+ fig.savefig(buf, format="pdf", bbox_inches="tight", dpi=300)
+ return buf.getvalue()
+
+ try:
+ project_root = Path(__file__).parent.parent.resolve()
+ except NameError:
+ project_root = Path.cwd()
+
+ sys.path.insert(0, str(project_root))
+ sys.path.insert(0, str(project_root / "src"))
+ return Line2D, fig_to_pdf_bytes, mo, np, pl, plt, project_root, stats
+
+
+@app.cell
+def _(mo):
+ """Top-N slider and method selector for filtering pre-computed recovery data."""
+ topn_slider = mo.ui.slider(
+ start=25, stop=250, step=25, value=100,
+ label="Top-N Recovery Threshold",
+ )
+ _all_methods = [
+ "TACTICS Enhanced-TT-TS (GMIC)",
+ "TACTICS Enhanced-RWS (GMIC)",
+ "Legacy Enhanced-RWS",
+ "Legacy Standard-Greedy",
+ ]
+ method_selector = mo.ui.multiselect(
+ options=_all_methods,
+ value=_all_methods,
+ label="Methods to include",
+ )
+ mo.vstack([topn_slider, method_selector])
+ return method_selector, topn_slider
+
+
+@app.cell
+def _(method_selector, mo, pl, project_root, topn_slider):
+ """Load pre-computed recovery and statistical analysis results."""
+ analysis_dir = project_root / "outputs" / "full_benchmark_ema_cats" / "analysis"
+
+ TOP_N = topn_slider.value
+ selected_methods = method_selector.value
+
+ # --- Recovery summary (all libraries × top-N thresholds ≤ 250) ---
+ # Top-N range capped at 250 manuscript-wide (per 2026-05-27 scope decision);
+ # wider top-N values exist in the parquet but are not used in any manuscript figure.
+ recovery_all_raw = (
+ pl.read_parquet(analysis_dir / "recovery_summary.parquet")
+ .filter(pl.col("top_n") <= 250)
+ )
+ recovery_all = recovery_all_raw.filter(pl.col("method").is_in(selected_methods))
+
+ # Filter to selected top-N threshold
+ df = recovery_all.filter(pl.col("top_n") == TOP_N)
+
+ # --- Statistical test results (filter to selected method pairs) ---
+ tukey_df = pl.read_parquet(analysis_dir / "tukey_hsd_results.parquet")
+ tukey_at_n = tukey_df.filter(
+ (pl.col("top_n") == TOP_N)
+ & pl.col("group1").is_in(selected_methods)
+ & pl.col("group2").is_in(selected_methods)
+ )
+
+ anova_df = pl.read_parquet(analysis_dir / "anova_results.parquet")
+ anova_at_n = anova_df.filter(pl.col("top_n") == TOP_N)
+
+ n_trials = len(df)
+ n_libraries = df["library_id"].n_unique()
+ libraries = sorted(df["library_id"].unique().to_list())
+ methods = sorted(df["method"].unique().to_list())
+
+ mo.md(
+ f"""
+ ## Data Loaded (Pre-Computed Post-Hoc Analysis)
+
+ - **Source**: `{analysis_dir}`
+ - **Top-N threshold**: {TOP_N} (use slider to change)
+ - **Available thresholds**: {sorted(recovery_all_raw['top_n'].unique().to_list())}
+ - **Selected methods** ({len(selected_methods)}/4): {', '.join(methods)}
+ - **Total trials at top-{TOP_N}**: {n_trials:,}
+ - **Libraries**: {n_libraries} ({', '.join(libraries)})
+ - **Replicates**: {df['replicate'].n_unique()}
+ - **Tukey HSD comparisons**: {len(tukey_at_n):,} ({tukey_at_n.filter(pl.col('reject')).height} significant)
+ - **ANOVA tests**: {len(anova_at_n)} ({anova_at_n.filter(pl.col('significance') != 'ns').height} significant)
+ """
+ )
+ return TOP_N, df, recovery_all, tukey_at_n
+
+
+@app.cell
+def _():
+ """Statistical-test selection rule and star color convention.
+
+ Rule (by scoring type):
+ - Docking libraries → Welch's t-test + Bonferroni-4 on the planned
+ TACTICS-vs-Legacy pairs. Docking benchmarks here use one query
+ per library × 20 replicates, so within-method variance (including
+ bimodal-failure modes) shows up directly. Tukey HSD's pooled-MSE
+ homoscedasticity assumption is violated whenever any single method
+ is bimodal, so Welch is the appropriate confirmatory test.
+ - ROCS libraries → Tukey HSD per Ash et al. ROCS uses 109 queries
+ × 20 replicates, and cross-query variance dominates the per-method
+ variance, so the variances are approximately homogeneous across
+ methods and Tukey's pooled-MSE assumption holds.
+
+ Docking libraries: thrombin, adenine, adenine_hybrid, amide, quinazoline.
+ """
+ DOCKING_LIBS = {
+ "thrombin", "adenine", "adenine_hybrid", "amide", "quinazoline",
+ "thrombin_hybrid", "amide_hybrid", "quinazoline_hybrid",
+ }
+ TUKEY_STAR_COLOR = "black"
+ WELCH_STAR_COLOR = "#4477AA"
+ return (TUKEY_STAR_COLOR,)
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 1 — Overall ROCS Recovery
+
+ Aggregate recovery across the 20 ROCS similarity libraries (109 queries × 20
+ replicates per library). Methods compared: TACTICS TT-TS, TACTICS RWS,
+ Legacy RWS, Legacy Standard-Greedy.
+ """)
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Sensitivity Analysis - Top N compounds
+ In this section we explore the roubustness of CATS as the interval for the top N compounds increases, if CATS is interpreting the SAR landscape correctly and systematically able to uncover mid tier hits.
+ """)
+ return
+
+
+@app.cell
+def _(mo, recovery_all):
+ """Summary of recovery data available for sensitivity analysis."""
+ _n_rows = len(recovery_all)
+ _n_libs = recovery_all["library_id"].n_unique()
+ _thresholds = sorted(recovery_all["top_n"].unique().to_list())
+ _methods = sorted(recovery_all["method"].unique().to_list())
+
+ mo.vstack([
+ mo.md(
+ f"""
+ ## Top-N Sensitivity Data (from Post-Hoc Analysis)
+
+ - **Rows**: {_n_rows:,}
+ - **Libraries**: {_n_libs}
+ - **Top-N thresholds**: {_thresholds}
+ - **Methods**: {', '.join(_methods)}
+ """
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, pl, plt, recovery_all):
+ """Overall % recovery vs top-N by component count (ROCS libraries, 10 thresholds)."""
+
+ _DOCK_LIBS = {
+ "thrombin", "adenine", "adenine_hybrid", "amide", "quinazoline",
+ "thrombin_hybrid", "amide_hybrid", "quinazoline_hybrid",
+ }
+ _ROCS_NCOMP = {
+ "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
+ "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
+ "amide-suzuki": 3, "betti": 3, "dobener": 3,
+ "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
+ "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
+ }
+
+ _rocs_all = (
+ recovery_all
+ .filter(~pl.col("library_id").is_in(_DOCK_LIBS))
+ .filter(pl.col("library_id").is_in(set(_ROCS_NCOMP.keys())))
+ .filter(pl.col("top_n") <= 250)
+ .with_columns(
+ pl.col("library_id").replace(_ROCS_NCOMP).cast(pl.Int64).alias("n_components")
+ )
+ )
+
+ mo.stop(len(_rocs_all) == 0, mo.md("No ROCS library data available."))
+
+ _method_colors = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "#E45756",
+ "TACTICS Enhanced-RWS (GMIC)": "#4C78A8",
+ "Legacy Standard-Greedy": "#76B7B2",
+ "Legacy Enhanced-RWS": "#B07AA1",
+ }
+ _method_markers = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "D",
+ "TACTICS Enhanced-RWS (GMIC)": "o",
+ "Legacy Standard-Greedy": "s",
+ "Legacy Enhanced-RWS": "^",
+ }
+ _method_order = [
+ "TACTICS Enhanced-TT-TS (GMIC)",
+ "TACTICS Enhanced-RWS (GMIC)",
+ "Legacy Standard-Greedy",
+ "Legacy Enhanced-RWS",
+ ]
+
+ _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white",
+ sharey=True)
+
+ for _panel_idx, _nc in enumerate([2, 3]):
+ _ax = _axes[_panel_idx]
+ _ax.set_facecolor("white")
+
+ _comp_data = _rocs_all.filter(pl.col("n_components") == _nc)
+
+ _agg = (
+ _comp_data.group_by(["method", "top_n"])
+ .agg([
+ pl.col("recovery_frac").mean().alias("mean_frac"),
+ pl.col("recovery_frac").std().alias("std_frac"),
+ pl.col("recovery_frac").count().alias("n"),
+ ])
+ .with_columns(
+ (pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac")
+ )
+ .sort(["method", "top_n"])
+ )
+
+ for _method in _method_order:
+ _mdata = _agg.filter(pl.col("method") == _method).sort("top_n")
+ if len(_mdata) == 0:
+ continue
+ _x = _mdata["top_n"].to_numpy()
+ _y = _mdata["mean_frac"].to_numpy() * 100
+ _se = _mdata["se_frac"].to_numpy() * 100
+ _color = _method_colors.get(_method, "#999999")
+ _marker = _method_markers.get(_method, "o")
+
+ _ax.plot(_x, _y, marker=_marker, color=_color, linewidth=2,
+ markersize=7, label=_method, zorder=3)
+ _ax.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se,
+ alpha=0.15, color=_color, zorder=2)
+
+ _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax.set_title(f"{_nc}-Component Libraries", fontsize=15,
+ fontweight="bold", color="black")
+ _ax.set_xticks(sorted(_agg["top_n"].unique().to_list()))
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(axis="both", alpha=0.3, color="grey", zorder=1)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+
+ _axes[0].set_ylabel("% Recovery", fontsize=14, color="black")
+
+ # Legend outside the figure below the panels
+ _handles, _labels = _axes[0].get_legend_handles_labels()
+ _fig.legend(
+ _handles, _labels,
+ loc="lower center", ncol=3, fontsize=12,
+ facecolor="white", edgecolor="black", labelcolor="black",
+ bbox_to_anchor=(0.5, -0.08),
+ )
+
+ _fig.suptitle("Mean % Recovery vs Top-N Threshold (ROCS Libraries)",
+ fontsize=15, fontweight="bold", color="black", y=1.01)
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="overall_recovery_vs_topn.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, np, pl, plt, recovery_all, stats):
+ """ROCS aggregate Δ recovery bars — TACTICS TT-TS vs each Legacy method, split by component count.
+
+ For each (library × top-N), compute mean recovery per method (across query × replicate),
+ then Δ_lib = mean(TT-TS_lib) − mean(Legacy_lib). Aggregate across the 10 libraries in
+ each component split: bar height = mean Δ, error bar = SE across libraries
+ (std / sqrt(10)). Significance: scipy.stats.ttest_1samp on the per-library Δ vector
+ (H0: Δ=0); *** p<0.001, ** p<0.01, * p<0.05, else ns. A paired across-library test
+ (one-sample t on Δ) is the natural aggregate analogue of per-library Welch tests and
+ avoids the pooled-variance contamination issue.
+
+ On ROCS, TACTICS TT-TS and TACTICS RWS are practically equivalent at the per-library
+ level: across 200 (library × top-N) cells the larger TACTICS method beats the other
+ by at most 0.79 pp (≤ 0.2 compounds, on `poparov` at top-25). TT-TS is reported here
+ as the headline TACTICS method because it's the per-library *statistical* winner on
+ 179/200 cells; the choice has no practical consequence on ROCS, so this figure can
+ be read straightforwardly as "TACTICS vs Legacy" without method-selection caveats.
+ """
+
+ _ROCS_NCOMP_BAR = {
+ "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
+ "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
+ "amide-suzuki": 3, "betti": 3, "dobener": 3,
+ "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
+ "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
+ }
+ _HEADLINE_TACTICS = "TACTICS Enhanced-TT-TS (GMIC)"
+ _LEGACIES_BAR = [
+ ("Legacy Standard-Greedy", "Legacy Greedy", "#F28E2B"),
+ ("Legacy Enhanced-RWS", "Legacy RWS", "#4C78A8"),
+ ]
+ _ROCS_LIBS_BAR = set(_ROCS_NCOMP_BAR.keys())
+
+ _rocs_bar_all = (
+ recovery_all
+ .filter(pl.col("library_id").is_in(_ROCS_LIBS_BAR))
+ .filter(pl.col("top_n") <= 250)
+ .filter(pl.col("method").is_in({_HEADLINE_TACTICS} | {m for m, _, _ in _LEGACIES_BAR}))
+ )
+
+ mo.stop(len(_rocs_bar_all) == 0, mo.md("No ROCS library data available."))
+
+ # Per (library × method × top_n) mean recovery_frac (collapse query × replicate)
+ _per_lib = (
+ _rocs_bar_all.group_by(["library_id", "method", "top_n"])
+ .agg(pl.col("recovery_frac").mean().alias("mean_frac"))
+ )
+
+ _topn_vals_bar = sorted(_rocs_bar_all["top_n"].unique().to_list())
+
+ def _sig_stars(_p):
+ if _p is None or np.isnan(_p):
+ return ""
+ if _p < 0.001:
+ return "***"
+ if _p < 0.01:
+ return "**"
+ if _p < 0.05:
+ return "*"
+ return ""
+
+ _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white", sharey=True)
+ _bar_width = 0.35
+
+ _panel_diffs = {2: [], 3: []}
+
+ for _panel_idx, _nc in enumerate([2, 3]):
+ _ax = _axes[_panel_idx]
+ _ax.set_facecolor("white")
+
+ _libs_nc = sorted([_lib for _lib, _v in _ROCS_NCOMP_BAR.items() if _v == _nc])
+ _x_pos = np.arange(len(_topn_vals_bar))
+
+ for _gi, (_leg_meth, _leg_lbl, _bar_color) in enumerate(_LEGACIES_BAR):
+ _mean_diffs = []
+ _se_diffs = []
+ _sigs = []
+
+ for _tn in _topn_vals_bar:
+ _per_lib_deltas = []
+ for _lib in _libs_nc:
+ _tt = _per_lib.filter(
+ (pl.col("library_id") == _lib)
+ & (pl.col("method") == _HEADLINE_TACTICS)
+ & (pl.col("top_n") == _tn)
+ )["mean_frac"]
+ _leg = _per_lib.filter(
+ (pl.col("library_id") == _lib)
+ & (pl.col("method") == _leg_meth)
+ & (pl.col("top_n") == _tn)
+ )["mean_frac"]
+ if len(_tt) == 0 or len(_leg) == 0:
+ continue
+ _delta = (float(_tt[0]) - float(_leg[0])) * 100.0
+ _per_lib_deltas.append(_delta)
+
+ _per_lib_deltas = np.array(_per_lib_deltas, dtype=float)
+ if len(_per_lib_deltas) == 0:
+ _mean_diffs.append(0.0)
+ _se_diffs.append(0.0)
+ _sigs.append("")
+ continue
+ _n_libs = len(_per_lib_deltas)
+ _mean_diffs.append(float(_per_lib_deltas.mean()))
+ _se_diffs.append(
+ float(_per_lib_deltas.std(ddof=1) / np.sqrt(_n_libs))
+ if _n_libs > 1 else 0.0
+ )
+ if _n_libs >= 2 and _per_lib_deltas.std(ddof=1) > 0:
+ _tres = stats.ttest_1samp(_per_lib_deltas, popmean=0.0)
+ _sigs.append(_sig_stars(_tres.pvalue))
+ else:
+ _sigs.append("")
+
+ _mean_diffs = np.array(_mean_diffs)
+ _se_diffs = np.array(_se_diffs)
+ _offset = (_gi - 0.5) * _bar_width
+ _ax.bar(
+ _x_pos + _offset, _mean_diffs, width=_bar_width,
+ color=_bar_color, edgecolor="black", linewidth=0.5,
+ yerr=_se_diffs, error_kw={"ecolor": "black", "capsize": 3, "elinewidth": 1},
+ zorder=3, label=f"vs {_leg_lbl}",
+ )
+
+ _dmax = max(abs(_mean_diffs).max() if len(_mean_diffs) else 1, 1e-6)
+ _star_pad = _dmax * 0.05
+ for _i, (_d, _se, _s) in enumerate(zip(_mean_diffs, _se_diffs, _sigs)):
+ if _s:
+ _yp = _d + _se + _star_pad if _d >= 0 else _d - _se - _star_pad
+ _ax.text(
+ _x_pos[_i] + _offset, _yp, _s, ha="center",
+ va="bottom" if _d >= 0 else "top",
+ fontsize=10, fontweight="bold", color="black",
+ )
+
+ _panel_diffs[_nc].extend(list(_mean_diffs + _se_diffs) + list(_mean_diffs - _se_diffs))
+
+ _ax.axhline(0, color="black", linewidth=0.8, zorder=2)
+ _ax.set_xticks(_x_pos)
+ _ax.set_xticklabels([str(t) for t in _topn_vals_bar], fontsize=12, color="black")
+ _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax.set_title(f"{_nc}-Component Libraries", fontsize=15,
+ fontweight="bold", color="black")
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(axis="y", alpha=0.3, color="grey", zorder=1)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+
+ _all_extrema = np.array(_panel_diffs[2] + _panel_diffs[3], dtype=float)
+ if len(_all_extrema) > 0:
+ _ymin = float(_all_extrema.min())
+ _ymax = float(_all_extrema.max())
+ _pad = max(abs(_ymax - _ymin) * 0.20, 1.0)
+ for _ax in _axes:
+ _ax.set_ylim(_ymin - _pad, _ymax + _pad)
+
+ _axes[0].set_ylabel(
+ "Δ % Recovery (TACTICS TT-TS − Legacy)", fontsize=14, color="black"
+ )
+
+ _handles, _labels = _axes[0].get_legend_handles_labels()
+ _fig.legend(
+ _handles, _labels,
+ loc="lower center", ncol=2, fontsize=12,
+ facecolor="white", edgecolor="black", labelcolor="black",
+ bbox_to_anchor=(0.5, -0.08),
+ )
+
+ _fig.suptitle(
+ "Δ Recovery — TACTICS TT-TS vs Legacy Methods (ROCS Libraries)",
+ fontsize=15, fontweight="bold", color="black", y=1.01,
+ )
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.md(
+ "**Headline method.** TACTICS TT-TS is reported here as the headline "
+ "TACTICS method. The two TACTICS strategies (TT-TS, RWS) are "
+ "practically equivalent on ROCS at the per-library level — the "
+ "largest gap on any (library × top-N) cell is 0.79 pp ≈ 0.2 "
+ "compounds — so this figure is essentially \"TACTICS vs Legacy\" "
+ "without a method-selection caveat."
+ ),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="rocs_tactics_vs_legacy_bars.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, np, pl, plt, recovery_all):
+ """Chained-decomposition stacked bars — split each TACTICS method's gain over
+ Legacy Greedy into its two additive components:
+
+ Δ(TACTICS, L-Grdy) = Δ(L-RWS, L-Grdy) + Δ(TACTICS, L-RWS)
+ └─ "Zhao 2025 lift" ┘ └ "TACTICS algorithmic lift" ┘
+
+ The two segments stack at the zero line. When Zhao lift is negative
+ (Legacy RWS underperforms Legacy Greedy — documented on 2-comp ROCS at
+ wider top-N), the Zhao segment extends below 0 and the TACTICS segment
+ starts from that negative point and stacks upward; the bar's top edge
+ is always Δ(TACTICS, L-Grdy). Two grouped bars per top-N (TT-TS, T-RWS);
+ panels split by 2-comp vs 3-comp ROCS.
+ """
+ _ROCS_NCOMP = {
+ "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
+ "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
+ "amide-suzuki": 3, "betti": 3, "dobener": 3,
+ "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
+ "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
+ }
+ _METHODS = {
+ "TT-TS": "TACTICS Enhanced-TT-TS (GMIC)",
+ "T-RWS": "TACTICS Enhanced-RWS (GMIC)",
+ "L-RWS": "Legacy Enhanced-RWS",
+ "L-Grdy": "Legacy Standard-Greedy",
+ }
+ _ZHAO_COLOR = "#B07AA1" # Zhao 2025 lift — purple, matches Legacy RWS palette
+ _TACTICS_COLORS = {"TT-TS": "#E45756", "T-RWS": "#4C78A8"}
+
+ _rocs = (
+ recovery_all
+ .filter(pl.col("library_id").is_in(set(_ROCS_NCOMP.keys())))
+ .filter(pl.col("top_n") <= 250)
+ .filter(pl.col("method").is_in(set(_METHODS.values())))
+ )
+ mo.stop(len(_rocs) == 0, mo.md("No ROCS data available."))
+
+ # Per-library mean recovery (collapse query × replicate); then per-method dict per lib.
+ _per_lib = (
+ _rocs.group_by(["library_id", "method", "top_n"])
+ .agg(pl.col("recovery_frac").mean().alias("mean_frac"))
+ )
+
+ _topn_vals = sorted(_rocs["top_n"].unique().to_list())
+
+ _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white", sharey=True)
+ _bar_w = 0.38
+
+ for _panel_idx, _nc in enumerate([2, 3]):
+ _ax = _axes[_panel_idx]
+ _ax.set_facecolor("white")
+ _libs_nc = sorted([_l for _l, _v in _ROCS_NCOMP.items() if _v == _nc])
+ _x_pos = np.arange(len(_topn_vals))
+
+ for _gi, _tac_short in enumerate(["TT-TS", "T-RWS"]):
+ _tac_method = _METHODS[_tac_short]
+ _zhao_means = []
+ _tac_means = []
+ for _tn in _topn_vals:
+ _zhao_per_lib = []
+ _tac_per_lib = []
+ for _lib in _libs_nc:
+ _tn_data = _per_lib.filter(
+ (pl.col("library_id") == _lib) & (pl.col("top_n") == _tn)
+ )
+ _by_method = {
+ _r["method"]: float(_r["mean_frac"])
+ for _r in _tn_data.iter_rows(named=True)
+ }
+ if not all(_m in _by_method for _m in _METHODS.values()):
+ continue
+ _zhao_per_lib.append(
+ (_by_method[_METHODS["L-RWS"]] - _by_method[_METHODS["L-Grdy"]]) * 100
+ )
+ _tac_per_lib.append(
+ (_by_method[_tac_method] - _by_method[_METHODS["L-RWS"]]) * 100
+ )
+ _zhao_means.append(np.mean(_zhao_per_lib) if _zhao_per_lib else 0.0)
+ _tac_means.append(np.mean(_tac_per_lib) if _tac_per_lib else 0.0)
+
+ _zhao_means = np.array(_zhao_means)
+ _tac_means = np.array(_tac_means)
+ _offset = (_gi - 0.5) * _bar_w
+
+ # Zhao segment anchored at 0. Hatched when negative to flag the regime
+ # where Legacy RWS underperforms Legacy Greedy.
+ _ax.bar(
+ _x_pos + _offset, _zhao_means, width=_bar_w,
+ color=_ZHAO_COLOR, edgecolor="black", linewidth=0.5,
+ hatch=["////" if _z < 0 else "" for _z in _zhao_means],
+ label="Zhao 2025 lift (L-RWS − L-Grdy)" if _gi == 0 else None,
+ zorder=3,
+ )
+ # TACTICS segment stacked on top of Zhao segment.
+ _ax.bar(
+ _x_pos + _offset, _tac_means, width=_bar_w,
+ bottom=_zhao_means,
+ color=_TACTICS_COLORS[_tac_short], edgecolor="black", linewidth=0.5,
+ label=f"TACTICS lift ({_tac_short} − L-RWS)",
+ zorder=3,
+ )
+
+ # Net total marker (top edge of full stack = Δ(TACTICS, L-Grdy))
+ _net = _zhao_means + _tac_means
+ for _xi, _net_yi in zip(_x_pos + _offset, _net):
+ _ax.plot([_xi - _bar_w/2, _xi + _bar_w/2], [_net_yi, _net_yi],
+ color="black", linewidth=1.2, zorder=5)
+
+ _ax.axhline(0, color="black", linewidth=1.0, zorder=2)
+ _ax.set_xticks(_x_pos)
+ _ax.set_xticklabels([str(t) for t in _topn_vals], fontsize=12, color="black")
+ _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax.set_title(f"{_nc}-Component ROCS Libraries", fontsize=15,
+ fontweight="bold", color="black")
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(axis="y", alpha=0.3, color="grey", zorder=1)
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+
+ _axes[0].set_ylabel(
+ "Δ % Recovery vs Legacy Greedy (chained decomposition)",
+ fontsize=14, color="black",
+ )
+
+ _handles, _labels = _axes[0].get_legend_handles_labels()
+ _fig.legend(
+ _handles, _labels, loc="lower center", ncol=3, fontsize=11,
+ facecolor="white", edgecolor="black", labelcolor="black",
+ bbox_to_anchor=(0.5, -0.10),
+ )
+ _fig.suptitle(
+ "Chained-Decomposition Stack — Zhao 2025 Lift + TACTICS Algorithmic Lift (ROCS)",
+ fontsize=14, fontweight="bold", color="black", y=1.01,
+ )
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.md(
+ "**Reading guide.** Black tick at the top of each bar = net "
+ "Δ(TACTICS, Legacy Greedy). Lower (purple) segment = Δ(Legacy RWS, "
+ "Legacy Greedy), i.e. the lift Zhao 2025 already provided — hatched "
+ "when it goes negative (Legacy RWS hurts vs Greedy, expected at "
+ "wider top-N on 2-comp ROCS per STATUS line 24). Upper (colored) "
+ "segment = Δ(TACTICS, Legacy RWS), the algorithmic lift this work "
+ "adds on top. The total visual height of the colored region equals "
+ "Δ(TACTICS, Legacy RWS), independent of how Zhao 2025 compares to "
+ "Greedy — so the TACTICS contribution stays legible even where the "
+ "Zhao baseline is negative."
+ ),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="rocs_chained_decomposition_stack.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 2 — Diversity vs Recovery (Zhao Fig 5 extended)
+
+ Per-library recovery rate plotted against top-100 chemical diversity (BMS metric,
+ following Zhao et al. 2025 Fig 5). Both Legacy and TACTICS methods decline with
+ diversity; TACTICS' data-driven reallocation produces gentler decay.
+ """)
+ return
+
+
+@app.cell
+def _(mo, np, pl, plt, project_root, recovery_all, stats):
+ """Zhao 2025 Fig 5 — data prep + helper functions.
+
+ Computes Zhao's diversity metric (total unique reagents in top-100, averaged
+ across queries per library) for the 20 ROCS libraries (10 two-component, 10
+ three-component) and joins it with our per-library top-100 recovery. The
+ result is one (mean_diversity, mean_recovery) point per (library, method)
+ pair — exactly Zhao's per-library aggregation.
+
+ Defines two plotting helpers that share `zhao_data`:
+ - `zhao_pairwise_plot`: 2-panel scatter (2-comp / 3-comp) for any
+ (challenger, baseline) pair, with per-library vertical arrows showing the
+ within-library Δ recovery (baseline → challenger).
+ - `zhao_delta_plot`: 2-panel companion that plots Δ = challenger − baseline
+ directly against diversity, with linear fit + slope + R².
+ """
+ _ROCS_NCOMP = {
+ "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
+ "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
+ "amide-suzuki": 3, "betti": 3, "dobener": 3,
+ "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
+ "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
+ }
+ _SCORES_DIR = project_root / "data" / "scores"
+
+ _diversity_records = []
+ for _lib_id, _n_comp in _ROCS_NCOMP.items():
+ _path = _SCORES_DIR / f"{_lib_id}.parquet"
+ if not _path.exists():
+ continue
+ _df = pl.read_parquet(_path)
+ _query_cols = sorted(c for c in _df.columns if c.startswith("query_"))
+ for _qcol in _query_cols:
+ _sub = _df.select(["Name", _qcol]).filter(pl.col(_qcol).is_not_nan())
+ _top100 = _sub.sort(_qcol, descending=True).head(100)
+ _names = _top100["Name"].to_list()
+ _comp_sets = [set() for _ in range(_n_comp)]
+ for _nm in _names:
+ _parts = _nm.split("_")
+ if len(_parts) != _n_comp:
+ continue
+ for _i, _r in enumerate(_parts):
+ _comp_sets[_i].add(_r)
+ _diversity_records.append({
+ "library_id": _lib_id, "query_id": _qcol,
+ "n_components": _n_comp, "diversity": sum(len(_s) for _s in _comp_sets),
+ })
+ _diversity_df = pl.DataFrame(_diversity_records)
+ mo.stop(len(_diversity_df) == 0, mo.md("No ROCS diversity data could be computed."))
+
+ _rec_per_query = (
+ recovery_all
+ .filter(pl.col("library_id").is_in(set(_ROCS_NCOMP.keys())))
+ .filter(pl.col("top_n") == 100)
+ .group_by(["library_id", "query_id", "method"])
+ .agg(pl.col("recovery_frac").mean().alias("rec_frac"))
+ )
+ _joined = _rec_per_query.join(_diversity_df, on=["library_id", "query_id"], how="inner")
+ zhao_data = (
+ _joined.group_by(["library_id", "method", "n_components"])
+ .agg([
+ pl.col("rec_frac").mean().alias("mean_rec"),
+ pl.col("diversity").mean().alias("mean_diversity"),
+ ])
+ )
+
+ mo.md(
+ f"**Zhao Fig 5 data prepared.** "
+ f"{zhao_data.filter(pl.col('n_components')==2).group_by('method').len()['len'].max()} "
+ f"libraries per method in 2-comp panel, "
+ f"{zhao_data.filter(pl.col('n_components')==3).group_by('method').len()['len'].max()} "
+ f"in 3-comp panel."
+ )
+
+ def zhao_pairwise_plot(
+ challenger, baseline,
+ challenger_color, baseline_color,
+ challenger_label, baseline_label,
+ title,
+ ):
+ """Pairwise scatter + linear-fit, two panels (2-comp left, 3-comp right).
+
+ Each panel: 10 ROCS libraries as data points per method. X-axis = mean
+ Zhao diversity (unique reagents in top-100, averaged across queries),
+ Y-axis = mean top-100 recovery. Per-library vertical arrows connect
+ baseline → challenger to make the within-library Δ visible (diversity
+ is a library-level property computed from the ground-truth top-100, so
+ the same x for both methods). Slopes and R² shown in legend.
+ """
+ _fig, _axes = plt.subplots(1, 2, figsize=(18, 8), facecolor="white", sharey=False)
+ _series = [
+ (challenger, challenger_color, challenger_label, "o"),
+ (baseline, baseline_color, baseline_label, "o"),
+ ]
+ for _panel_idx, _nc in enumerate([2, 3]):
+ _ax = _axes[_panel_idx]
+ _ax.set_facecolor("white")
+ _panel = zhao_data.filter(pl.col("n_components") == _nc)
+ _n_libs = _panel.filter(pl.col("method") == challenger).height
+
+ _ch_libs = _panel.filter(pl.col("method") == challenger).sort("library_id")
+ _bs_libs = _panel.filter(pl.col("method") == baseline).sort("library_id")
+ _paired = _ch_libs.join(_bs_libs, on=["library_id"], suffix="_bs")
+ for _row in _paired.iter_rows(named=True):
+ _x_lib = _row["mean_diversity"]
+ _y_bs = _row["mean_rec_bs"] * 100
+ _y_ch = _row["mean_rec"] * 100
+ _ax.annotate(
+ "", xy=(_x_lib, _y_ch), xytext=(_x_lib, _y_bs),
+ arrowprops=dict(arrowstyle="-|>", color="#333333",
+ alpha=0.65, lw=1.4,
+ shrinkA=5, shrinkB=5,
+ mutation_scale=20),
+ zorder=2.5,
+ )
+
+ for _method, _color, _short, _marker in _series:
+ _md = _panel.filter(pl.col("method") == _method).sort("library_id")
+ if len(_md) < 2:
+ continue
+ _x = _md["mean_diversity"].to_numpy()
+ _y = _md["mean_rec"].to_numpy() * 100
+ _slope, _intercept, _r, _p, _se = stats.linregress(_x, _y)
+ _x_fit = np.linspace(_x.min(), _x.max(), 50)
+ _y_fit = _slope * _x_fit + _intercept
+ _ax.scatter(
+ _x, _y, color=_color, marker=_marker, s=110,
+ alpha=0.85, edgecolor="black", linewidth=0.7, zorder=3,
+ label=f"{_short} (n={len(_x)}, slope={_slope:+.3f}, R²={_r**2:.2f})",
+ )
+ _ax.plot(_x_fit, _y_fit, color=_color, linewidth=2.5, alpha=0.7, zorder=2)
+ _ax.set_xlabel(
+ "Total unique reagents in top-100",
+ fontsize=13, color="black",
+ )
+ _ax.set_ylabel("Mean Top-100 Recovery (%)", fontsize=13, color="black")
+ _ax.set_title(
+ f"{_nc}-Component ROCS Libraries (n={_n_libs})",
+ fontsize=14, fontweight="bold", color="black",
+ )
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(alpha=0.3, color="grey", zorder=1)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+ _ax.legend(
+ loc="upper center", bbox_to_anchor=(0.5, -0.14),
+ ncol=1, fontsize=11, facecolor="white",
+ edgecolor="black", labelcolor="black", framealpha=0.95,
+ )
+ _fig.suptitle(title, fontsize=14, fontweight="bold", color="black", y=1.02)
+ _fig.tight_layout(rect=[0, 0.08, 1, 0.96])
+ return _fig
+
+ def zhao_delta_plot(
+ challenger, baseline,
+ challenger_color,
+ challenger_label, baseline_label,
+ title,
+ ):
+ """Δ recovery vs diversity, two panels (2-comp left, 3-comp right).
+
+ For each library, plots Δ = challenger recovery − baseline recovery
+ (percentage points) against ground-truth top-100 diversity. Linear fit,
+ slope, R², and zero-Δ reference line shown. Positive Δ = challenger
+ wins on that library. Companion to `zhao_pairwise_plot`: makes the
+ within-library Δ the primary visual quantity rather than a paired
+ scatter.
+ """
+ _fig, _axes = plt.subplots(1, 2, figsize=(18, 8), facecolor="white", sharey=False)
+ for _panel_idx, _nc in enumerate([2, 3]):
+ _ax = _axes[_panel_idx]
+ _ax.set_facecolor("white")
+ _panel = zhao_data.filter(pl.col("n_components") == _nc)
+ _ch = _panel.filter(pl.col("method") == challenger).sort("library_id")
+ _bs = _panel.filter(pl.col("method") == baseline).sort("library_id")
+ _paired = _ch.join(_bs, on=["library_id"], suffix="_bs")
+ if _paired.height < 2:
+ continue
+ _x = _paired["mean_diversity"].to_numpy()
+ _y = (_paired["mean_rec"].to_numpy() - _paired["mean_rec_bs"].to_numpy()) * 100
+ _slope, _intercept, _r, _p, _se = stats.linregress(_x, _y)
+ _x_fit = np.linspace(_x.min(), _x.max(), 50)
+ _y_fit = _slope * _x_fit + _intercept
+ _ax.axhline(0, color="black", linewidth=1, linestyle="--", alpha=0.5, zorder=1)
+ _ax.scatter(
+ _x, _y, color=challenger_color, marker="D", s=110,
+ alpha=0.85, edgecolor="black", linewidth=0.7, zorder=3,
+ label=f"Δ (n={len(_x)}, slope={_slope:+.3f}, R²={_r**2:.2f})",
+ )
+ _ax.plot(_x_fit, _y_fit, color=challenger_color, linewidth=2.5, alpha=0.7, zorder=2)
+ _ax.set_xlabel(
+ "Total unique reagents in top-100",
+ fontsize=13, color="black",
+ )
+ _ax.set_ylabel(
+ f"Δ Recovery (pp): {challenger_label} − {baseline_label}",
+ fontsize=13, color="black",
+ )
+ _ax.set_title(
+ f"{_nc}-Component ROCS Libraries (n={len(_x)})",
+ fontsize=14, fontweight="bold", color="black",
+ )
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(alpha=0.3, color="grey", zorder=1)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+ _ax.legend(
+ loc="best", fontsize=11, facecolor="white",
+ edgecolor="black", labelcolor="black", framealpha=0.95,
+ )
+ _fig.suptitle(title, fontsize=14, fontweight="bold", color="black", y=1.02)
+ _fig.tight_layout()
+ return _fig
+
+ return zhao_delta_plot, zhao_pairwise_plot
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_pairwise_plot):
+ """Zhao Fig 5 — TACTICS RWS vs Legacy RWS (Zhao 2025)."""
+ _fig = zhao_pairwise_plot(
+ challenger="TACTICS Enhanced-RWS (GMIC)",
+ baseline="Legacy Enhanced-RWS",
+ challenger_color="#4C78A8",
+ baseline_color="#B07AA1",
+ challenger_label="TACTICS RWS (GMIC)",
+ baseline_label="Legacy RWS (Zhao 2025)",
+ title="Top-100 Recovery vs Library Diversity — TACTICS RWS vs Legacy RWS (Zhao 2025)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_tactics_rws_vs_legacy_rws.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_pairwise_plot):
+ """Zhao Fig 5 — TACTICS RWS vs Legacy Greedy (Klarich 2024 TS)."""
+ _fig = zhao_pairwise_plot(
+ challenger="TACTICS Enhanced-RWS (GMIC)",
+ baseline="Legacy Standard-Greedy",
+ challenger_color="#4C78A8",
+ baseline_color="#76B7B2",
+ challenger_label="TACTICS RWS (GMIC)",
+ baseline_label="Legacy Greedy (Klarich 2024 TS)",
+ title="Top-100 Recovery vs Library Diversity — TACTICS RWS vs Legacy Greedy (Klarich TS)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_tactics_rws_vs_legacy_greedy.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_pairwise_plot):
+ """Zhao Fig 5 — TT-TS vs Legacy RWS (Zhao 2025)."""
+ _fig = zhao_pairwise_plot(
+ challenger="TACTICS Enhanced-TT-TS (GMIC)",
+ baseline="Legacy Enhanced-RWS",
+ challenger_color="#E45756",
+ baseline_color="#B07AA1",
+ challenger_label="TACTICS TT-TS (GMIC)",
+ baseline_label="Legacy RWS (Zhao 2025)",
+ title="Top-100 Recovery vs Library Diversity — TACTICS TT-TS vs Legacy RWS (Zhao 2025)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_tt_ts_vs_legacy_rws.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_pairwise_plot):
+ """Zhao Fig 5 — TT-TS vs Legacy Greedy (Klarich 2024 TS)."""
+ _fig = zhao_pairwise_plot(
+ challenger="TACTICS Enhanced-TT-TS (GMIC)",
+ baseline="Legacy Standard-Greedy",
+ challenger_color="#E45756",
+ baseline_color="#76B7B2",
+ challenger_label="TACTICS TT-TS (GMIC)",
+ baseline_label="Legacy Greedy (Klarich 2024 TS)",
+ title="Top-100 Recovery vs Library Diversity — TACTICS TT-TS vs Legacy Greedy (Klarich TS)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_tt_ts_vs_legacy_greedy.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_delta_plot):
+ """Zhao Fig 5 Δ — TACTICS RWS minus Legacy RWS (Zhao 2025) vs diversity."""
+ _fig = zhao_delta_plot(
+ challenger="TACTICS Enhanced-RWS (GMIC)",
+ baseline="Legacy Enhanced-RWS",
+ challenger_color="#4C78A8",
+ challenger_label="TACTICS RWS (GMIC)",
+ baseline_label="Legacy RWS (Zhao 2025)",
+ title="Δ Top-100 Recovery vs Library Diversity — TACTICS RWS − Legacy RWS (Zhao 2025)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_delta_tactics_rws_vs_legacy_rws.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_delta_plot):
+ """Zhao Fig 5 Δ — TACTICS RWS minus Legacy Greedy (Klarich TS) vs diversity."""
+ _fig = zhao_delta_plot(
+ challenger="TACTICS Enhanced-RWS (GMIC)",
+ baseline="Legacy Standard-Greedy",
+ challenger_color="#4C78A8",
+ challenger_label="TACTICS RWS (GMIC)",
+ baseline_label="Legacy Greedy (Klarich 2024 TS)",
+ title="Δ Top-100 Recovery vs Library Diversity — TACTICS RWS − Legacy Greedy (Klarich TS)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_delta_tactics_rws_vs_legacy_greedy.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_delta_plot):
+ """Zhao Fig 5 Δ — TT-TS minus Legacy RWS (Zhao 2025) vs diversity."""
+ _fig = zhao_delta_plot(
+ challenger="TACTICS Enhanced-TT-TS (GMIC)",
+ baseline="Legacy Enhanced-RWS",
+ challenger_color="#E45756",
+ challenger_label="TACTICS TT-TS (GMIC)",
+ baseline_label="Legacy RWS (Zhao 2025)",
+ title="Δ Top-100 Recovery vs Library Diversity — TACTICS TT-TS − Legacy RWS (Zhao 2025)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_delta_tt_ts_vs_legacy_rws.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, plt, zhao_delta_plot):
+ """Zhao Fig 5 Δ — TT-TS minus Legacy Greedy (Klarich TS) vs diversity."""
+ _fig = zhao_delta_plot(
+ challenger="TACTICS Enhanced-TT-TS (GMIC)",
+ baseline="Legacy Standard-Greedy",
+ challenger_color="#E45756",
+ challenger_label="TACTICS TT-TS (GMIC)",
+ baseline_label="Legacy Greedy (Klarich 2024 TS)",
+ title="Δ Top-100 Recovery vs Library Diversity — TACTICS TT-TS − Legacy Greedy (Klarich TS)",
+ )
+ mo.vstack([
+ plt.gcf(),
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename="zhao_fig5_delta_tt_ts_vs_legacy_greedy.pdf",
+ mimetype="application/pdf", label="Download as PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 3 — Tukey HSD (ROCS libraries) + Per-library ROCS plots
+
+ Tukey HSD pairwise comparisons across the 20 ROCS libraries, plus per-library
+ Top-N curves + best-TACTICS Δ bars with Tukey HSD significance stars (per the
+ scoring-type rule — ROCS satisfies repeated-measures structure required for Tukey).
+ """)
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ ## Simultaneous Confidence Interval Plots (Tukey HSD)
+
+ Per-library: select a library to view all pairwise method comparisons.
+ Overall: aggregated across ROCS libraries and docking libraries separately.
+ """)
+ return
+
+
+@app.cell
+def _(df, mo, pl):
+ """Library selector for per-library Tukey HSD CI plot (ROCS libraries only)."""
+ _DOCKING_LIBS = {
+ "thrombin", "adenine", "adenine_hybrid", "amide", "quinazoline",
+ "thrombin_hybrid", "amide_hybrid", "quinazoline_hybrid",
+ }
+ _libraries = sorted(
+ df.filter(~pl.col("library_id").is_in(_DOCKING_LIBS))["library_id"]
+ .unique().to_list()
+ )
+ library_dropdown = mo.ui.dropdown(
+ options=_libraries,
+ value=_libraries[0] if _libraries else "",
+ label="ROCS library",
+ )
+ library_dropdown
+ return (library_dropdown,)
+
+
+@app.cell
+def _(
+ Line2D,
+ TOP_N,
+ fig_to_pdf_bytes,
+ library_dropdown,
+ mo,
+ np,
+ pl,
+ plt,
+ tukey_at_n,
+):
+ """Option 4 (per-library): Simultaneous CI plot for one library."""
+ _lib_id = library_dropdown.value
+ _lib_tukey = tukey_at_n.filter(pl.col("library_id") == _lib_id)
+
+ mo.stop(len(_lib_tukey) == 0, mo.md(f"No Tukey HSD results for {_lib_id} at this top-N."))
+
+ _n_comparisons = len(_lib_tukey)
+
+ _left_labels = []
+ _right_labels = []
+ _means = []
+ _ci_lows = []
+ _ci_highs = []
+ _rejects = []
+
+ # Normalize: mean_diff = group2 - group1 from Tukey HSD.
+ # Ensure the better method (higher recovery) is always the challenger (right).
+ # If mean_diff > 0, group2 is better → group1=baseline(left), group2=challenger(right).
+ # If mean_diff < 0, group1 is better → flip: group2=baseline(left), group1=challenger(right).
+ for _row in _lib_tukey.iter_rows(named=True):
+ if _row["mean_diff"] >= 0:
+ _left_labels.append(_row["group1"])
+ _right_labels.append(_row["group2"])
+ _means.append(_row["mean_diff"])
+ _ci_lows.append(_row["ci_low"])
+ _ci_highs.append(_row["ci_high"])
+ else:
+ _left_labels.append(_row["group2"])
+ _right_labels.append(_row["group1"])
+ _means.append(-_row["mean_diff"])
+ _ci_lows.append(-_row["ci_high"])
+ _ci_highs.append(-_row["ci_low"])
+ _rejects.append(_row["reject"])
+
+ # Sort by mean_diff descending after normalization
+ _order = np.argsort(_means)[::-1]
+ _left_labels = [_left_labels[i] for i in _order]
+ _right_labels = [_right_labels[i] for i in _order]
+ _rejects = [_rejects[i] for i in _order]
+ _means = np.array([_means[i] for i in _order])
+ _ci_lows = np.array([_ci_lows[i] for i in _order])
+ _ci_highs = np.array([_ci_highs[i] for i in _order])
+
+ _means = np.array(_means)
+ _ci_lows = np.array(_ci_lows)
+ _ci_highs = np.array(_ci_highs)
+
+ # Extract p-values for significance stars
+ _p_values = []
+ for _row in _lib_tukey.iter_rows(named=True):
+ _p_values.append(_row["p_adj"])
+ _p_values = np.array([_p_values[i] for i in _order])
+
+ _fig, _ax = plt.subplots(
+ figsize=(12, max(4, _n_comparisons * 0.55)),
+ facecolor="white",
+ )
+ _ax.set_facecolor("white")
+ _y = np.arange(_n_comparisons)
+
+ _colors = ["#4C78A8" if r else "#999999" for r in _rejects]
+
+ for _i in range(_n_comparisons):
+ _ax.plot(
+ [_ci_lows[_i], _ci_highs[_i]], [_i, _i],
+ color=_colors[_i], linewidth=2.5, solid_capstyle="round", zorder=3,
+ )
+ _ax.plot(_means[_i], _i, "o", color=_colors[_i], markersize=8, zorder=4)
+ # Significance stars
+ _p = _p_values[_i]
+ _sig_str = (
+ "****" if _p <= 0.0001
+ else "***" if _p <= 0.001
+ else "**" if _p <= 0.01
+ else "*" if _p <= 0.05
+ else "ns"
+ )
+ _ax.text(
+ _ci_highs[_i] + 0.15, _i, _sig_str,
+ ha="left", va="center",
+ fontsize=12, fontweight="bold", color="black",
+ )
+
+ _ax.axvline(0, color="black", linewidth=0.8, linestyle="-", zorder=2)
+
+ # Expand x-limits so stars fit inside the axes
+ _x_pad = max(_ci_highs.max(), abs(_ci_lows.min())) * 0.18
+ _ax.set_xlim(
+ min(_ci_lows.min(), 0) - _x_pad * 0.5,
+ _ci_highs.max() + _x_pad,
+ )
+
+ # Define key comparisons
+ _legacy_methods = {"Legacy Enhanced-RWS", "Legacy Standard-Greedy"}
+ _tactics_adaptive = {"TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"}
+
+ def _is_tactics_vs_legacy(_left, _right):
+ return (
+ (_left in _legacy_methods and _right in _tactics_adaptive)
+ or (_left in _tactics_adaptive and _right in _legacy_methods)
+ )
+
+ _COLOR_KEY_TICK = "#4C78A8"
+
+ # Left y-axis: challenger (better method)
+ _ax.set_yticks(_y)
+ _ax.set_yticklabels(_right_labels, fontsize=12, color="black")
+ _ax.set_ylabel("Challenger", fontsize=14, color="black")
+
+ # Pad y-limits so effect size labels above top row aren't clipped
+ _ax.set_ylim(-0.6, _n_comparisons - 0.4)
+
+ # Invert y so largest diff is at top (do this BEFORE twinx so both share orientation)
+ _ax.invert_yaxis()
+
+ # Right y-axis: baseline (worse method)
+ _ax_right = _ax.twinx()
+ _ax_right.set_ylim(_ax.get_ylim())
+ _ax_right.set_yticks(_y)
+ _ax_right.set_yticklabels(_left_labels, fontsize=12, color="black")
+ _ax_right.set_ylabel("Baseline", fontsize=14, color="black")
+ _ax_right.tick_params(colors="black", labelsize=12)
+ for _spine in _ax_right.spines.values():
+ _spine.set_color("black")
+
+ # Per-tick styling: challenger red+bold, baseline bold for key comparisons
+ for _i in range(_n_comparisons):
+ if _is_tactics_vs_legacy(_left_labels[_i], _right_labels[_i]):
+ _ax.get_yticklabels()[_i].set_color(_COLOR_KEY_TICK)
+ _ax.get_yticklabels()[_i].set_fontweight("bold")
+ _ax_right.get_yticklabels()[_i].set_fontweight("bold")
+
+ _ax.set_xlabel(
+ f"\u0394 Recovery: Challenger \u2212 Baseline (top-{TOP_N})",
+ fontsize=14, color="black",
+ )
+ _ax.set_title(
+ f"Tukey HSD Pairwise CIs: {_lib_id} (top-{TOP_N})",
+ fontsize=15, fontweight="bold", color="black",
+ )
+ _ax.grid(axis="x", alpha=0.3, zorder=1, color="grey")
+ _ax.tick_params(axis="x", colors="black", labelsize=12)
+ _ax.tick_params(axis="y", color="black", labelsize=12)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+
+ _legend_elements = [
+ Line2D([0], [0], color="#4C78A8", linewidth=2.5, marker="o",
+ markersize=8, label="Significant (p < 0.05)"),
+ Line2D([0], [0], color="#999999", linewidth=2.5, marker="o",
+ markersize=8, label="Not significant"),
+ ]
+ _ax.legend(handles=_legend_elements, loc="lower right", fontsize=12,
+ facecolor="white", edgecolor="black", labelcolor="black")
+
+ _fig.tight_layout()
+
+ _pdf_name = f"tukey_ci_{_lib_id}_top{TOP_N}.pdf".replace(" ", "_")
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename=_pdf_name,
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(Line2D, TOP_N, df, fig_to_pdf_bytes, mo, np, pl, plt):
+ """Across-library Tukey HSD pairwise CI plot (ROCS libraries only).
+
+ Excludes docking libraries (thrombin, adenine, amide, quinazoline) which
+ have different score distributions and only 1 query each. Docking results
+ are shown in the separate docking Tukey HSD CI plot.
+
+ Averages replicates within each (library, query) to get one independent
+ observation per query, then runs statsmodels pairwise_tukeyhsd for
+ FWER-controlled simultaneous CIs and adjusted p-values.
+
+ Layout matches the per-library Tukey CI plot: positive values always mean
+ the challenger (right y-axis) outperforms the baseline (left y-axis).
+ """
+ from statsmodels.stats.multicomp import pairwise_tukeyhsd
+
+ # Exclude docking libraries — different score distributions, 1 query each
+ _DOCKING_LIBS = {
+ "thrombin", "adenine", "adenine_hybrid", "amide", "quinazoline",
+ "thrombin_hybrid", "amide_hybrid", "quinazoline_hybrid",
+ }
+ _rocs_df = df.filter(~pl.col("library_id").is_in(_DOCKING_LIBS))
+
+ # Average replicates: one observation per (library, query, method)
+ _query_means = (
+ _rocs_df.group_by(["library_id", "query_id", "method"])
+ .agg(pl.col("recovered").mean().alias("mean_recovered"))
+ )
+
+ _values = _query_means["mean_recovered"].to_numpy()
+ _groups = _query_means["method"].to_list()
+
+ _tukey = pairwise_tukeyhsd(_values, _groups, alpha=0.05)
+ _summary = _tukey.summary()
+
+ # Parse Tukey HSD results into lists
+ _left_labels = [] # baseline (worse method)
+ _right_labels = [] # challenger (better method)
+ _grand_means = []
+ _ci_lows = []
+ _ci_highs = []
+ _p_values = []
+ _rejects = []
+
+ for _row in _summary.data[1:]: # skip header row
+ _g1, _g2, _meandiff, _p_adj, _lower, _upper, _reject = _row
+ _meandiff = float(_meandiff)
+ _p_adj = float(_p_adj)
+ _lower = float(_lower)
+ _upper = float(_upper)
+
+ # Normalize direction: positive = challenger (right) is better.
+ if _meandiff >= 0:
+ _left_labels.append(_g1)
+ _right_labels.append(_g2)
+ _grand_means.append(_meandiff)
+ _ci_lows.append(_lower)
+ _ci_highs.append(_upper)
+ else:
+ _left_labels.append(_g2)
+ _right_labels.append(_g1)
+ _grand_means.append(-_meandiff)
+ _ci_lows.append(-_upper)
+ _ci_highs.append(-_lower)
+
+ _p_values.append(_p_adj)
+ _rejects.append(bool(_reject))
+
+ _grand_means = np.array(_grand_means)
+ _ci_lows = np.array(_ci_lows)
+ _ci_highs = np.array(_ci_highs)
+ _p_values = np.array(_p_values)
+
+ # Sort by grand mean descending (largest positive diff at top)
+ _order = np.argsort(_grand_means)[::-1]
+ _left_labels = [_left_labels[i] for i in _order]
+ _right_labels = [_right_labels[i] for i in _order]
+ _rejects = [_rejects[i] for i in _order]
+ _grand_means = _grand_means[_order]
+ _ci_lows = _ci_lows[_order]
+ _ci_highs = _ci_highs[_order]
+ _p_values = _p_values[_order]
+
+ _n_comparisons = len(_left_labels)
+ _y = np.arange(_n_comparisons)
+
+ _fig, _ax = plt.subplots(
+ figsize=(12, max(4, _n_comparisons * 0.55)),
+ facecolor="white",
+ )
+ _ax.set_facecolor("white")
+
+ # Define the key TACTICS-vs-Legacy comparisons to highlight
+ _legacy_methods = {"Legacy Enhanced-RWS", "Legacy Standard-Greedy"}
+ _tactics_adaptive = {"TACTICS Enhanced-RWS (GMIC)", "TACTICS Enhanced-TT-TS (GMIC)"}
+
+ def _is_tactics_vs_legacy(_left, _right):
+ """True if one side is a legacy method and the other is a TACTICS adaptive method."""
+ return (
+ (_left in _legacy_methods and _right in _tactics_adaptive)
+ or (_left in _tactics_adaptive and _right in _legacy_methods)
+ )
+
+ _COLOR_SIG = "#4C78A8" # blue — significant pairs
+ _COLOR_NS = "#999999" # grey — not significant
+ _COLOR_KEY_TICK = "#4C78A8" # blue — TACTICS vs Legacy tick labels
+
+ for _i in range(_n_comparisons):
+ _color = _COLOR_SIG if _rejects[_i] else _COLOR_NS
+
+ # CI line
+ _ax.plot(
+ [_ci_lows[_i], _ci_highs[_i]], [_i, _i],
+ color=_color, linewidth=2.5, solid_capstyle="round", zorder=3,
+ )
+ # Point estimate
+ _ax.plot(
+ _grand_means[_i], _i, "o",
+ color=_color, markersize=8, zorder=4,
+ )
+ # Significance stars (Tukey HSD adjusted p-values, FWER=0.05)
+ _p = _p_values[_i]
+ _sig_str = (
+ "****" if _p <= 0.0001
+ else "***" if _p <= 0.001
+ else "**" if _p <= 0.01
+ else "*" if _p <= 0.05
+ else "ns"
+ )
+ _ax.text(
+ _ci_highs[_i] + 0.15, _i, _sig_str,
+ ha="left", va="center",
+ fontsize=12, fontweight="bold", color="black",
+ )
+
+ _ax.axvline(0, color="black", linewidth=0.8, linestyle="-", zorder=2)
+
+ # Expand x-limits so stars and CI lines fit inside the axes
+ _x_pad = max(_ci_highs.max(), abs(_ci_lows.min())) * 0.18
+ _ax.set_xlim(
+ min(_ci_lows.min(), 0) - _x_pad * 0.5,
+ _ci_highs.max() + _x_pad,
+ )
+
+ # Left y-axis: challenger (better method)
+ _ax.set_yticks(_y)
+ _ax.set_yticklabels(_right_labels, fontsize=12, color="black")
+ _ax.set_ylabel("Challenger", fontsize=14, color="black")
+
+ # Pad y-limits so effect size labels above top/bottom rows aren't clipped
+ _ax.set_ylim(-0.6, _n_comparisons - 0.4)
+
+ # Invert y so largest diff is at top (before twinx so both share orientation)
+ _ax.invert_yaxis()
+
+ # Right y-axis: baseline (worse method)
+ _ax_right = _ax.twinx()
+ _ax_right.set_ylim(_ax.get_ylim())
+ _ax_right.set_yticks(_y)
+ _ax_right.set_yticklabels(_left_labels, fontsize=12, color="black")
+ _ax_right.set_ylabel("Baseline", fontsize=14, color="black")
+ _ax_right.tick_params(colors="black", labelsize=12)
+ for _spine in _ax_right.spines.values():
+ _spine.set_color("black")
+
+ # Per-tick styling: challenger red+bold, baseline bold for key comparisons
+ # Applied after both axes are fully configured so nothing resets them
+ for _i in range(_n_comparisons):
+ if _is_tactics_vs_legacy(_left_labels[_i], _right_labels[_i]):
+ _ax.get_yticklabels()[_i].set_color(_COLOR_KEY_TICK)
+ _ax.get_yticklabels()[_i].set_fontweight("bold")
+ _ax_right.get_yticklabels()[_i].set_fontweight("bold")
+
+ _ax.set_xlabel(
+ f"\u0394 Recovery: Challenger \u2212 Baseline (top-{TOP_N})",
+ fontsize=14, color="black",
+ )
+ _n_rocs_libs = _rocs_df["library_id"].n_unique()
+ _ax.set_title(
+ f"Tukey HSD Pairwise CIs \u2014 {_n_rocs_libs} ROCS Libraries (top-{TOP_N}, FWER=0.05)",
+ fontsize=15, fontweight="bold", color="black",
+ )
+ _ax.grid(axis="x", alpha=0.3, zorder=1, color="grey")
+ _ax.tick_params(axis="x", colors="black", labelsize=12)
+ _ax.tick_params(axis="y", color="black", labelsize=12)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+
+ _legend_elements = [
+ Line2D([0], [0], color=_COLOR_SIG, linewidth=2.5, marker="o",
+ markersize=8, label="Significant (p < 0.05)"),
+ Line2D([0], [0], color=_COLOR_NS, linewidth=2.5, marker="o",
+ markersize=8, label="Not significant"),
+ ]
+ _ax.legend(handles=_legend_elements, loc="lower right", fontsize=12,
+ facecolor="white", edgecolor="black", labelcolor="black")
+
+ _fig.tight_layout()
+
+ _pdf_name = f"tukey_hsd_ci_across_libraries_top{TOP_N}.pdf"
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename=_pdf_name,
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(mo, recovery_all):
+ """Per-library ROCS recovery — library selector (20 ROCS libraries only)."""
+ _ROCS_LIBS = sorted(
+ lib for lib in recovery_all["library_id"].unique().to_list()
+ if lib not in {
+ "thrombin", "adenine", "adenine_hybrid", "amide", "quinazoline",
+ "thrombin_hybrid", "amide_hybrid", "quinazoline_hybrid",
+ }
+ )
+ rocs_library_selector = mo.ui.dropdown(
+ options=_ROCS_LIBS,
+ value=_ROCS_LIBS[0] if _ROCS_LIBS else None,
+ label="ROCS library",
+ )
+ mo.vstack([
+ mo.md(r"""
+ ## Per-library ROCS recovery — Top-N curves + best-TACTICS Δ bars
+
+ Pick a ROCS library to view its 4-method Top-N recovery curves (left panel)
+ plus best-TACTICS Δ-recovery bars vs Legacy Greedy / Legacy RWS (right panel).
+ Significance stars use **Tukey HSD** (per the scoring-type rule: ROCS libraries
+ have 109 queries × 20 replicates → repeated-measures structure → Tukey HSD applies).
+ """),
+ rocs_library_selector,
+ ])
+ return (rocs_library_selector,)
+
+
+@app.cell
+def _(
+ TUKEY_STAR_COLOR,
+ fig_to_pdf_bytes,
+ mo,
+ np,
+ pl,
+ plt,
+ project_root,
+ recovery_all,
+ rocs_library_selector,
+):
+ """Per-library ROCS bar+line plot — Top-N curves + TT-TS-vs-Legacy Δ bars with Tukey HSD stars.
+
+ Bar height = mean(TT-TS) − mean(Legacy) on this library at this top-N.
+ On ROCS the two TACTICS methods are practically equivalent at the per-library
+ level (max gap 0.79 pp ≈ 0.2 compounds, on `poparov` at top-25), so headlining
+ TT-TS is the simpler and equally accurate story.
+ """
+
+ _LIB = rocs_library_selector.value
+ _CHALLENGERS = [
+ ("TACTICS Enhanced-TT-TS (GMIC)", "TT-TS", "#E45756", "D"),
+ ("TACTICS Enhanced-RWS (GMIC)", "TACTICS RWS", "#4C78A8", "o"),
+ ]
+ _LEGACIES = [
+ ("Legacy Standard-Greedy", "Legacy Greedy", "#76B7B2", "s", ""),
+ ("Legacy Enhanced-RWS", "Legacy RWS", "#B07AA1", "^", "////"),
+ ]
+
+ # Tukey HSD significance results for this library (ROCS uses Tukey per the rule).
+ _tukey_all = pl.read_parquet(
+ project_root / "outputs" / "full_benchmark_ema_cats" / "analysis" / "tukey_hsd_results.parquet"
+ ).filter(pl.col("library_id") == _LIB)
+
+ _methods_keep = {m for m, _, _, _ in _CHALLENGERS} | {m for m, _, _, _, _ in _LEGACIES}
+ _lib_all = recovery_all.filter(
+ (pl.col("library_id") == _LIB) & pl.col("method").is_in(_methods_keep)
+ )
+ _topn_vals = sorted([_tn for _tn in _lib_all["top_n"].unique().to_list() if _tn <= 250])
+
+ mo.stop(_lib_all.height == 0, mo.md(f"No data for {_LIB}."))
+
+ _fig, (_ax1, _ax2) = plt.subplots(1, 2, figsize=(16, 6), facecolor="white")
+ _ax1.set_facecolor("white")
+
+ _agg = (
+ _lib_all.group_by(["method", "top_n"])
+ .agg([
+ pl.col("recovery_frac").mean().alias("mean_frac"),
+ pl.col("recovery_frac").std().alias("std_frac"),
+ pl.col("recovery_frac").count().alias("n"),
+ ])
+ .with_columns((pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac"))
+ .sort(["method", "top_n"])
+ )
+
+ for _meth, _lbl, _clr, _mkr, _ in _LEGACIES:
+ _md = _agg.filter(pl.col("method") == _meth).sort("top_n")
+ if len(_md) == 0:
+ continue
+ _x = _md["top_n"].to_numpy()
+ _y = _md["mean_frac"].to_numpy() * 100
+ _se = _md["se_frac"].to_numpy() * 100
+ _ax1.plot(_x, _y, marker=_mkr, color=_clr, linewidth=2, markersize=7, label=_lbl, zorder=3)
+ _ax1.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se, alpha=0.15, color=_clr, zorder=2)
+
+ for _meth, _lbl, _clr, _mkr in _CHALLENGERS:
+ _md = _agg.filter(pl.col("method") == _meth).sort("top_n")
+ if len(_md) == 0:
+ continue
+ _x = _md["top_n"].to_numpy()
+ _y = _md["mean_frac"].to_numpy() * 100
+ _se = _md["se_frac"].to_numpy() * 100
+ _ax1.plot(_x, _y, marker=_mkr, color=_clr, linewidth=2, markersize=7, label=_lbl, zorder=4)
+ _ax1.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se, alpha=0.15, color=_clr, zorder=2)
+
+ _ax1.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax1.set_ylabel("% Recovery", fontsize=14, color="black")
+ _ax1.set_title(f"{_LIB} (ROCS) — Recovery vs Top-N", fontsize=14, fontweight="bold", color="black")
+ _ax1.set_xticks(_topn_vals)
+ _ax1.tick_params(colors="black", labelsize=12)
+ _ax1.grid(alpha=0.3, color="grey", zorder=1)
+ _ax1.legend(fontsize=11, facecolor="white", edgecolor="black", labelcolor="black")
+ for _sp in _ax1.spines.values():
+ _sp.set_color("black")
+
+ _BAR_COLORS = ["#F28E2B", "#4E79A7"] # orange (vs Greedy), steel blue (vs Legacy RWS)
+ _ax2.set_facecolor("white")
+ _bar_width = 0.35
+ _x_pos = np.arange(len(_topn_vals))
+
+ def _tukey_sig(library, top_n, method_a, method_b):
+ """Look up Tukey HSD significance for (method_a, method_b) at top_n; try both directions."""
+ _row = _tukey_all.filter(
+ (pl.col("top_n") == top_n)
+ & (
+ ((pl.col("group1") == method_a) & (pl.col("group2") == method_b))
+ | ((pl.col("group1") == method_b) & (pl.col("group2") == method_a))
+ )
+ )
+ return _row["significance"][0] if _row.height > 0 else ""
+
+ _HEADLINE_TACTICS = "TACTICS Enhanced-TT-TS (GMIC)"
+
+ for _gi, (_leg_meth, _leg_lbl, _, _, _) in enumerate(_LEGACIES):
+ _diffs = []
+ _sigs = []
+ for _tn in _topn_vals:
+ _tt_frac = _lib_all.filter(
+ (pl.col("method") == _HEADLINE_TACTICS) & (pl.col("top_n") == _tn)
+ )["recovery_frac"].mean()
+ _leg_frac = _lib_all.filter(
+ (pl.col("method") == _leg_meth) & (pl.col("top_n") == _tn)
+ )["recovery_frac"].mean()
+ _diffs.append(
+ float((_tt_frac - _leg_frac) * 100)
+ if _tt_frac is not None and _leg_frac is not None else 0
+ )
+ _sigs.append(_tukey_sig(_LIB, _tn, _HEADLINE_TACTICS, _leg_meth))
+
+ _diffs = np.array(_diffs)
+ _offset = (_gi - 0.5) * _bar_width
+ _ax2.bar(_x_pos + _offset, _diffs, width=_bar_width, color=_BAR_COLORS[_gi],
+ edgecolor="black", linewidth=0.5, zorder=3, label=f"vs {_leg_lbl}")
+
+ _dmax = max(abs(_diffs)) if len(_diffs) > 0 else 1
+ _star_pad = _dmax * 0.03
+ for _i, (_d, _s) in enumerate(zip(_diffs, _sigs)):
+ if _s and _s != "ns":
+ _yp = _d + _star_pad if _d >= 0 else _d - _star_pad
+ _ax2.text(_x_pos[_i] + _offset, _yp, _s, ha="center",
+ va="bottom" if _d >= 0 else "top",
+ fontsize=9, fontweight="bold", color=TUKEY_STAR_COLOR,
+ rotation=90)
+
+ _ax2.axhline(0, color="black", linewidth=0.8, zorder=2)
+ _yl, _yh = _ax2.get_ylim()
+ _ax2.set_ylim(_yl - abs(_yh - _yl) * 0.10, _yh + abs(_yh - _yl) * 0.30)
+ _ax2.set_xticks(_x_pos)
+ _ax2.set_xticklabels([str(t) for t in _topn_vals], fontsize=12, color="black")
+ _ax2.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax2.set_ylabel(
+ "Δ % Recovery (TACTICS TT-TS − Legacy)",
+ fontsize=14, color="black",
+ )
+ _ax2.set_title(
+ f"{_LIB} (ROCS) — TACTICS TT-TS Advantage",
+ fontsize=14, fontweight="bold", color="black",
+ )
+ _ax2.tick_params(colors="black", labelsize=12)
+ _ax2.grid(axis="y", alpha=0.3, color="grey", zorder=1)
+ _ax2.legend(fontsize=11, facecolor="white", edgecolor="black", labelcolor="black")
+ for _sp in _ax2.spines.values():
+ _sp.set_color("black")
+ _fig.tight_layout()
+ mo.vstack([
+ _fig,
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename=f"{_LIB}_topn_tactics_vs_legacy_ROCS.pdf",
+ mimetype="application/pdf", label="Download PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 4 — Budget Sensitivity Analysis (1% → 5%) — ROCS
+
+ Recovery and TACTICS-vs-Legacy gap as a function of library-fraction screening
+ budget for ROCS libraries, at the canonical top-100 threshold. ROCS aggregates
+ are tight (uniform per-library response within each component-count cell);
+ the parallel DOCK sensitivity panel lives in `manuscript_plots.py`.
+
+ 2×2 panel grid:
+ - **Top row** (line plots): per-method mean recovery vs budget fraction.
+ - **Bottom row** (bar plots): Δ(best-TACTICS − best-Legacy) per budget with error bars.
+ - **Left column**: 2-component libraries. **Right column**: 3-component libraries.
+
+ Recovery is averaged per query × replicate within each library, then aggregated
+ across libraries within each (n_comp) cell.
+ """)
+ return
+
+
+@app.cell
+def _(pl, project_root):
+ """Load top-100 recovery from all 5 budget benchmarks into a single dataframe."""
+ _budgets = [
+ (1, project_root / "outputs/full_benchmark_1pct/analysis/recovery_summary.parquet"),
+ (2, project_root / "outputs/full_benchmark_ema_cats/analysis/recovery_summary.parquet"),
+ (3, project_root / "outputs/full_benchmark_3pct/analysis/recovery_summary.parquet"),
+ (4, project_root / "outputs/full_benchmark_4pct/analysis/recovery_summary.parquet"),
+ (5, project_root / "outputs/full_benchmark_5pct/analysis/recovery_summary.parquet"),
+ ]
+ _frames = []
+ for _budget, _path in _budgets:
+ if not _path.exists():
+ continue
+ _sub = (
+ pl.read_parquet(_path)
+ .filter(pl.col("top_n") == 100)
+ .with_columns(pl.lit(_budget).alias("budget_pct"))
+ )
+ _frames.append(_sub)
+ sensitivity_data = pl.concat(_frames)
+ return (sensitivity_data,)
+
+
+@app.cell
+def _(pl, sensitivity_data):
+ """Compute per-(library, method, budget) mean recovery + per-(n_comp, method, budget) aggregates.
+
+ Pipeline:
+ 1. Per (library, query, replicate, method, budget) → already in recovery_summary
+ 2. Mean over (query, replicate) → per (library, method, budget) library-level mean
+ 3. Mean across libraries in each (n_comp, method, budget) cell → aggregate mean + std
+ 4. Per-library Δ = best-T(lib) − best-L(lib); aggregate Δ mean + std across libs
+
+ Returns aggregate dataframes ready for plotting.
+ """
+ _METHOD_SHORT = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "TT-TS",
+ "TACTICS Enhanced-RWS (GMIC)": "T-RWS",
+ "Legacy Enhanced-RWS": "L-RWS",
+ "Legacy Standard-Greedy": "L-Grdy",
+ }
+ _ROCS_NCOMP = {
+ "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
+ "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
+ "amide-suzuki": 3, "betti": 3, "dobener": 3, "groebke-blackburn-bienayme": 3,
+ "mannich": 3, "niementowski": 3, "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
+ }
+ _DOCK_NCOMP = {
+ "thrombin": 2, "thrombin_hybrid": 2, "amide": 2, "amide_hybrid": 2,
+ "adenine": 3, "adenine_hybrid": 3, "quinazoline": 3, "quinazoline_hybrid": 3,
+ }
+
+ # Add kind + n_comp to the data
+ _ncomp_map = {**_ROCS_NCOMP, **_DOCK_NCOMP}
+ _kind_map = {lib: "ROCS" for lib in _ROCS_NCOMP}
+ _kind_map.update({lib: "DOCK" for lib in _DOCK_NCOMP})
+
+ _df = (
+ sensitivity_data
+ .with_columns(pl.col("method").replace_strict(_METHOD_SHORT).alias("m"))
+ .filter(pl.col("library_id").is_in(set(_ncomp_map.keys())))
+ .with_columns([
+ pl.col("library_id").replace(_ncomp_map).cast(pl.Int64).alias("n_comp"),
+ pl.col("library_id").replace(_kind_map).alias("kind"),
+ ])
+ )
+
+ # Step 1: per-library mean across (query, replicate)
+ sens_per_lib = (
+ _df.group_by(["library_id", "kind", "n_comp", "m", "budget_pct"])
+ .agg(pl.col("recovery_frac").mean().alias("rec"))
+ )
+
+ # Step 2: per-method aggregate across libraries within each (kind, n_comp, budget)
+ sens_per_method = (
+ sens_per_lib.group_by(["kind", "n_comp", "m", "budget_pct"])
+ .agg([
+ pl.col("rec").mean().alias("mean_rec"),
+ pl.col("rec").std().alias("std_rec"),
+ pl.col("rec").count().alias("n_libs"),
+ ])
+ .with_columns(
+ (pl.col("std_rec") / pl.col("n_libs").sqrt()).alias("se_rec")
+ )
+ )
+
+ # Step 3: per-library Δ = best-T − best-L
+ _wide = (
+ sens_per_lib.pivot(
+ values="rec", index=["library_id", "kind", "n_comp", "budget_pct"], on="m"
+ )
+ )
+ _delta_per_lib = _wide.with_columns([
+ pl.max_horizontal(["TT-TS", "T-RWS"]).alias("best_T"),
+ pl.max_horizontal(["L-RWS", "L-Grdy"]).alias("best_L"),
+ ]).with_columns(
+ ((pl.col("best_T") - pl.col("best_L")) * 100).alias("delta_pp")
+ )
+ sens_delta_per_lib = _delta_per_lib
+
+ # Step 4: aggregate Δ per (kind, n_comp, budget)
+ sens_delta_agg = (
+ _delta_per_lib.group_by(["kind", "n_comp", "budget_pct"])
+ .agg([
+ pl.col("delta_pp").mean().alias("mean_delta"),
+ pl.col("delta_pp").std().alias("std_delta"),
+ pl.col("delta_pp").count().alias("n_libs"),
+ ])
+ .with_columns(
+ (pl.col("std_delta") / pl.col("n_libs").sqrt()).alias("se_delta")
+ )
+ .sort(["kind", "n_comp", "budget_pct"])
+ )
+ return sens_delta_agg, sens_per_method
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, pl, plt, sens_delta_agg, sens_per_method):
+ """ROCS sensitivity figure: 2×2 grid (line plots top, bar plots bottom; 2-comp left, 3-comp right)."""
+ _METHOD_ORDER = ["TT-TS", "T-RWS", "L-RWS", "L-Grdy"]
+ _METHOD_LABELS = {
+ "TT-TS": "TACTICS TT-TS (GMIC)",
+ "T-RWS": "TACTICS RWS (GMIC)",
+ "L-RWS": "Legacy Enhanced-RWS",
+ "L-Grdy": "Legacy Standard-Greedy",
+ }
+ _METHOD_COLORS = {
+ "TT-TS": "#E45756", "T-RWS": "#4C78A8",
+ "L-RWS": "#B07AA1", "L-Grdy": "#76B7B2",
+ }
+ _METHOD_MARKERS = {
+ "TT-TS": "D", "T-RWS": "o", "L-RWS": "^", "L-Grdy": "s",
+ }
+
+ _fig, _axes = plt.subplots(2, 2, figsize=(14, 9), facecolor="white", sharex=True)
+
+ for _col_idx, _nc in enumerate([2, 3]):
+ _ax_line = _axes[0, _col_idx]
+ _ax_bar = _axes[1, _col_idx]
+ _ax_line.set_facecolor("white"); _ax_bar.set_facecolor("white")
+
+ # --- Top row: line plot of recovery vs budget ---
+ _sub_method = sens_per_method.filter(
+ (pl.col("kind") == "ROCS") & (pl.col("n_comp") == _nc)
+ )
+ _n_libs = int(_sub_method["n_libs"][0]) if _sub_method.height > 0 else 0
+ for _m in _METHOD_ORDER:
+ _md = _sub_method.filter(pl.col("m") == _m).sort("budget_pct")
+ if _md.height == 0: continue
+ _x = _md["budget_pct"].to_numpy()
+ _y = _md["mean_rec"].to_numpy() * 100
+ _se = _md["se_rec"].to_numpy() * 100
+ _ax_line.errorbar(
+ _x, _y, yerr=1.96 * _se, color=_METHOD_COLORS[_m],
+ marker=_METHOD_MARKERS[_m], markersize=7, linewidth=2,
+ capsize=4, capthick=1.2, label=_METHOD_LABELS[_m], zorder=3,
+ )
+ _ax_line.set_ylabel("Mean Top-100 Recovery (%)", fontsize=12, color="black")
+ _ax_line.set_title(f"ROCS {_nc}-Component (n = {_n_libs} libraries)",
+ fontsize=13, fontweight="bold", color="black")
+ _ax_line.grid(alpha=0.3, color="grey", zorder=1)
+ _ax_line.tick_params(colors="black", labelsize=11)
+ for _sp in _ax_line.spines.values(): _sp.set_color("black")
+
+ # --- Bottom row: Δ bar plot ---
+ _sub_delta = sens_delta_agg.filter(
+ (pl.col("kind") == "ROCS") & (pl.col("n_comp") == _nc)
+ ).sort("budget_pct")
+ _x = _sub_delta["budget_pct"].to_numpy()
+ _y = _sub_delta["mean_delta"].to_numpy()
+ _se = _sub_delta["se_delta"].to_numpy()
+ _ax_bar.bar(_x, _y, yerr=1.96 * _se, color="#4C78A8", alpha=0.85,
+ edgecolor="black", linewidth=0.8, capsize=5,
+ error_kw={"linewidth": 1.2, "color": "black"}, zorder=3)
+ _ax_bar.axhline(0, color="black", linewidth=0.8, zorder=2)
+ _ax_bar.set_xlabel("Library Fraction (%)", fontsize=12, color="black")
+ _ax_bar.set_ylabel("Δ Best-TACTICS − Best-Legacy (pp)", fontsize=12, color="black")
+ _ax_bar.set_xticks([1, 2, 3, 4, 5])
+ _ax_bar.grid(axis="y", alpha=0.3, color="grey", zorder=1)
+ _ax_bar.tick_params(colors="black", labelsize=11)
+ for _sp in _ax_bar.spines.values(): _sp.set_color("black")
+ # Annotate bar values
+ for _xi, _yi in zip(_x, _y):
+ _ax_bar.text(_xi, _yi + 0.15, f"{_yi:+.2f}", ha="center", va="bottom",
+ fontsize=10, color="black", fontweight="bold")
+
+ _handles, _labels = _axes[0, 0].get_legend_handles_labels()
+ _fig.legend(_handles, _labels, loc="lower center", ncol=5, fontsize=11,
+ facecolor="white", edgecolor="black", labelcolor="black",
+ bbox_to_anchor=(0.5, -0.04))
+ _fig.suptitle("Budget Sensitivity — ROCS Libraries (Top-100 Recovery)",
+ fontsize=14, fontweight="bold", color="black", y=1.00)
+ _fig.tight_layout(rect=[0, 0.04, 1, 0.98])
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="sensitivity_rocs.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+if __name__ == "__main__":
+ app.run()
diff --git a/tutorials/manuscript_plots_docking.py b/tutorials/manuscript_plots_docking.py
new file mode 100644
index 0000000..83bb0b2
--- /dev/null
+++ b/tutorials/manuscript_plots_docking.py
@@ -0,0 +1,1047 @@
+"""
+TACTICS Manuscript Plots — Docking Libraries
+
+Docking-only figure notebook (8 libraries: 4 FRED + 4 HYBRID). Loads the canonical
+recovery summary (`full_benchmark_ema_cats`) and renders, in order:
+ §1 method × docking-library top-100 recovery heatmap,
+ §2 per-library two-panel recovery + significance (dropdown-selected),
+ §3 aggregate recovery vs top-N (FRED / HYBRID),
+ §4 statistical-test convention reminder,
+ §5 budget-sensitivity analysis (1% → 5%).
+The ROCS counterpart lives in manuscript_plots_ROCS.py.
+
+Run as app: marimo run tutorials/manuscript_plots_docking.py
+Edit mode: marimo edit tutorials/manuscript_plots_docking.py
+"""
+
+import marimo
+
+__generated_with = "0.21.1"
+app = marimo.App(width="full", app_title="TACTICS Manuscript — Docking")
+
+
+@app.cell
+def _():
+ """Imports."""
+ import io
+ import marimo as mo
+ import sys
+ from pathlib import Path
+
+ import matplotlib.pyplot as plt
+ from matplotlib.lines import Line2D
+ from matplotlib.patches import Patch
+ import numpy as np
+ import polars as pl
+ from scipy import stats
+
+ def fig_to_pdf_bytes(fig):
+ """Convert a matplotlib figure to PDF bytes in memory."""
+ buf = io.BytesIO()
+ fig.savefig(buf, format="pdf", bbox_inches="tight", dpi=300)
+ return buf.getvalue()
+
+ try:
+ project_root = Path(__file__).parent.parent.resolve()
+ except NameError:
+ project_root = Path.cwd()
+
+ sys.path.insert(0, str(project_root))
+ sys.path.insert(0, str(project_root / "src"))
+ return fig_to_pdf_bytes, mo, np, pl, plt, project_root
+
+
+@app.cell
+def _(mo, pl, project_root):
+ """Load the canonical docking recovery summary (docking libraries only).
+
+ Recovery from the canonical 25-library benchmark (`full_benchmark_ema_cats`),
+ capped at top-N ≤ 250 (manuscript-wide scope) and filtered to the eight docking
+ libraries. All five methods are kept so the §1 heatmap and the §2 per-library
+ figures always have the full method set.
+ """
+ analysis_dir = project_root / "outputs" / "full_benchmark_ema_cats" / "analysis"
+
+ DOCK_LIBS = [
+ "adenine", "adenine_hybrid",
+ "amide", "amide_hybrid",
+ "quinazoline", "quinazoline_hybrid",
+ "thrombin", "thrombin_hybrid",
+ ]
+
+ recovery_all = (
+ pl.read_parquet(analysis_dir / "recovery_summary.parquet")
+ .filter(pl.col("top_n") <= 250)
+ .filter(pl.col("library_id").is_in(DOCK_LIBS))
+ )
+
+ mo.md(
+ f"""
+ ## Data Loaded — Docking Libraries Only
+
+ - **Source**: `{analysis_dir}/recovery_summary.parquet`
+ - **Docking libraries** ({len(DOCK_LIBS)}): {', '.join(DOCK_LIBS)}
+ - **Top-N thresholds**: {sorted(recovery_all['top_n'].unique().to_list())}
+ - **Methods**: {', '.join(sorted(recovery_all['method'].unique().to_list()))}
+ """
+ )
+ return DOCK_LIBS, recovery_all
+
+
+@app.cell
+def _():
+ """Statistical-test selection rule and star color convention.
+
+ Rule (by scoring type):
+ - Docking libraries → Welch's t-test + Bonferroni-4 on the planned
+ TACTICS-vs-Legacy pairs. Docking benchmarks here use one query
+ per library × 20 replicates, so within-method variance (including
+ bimodal-failure modes) shows up directly. Tukey HSD's pooled-MSE
+ homoscedasticity assumption is violated whenever any single method
+ is bimodal, so Welch is the appropriate confirmatory test.
+ - ROCS libraries → Tukey HSD per Ash et al. ROCS uses 109 queries
+ × 20 replicates, and cross-query variance dominates the per-method
+ variance, so the variances are approximately homogeneous across
+ methods and Tukey's pooled-MSE assumption holds.
+
+ Docking libraries: thrombin, adenine, adenine_hybrid, amide, quinazoline.
+ """
+ DOCKING_LIBS = {
+ "thrombin", "adenine", "adenine_hybrid", "amide", "quinazoline",
+ "thrombin_hybrid", "amide_hybrid", "quinazoline_hybrid",
+ }
+ TUKEY_STAR_COLOR = "black"
+ WELCH_STAR_COLOR = "#4477AA"
+ return (WELCH_STAR_COLOR,)
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 1 — Method × Docking-Library Performance Heatmap
+
+ Top-100 recovery (%) for all five methods across the eight docking landscapes
+ (4 FRED + 4 HYBRID). Columns are ordered by the best-TACTICS − best-Legacy gap
+ (largest on the left), so the libraries where TACTICS helps most appear first.
+ """)
+ return
+
+
+@app.cell
+def _(DOCK_LIBS, fig_to_pdf_bytes, mo, np, pl, plt, recovery_all):
+ """Ranking heatmap — top-100 recovery (%) for 5 methods × 8 docking landscapes.
+
+ Computed from the canonical recovery_summary (mean recovery_frac at top-100 per
+ method × library), so it matches the §2 per-library figures exactly. Columns are
+ ordered by the best-TACTICS − best-Legacy gap (largest on the left).
+ """
+ _METHODS = [
+ "TACTICS Enhanced-TT-TS (GMIC)",
+ "TACTICS Enhanced-RWS (GMIC)",
+ "Legacy Enhanced-RWS",
+ "Legacy Standard-Greedy",
+ ]
+ _MSHORT = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "TACTICS TT-TS",
+ "TACTICS Enhanced-RWS (GMIC)": "TACTICS RWS",
+ "Legacy Enhanced-RWS": "Legacy RWS",
+ "Legacy Standard-Greedy": "Legacy Greedy",
+ }
+ _TT, _RWS = "TACTICS Enhanced-TT-TS (GMIC)", "TACTICS Enhanced-RWS (GMIC)"
+ _LRWS, _LGRD = "Legacy Enhanced-RWS", "Legacy Standard-Greedy"
+
+ # Mean top-100 recovery (%) per (library, method).
+ _agg = (
+ recovery_all.filter(pl.col("top_n") == 100)
+ .group_by(["library_id", "method"])
+ .agg((pl.col("recovery_frac").mean() * 100).alias("recovery"))
+ )
+ _pairs = {
+ (_row["library_id"], _row["method"]): _row["recovery"]
+ for _row in _agg.iter_rows(named=True)
+ }
+ _recov = {
+ _lab: {_m: _pairs.get((_lab, _m), float("nan")) for _m in _METHODS}
+ for _lab in DOCK_LIBS
+ }
+ _delta = {
+ _lab: max(_recov[_lab][_TT], _recov[_lab][_RWS])
+ - max(_recov[_lab][_LRWS], _recov[_lab][_LGRD])
+ for _lab in DOCK_LIBS
+ }
+ _libs = sorted(DOCK_LIBS, key=lambda _l: _delta[_l], reverse=True)
+ _mat = np.array([[_recov[_l][_m] for _l in _libs] for _m in _METHODS])
+
+ _fig, _ax = plt.subplots(figsize=(14, 5.2), facecolor="white")
+ _ax.set_facecolor("white")
+ _im = _ax.imshow(_mat, cmap="viridis", aspect="auto", vmin=0, vmax=100)
+ _ax.set_xticks(range(len(_libs)))
+ _ax.set_xticklabels(_libs, rotation=30, ha="right", rotation_mode="anchor",
+ fontsize=11, color="black")
+ _ax.set_yticks(range(len(_METHODS)))
+ _ax.set_yticklabels([_MSHORT[_m] for _m in _METHODS], fontsize=12, color="black")
+ for _r in range(_mat.shape[0]):
+ for _c in range(_mat.shape[1]):
+ _val = _mat[_r, _c]
+ _txt_c = "white" if _val < 55 else "black"
+ _ax.text(_c, _r, f"{_val:.0f}", ha="center", va="center",
+ fontsize=11, color=_txt_c, fontweight="bold")
+ _ax.set_title("Top-100 recovery (%) — methods × docking landscapes",
+ fontsize=14, fontweight="bold", color="black")
+ _ax.tick_params(colors="black")
+ _cbar = _fig.colorbar(_im, ax=_ax, fraction=0.025, pad=0.015)
+ _cbar.set_label("Top-100 recovery (%)", fontsize=12, color="black")
+ _cbar.ax.tick_params(colors="black", labelsize=10)
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="docking_ranking_heatmap.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 2 — Per-Library Recovery & Significance
+
+ Select a docking library to view its two-panel figure: **(left)** % recovery vs
+ top-N for the two TACTICS methods and the two Legacy baselines with 95% CI bands,
+ and **(right)** Δ recovery (best-TACTICS − each Legacy) at every top-N with planned
+ Welch + Bonferroni-4 significance stars (docking-libraries scoring-type rule).
+ """)
+ return
+
+
+@app.cell
+def _(DOCK_LIBS, mo):
+ """Dropdown to select which docking library's two-panel figure to display."""
+ dock_library_dropdown = mo.ui.dropdown(
+ options=DOCK_LIBS, value="adenine", label="Docking library",
+ )
+ dock_library_dropdown
+ return (dock_library_dropdown,)
+
+
+@app.cell
+def _(
+ WELCH_STAR_COLOR,
+ dock_library_dropdown,
+ fig_to_pdf_bytes,
+ mo,
+ np,
+ pl,
+ plt,
+ project_root,
+ recovery_all,
+):
+ """Per-library two-panel figure for the dropdown-selected docking library.
+
+ Left: % recovery vs top-N (95% CI bands) for the two TACTICS methods + two Legacy
+ baselines. Right: Δ recovery (best-TACTICS − each Legacy) at each top-N with planned
+ Welch + Bonferroni-4 stars (docking scoring-type rule). All eight libraries use the
+ canonical full_benchmark_ema_cats recovery + Welch results, so the figure is
+ consistent with the §1 heatmap.
+ """
+ _LIB = dock_library_dropdown.value
+ _TITLE_SUFFIX = {
+ "adenine": "FRED (A2A 4EIY)",
+ "adenine_hybrid": "HYBRID (A2A-Lenselink)",
+ "amide": "FRED (JNK3 2ZDT)",
+ "amide_hybrid": "HYBRID (JNK3 2ZDT)",
+ "quinazoline": "FRED (JNK3 2ZDT)",
+ "quinazoline_hybrid": "HYBRID (JNK3 2ZDT)",
+ "thrombin": "FRED",
+ "thrombin_hybrid": "HYBRID (2ZFF)",
+ }
+ _suffix = _TITLE_SUFFIX.get(_LIB, "")
+
+ _CHALLENGERS = [
+ ("TACTICS Enhanced-TT-TS (GMIC)", "TT-TS", "#E45756", "D"),
+ ("TACTICS Enhanced-RWS (GMIC)", "TACTICS RWS", "#4C78A8", "o"),
+ ]
+ _LEGACIES = [
+ ("Legacy Standard-Greedy", "Legacy Greedy", "#76B7B2", "s", ""),
+ ("Legacy Enhanced-RWS", "Legacy RWS", "#B07AA1", "^", "////"),
+ ]
+
+ # Planned Welch's t-test results — docking-libraries scoring-type rule.
+ _welch_all = pl.read_parquet(
+ project_root / "outputs" / "full_benchmark_ema_cats" / "analysis" / "welch_planned_results.parquet"
+ ).filter(pl.col("library_id") == _LIB)
+
+ _methods_keep = {m for m, _, _, _ in _CHALLENGERS} | {m for m, _, _, _, _ in _LEGACIES}
+ _lib_all = recovery_all.filter(
+ (pl.col("library_id") == _LIB) & pl.col("method").is_in(_methods_keep)
+ )
+ _topn_vals = sorted([_tn for _tn in _lib_all["top_n"].unique().to_list() if _tn <= 250])
+
+ _fig, (_ax1, _ax2) = plt.subplots(1, 2, figsize=(16, 6), facecolor="white")
+ _ax1.set_facecolor("white")
+
+ _agg = (
+ _lib_all.group_by(["method", "top_n"])
+ .agg([
+ pl.col("recovery_frac").mean().alias("mean_frac"),
+ pl.col("recovery_frac").std().alias("std_frac"),
+ pl.col("recovery_frac").count().alias("n"),
+ ])
+ .with_columns((pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac"))
+ .sort(["method", "top_n"])
+ )
+
+ for _meth, _lbl, _clr, _mkr, _ in _LEGACIES:
+ _md = _agg.filter(pl.col("method") == _meth).sort("top_n")
+ if len(_md) == 0:
+ continue
+ _x = _md["top_n"].to_numpy()
+ _y = _md["mean_frac"].to_numpy() * 100
+ _se = _md["se_frac"].to_numpy() * 100
+ _ax1.plot(_x, _y, marker=_mkr, color=_clr, linewidth=2, markersize=7, label=_lbl, zorder=3)
+ _ax1.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se, alpha=0.15, color=_clr, zorder=2)
+
+ for _meth, _lbl, _clr, _mkr in _CHALLENGERS:
+ _md = _agg.filter(pl.col("method") == _meth).sort("top_n")
+ if len(_md) == 0:
+ continue
+ _x = _md["top_n"].to_numpy()
+ _y = _md["mean_frac"].to_numpy() * 100
+ _se = _md["se_frac"].to_numpy() * 100
+ _ax1.plot(_x, _y, marker=_mkr, color=_clr, linewidth=2, markersize=7, label=_lbl, zorder=4)
+ _ax1.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se, alpha=0.15, color=_clr, zorder=2)
+
+ _ax1.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax1.set_ylabel("% Recovery", fontsize=14, color="black")
+ _ax1.set_title(f"{_LIB} {_suffix} — Recovery vs Top-N", fontsize=14, fontweight="bold", color="black")
+ _ax1.set_xticks(_topn_vals)
+ _ax1.tick_params(colors="black", labelsize=12)
+ _ax1.grid(alpha=0.3, color="grey", zorder=1)
+ _ax1.legend(fontsize=11, facecolor="white", edgecolor="black", labelcolor="black")
+ for _sp in _ax1.spines.values():
+ _sp.set_color("black")
+
+ # Best-TACTICS Δ vs each legacy: 2 bars per top-N. At each threshold the
+ # best-performing TACTICS method (max mean recovery) is selected and its Δ vs
+ # each legacy is plotted, with planned Welch + Bonferroni-4 significance stars.
+ _BAR_COLORS = ["#F28E2B", "#4E79A7"] # orange (vs Greedy), steel blue (vs Legacy RWS)
+ _ax2.set_facecolor("white")
+ _bar_width = 0.35
+ _x_pos = np.arange(len(_topn_vals))
+
+ for _gi, (_leg_meth, _leg_lbl, _, _, _) in enumerate(_LEGACIES):
+ _diffs = []
+ _sigs = []
+ for _tn in _topn_vals:
+ _ch_fracs = {}
+ for _ch_meth, _, _, _ in _CHALLENGERS:
+ _f = _lib_all.filter((pl.col("method") == _ch_meth) & (pl.col("top_n") == _tn))["recovery_frac"].mean()
+ _ch_fracs[_ch_meth] = _f if _f is not None else float("-inf")
+ _best_meth = max(_ch_fracs, key=_ch_fracs.get)
+ _best_frac = _ch_fracs[_best_meth]
+ _leg_frac = _lib_all.filter((pl.col("method") == _leg_meth) & (pl.col("top_n") == _tn))["recovery_frac"].mean()
+ _diffs.append(float((_best_frac - _leg_frac) * 100) if _best_frac != float("-inf") and _leg_frac is not None else 0)
+ _wrow = _welch_all.filter(
+ (pl.col("top_n") == _tn)
+ & (pl.col("challenger") == _best_meth)
+ & (pl.col("baseline") == _leg_meth)
+ )
+ _sigs.append(_wrow["significance_bonf"][0] if len(_wrow) > 0 else "")
+
+ _diffs = np.array(_diffs)
+ _offset = (_gi - 0.5) * _bar_width
+ _ax2.bar(_x_pos + _offset, _diffs, width=_bar_width, color=_BAR_COLORS[_gi],
+ edgecolor="black", linewidth=0.5, zorder=3, label=f"vs {_leg_lbl}")
+
+ _dmax = max(abs(_diffs)) if len(_diffs) > 0 else 1
+ _star_pad = _dmax * 0.03
+ for _i, (_d, _s) in enumerate(zip(_diffs, _sigs)):
+ if _s and _s != "ns":
+ _yp = _d + _star_pad if _d >= 0 else _d - _star_pad
+ _ax2.text(_x_pos[_i] + _offset, _yp, _s, ha="center",
+ va="bottom" if _d >= 0 else "top",
+ fontsize=9, fontweight="bold", color=WELCH_STAR_COLOR,
+ rotation=90)
+
+ _ax2.axhline(0, color="black", linewidth=0.8, zorder=2)
+ _yl, _yh = _ax2.get_ylim()
+ _ax2.set_ylim(_yl - abs(_yh - _yl) * 0.10, _yh + abs(_yh - _yl) * 0.30)
+ _ax2.set_xticks(_x_pos)
+ _ax2.set_xticklabels([str(t) for t in _topn_vals], fontsize=12, color="black")
+ _ax2.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax2.set_ylabel("Δ % Recovery (Best TACTICS − Legacy)", fontsize=14, color="black")
+ _ax2.set_title(f"{_LIB} {_suffix} — Best TACTICS Advantage", fontsize=14, fontweight="bold", color="black")
+ _ax2.tick_params(colors="black", labelsize=12)
+ _ax2.grid(axis="y", alpha=0.3, color="grey", zorder=1)
+ _ax2.legend(fontsize=11, facecolor="white", edgecolor="black", labelcolor="black")
+ for _sp in _ax2.spines.values():
+ _sp.set_color("black")
+ _fig.tight_layout()
+ mo.vstack([
+ _fig,
+ mo.download(data=fig_to_pdf_bytes(_fig),
+ filename=f"{_LIB}_topn_tactics_vs_legacy.pdf".replace(" ", "_"),
+ mimetype="application/pdf", label="Download PDF"),
+ ])
+ return
+
+
+@app.cell
+def _(dock_library_dropdown, mo):
+ """Per-library interpretation note (reactive to the §2 dropdown)."""
+ _NOTES = {
+ "adenine": (
+ "**Adenine FRED** — hidden-hub landscape (amidine_003 = 68/100 top-100 "
+ "hits, mean-uninformative)."
+ ),
+ "adenine_hybrid": (
+ "**Adenine HYBRID (A2A-Lenselink)** — cross-docking-pipeline control "
+ "(same chemistry, different scoring). TT-TS dominant; TACTICS RWS bimodal "
+ "(4/20 reps fail on aldehyde_137 GMIC under-rotation). Planned Welch is used "
+ "because Tukey HSD's pooled MSE is contaminated by the bimodal method."
+ ),
+ "quinazoline": (
+ "**Quinazoline FRED (JNK3 2ZDT)** — mean-reachable / near-saturated "
+ "landscape; all five methods within ~2 pp. Canonical EMA-CATS data shown "
+ "(within-noise of the pre-fix run, p = 0.21)."
+ ),
+ "quinazoline_hybrid": (
+ "**Quinazoline HYBRID (JNK3 2ZDT)** — documented null case; all methods "
+ "within ~0.1 pp at top-100 (three-axis framework predicts no shift)."
+ ),
+ "amide": (
+ "**Amide FRED (JNK3 2ZDT)** — broadly-good, low-σ acids; greedy-reachable, "
+ "all methods converge near the ceiling (TACTICS still ≥ Legacy)."
+ ),
+ "amide_hybrid": (
+ "**Amide HYBRID (JNK3 2ZDT)** — pure score-function contrast; axis-1 "
+ "hidden-hub size on the acid component rises (~10% → ~31%), motivating the "
+ "TT-TS advantage."
+ ),
+ "thrombin": (
+ "**Thrombin FRED** — imbalanced (130 × 3844) mean-separable docking; the "
+ "architectural floor recovers Legacy RWS's damage, GMIC adds a modest gain."
+ ),
+ "thrombin_hybrid": (
+ "**Thrombin HYBRID (2ZFF)** — same chemistry, different scoring; axis-1 "
+ "hidden-hub size rises (~8% → ~25%), shifting the ranking toward TT-TS."
+ ),
+ }
+ _lib = dock_library_dropdown.value
+ mo.md(_NOTES.get(_lib, f"**{_lib}** — best-TACTICS vs Legacy across top-N."))
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 3 — Overall Docking Recovery (FRED + HYBRID)
+
+ Two parallel aggregate panels, one per scoring pipeline. NOT pooled across
+ pipelines (would mix structurally-different SAR landscapes — see manuscript
+ Discussion §Cross-docking-pipeline framework validation).
+ """)
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, pl, plt, recovery_all):
+ """Mean % recovery vs top-N for FRED docking libraries (2-comp and 3-comp panels).
+
+ FRED-only aggregate. HYBRID counterpart is the next cell. Two separate panels
+ avoid averaging across scoring pipelines (which would mix structurally-different
+ SAR landscapes).
+ """
+
+ _DOCK_LIBS = {"adenine", "amide", "quinazoline", "thrombin"}
+ # n_components is 0 for thrombin/quinazoline in the data — fix with known values
+ _DOCK_NCOMP = {
+ "thrombin": 2, "thrombin_hybrid": 2,
+ "amide": 2, "amide_hybrid": 2,
+ "adenine": 3, "adenine_hybrid": 3,
+ "quinazoline": 3, "quinazoline_hybrid": 3,
+ }
+
+ _dock_all = (
+ recovery_all
+ .filter(pl.col("library_id").is_in(_DOCK_LIBS))
+ .with_columns(
+ pl.col("library_id").replace(_DOCK_NCOMP).cast(pl.Int64).alias("n_components")
+ )
+ )
+
+ mo.stop(len(_dock_all) == 0, mo.md("No docking library data available."))
+
+ _method_colors = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "#E45756",
+ "TACTICS Enhanced-RWS (GMIC)": "#4C78A8",
+ "Legacy Standard-Greedy": "#76B7B2",
+ "Legacy Enhanced-RWS": "#B07AA1",
+ }
+ _method_markers = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "D",
+ "TACTICS Enhanced-RWS (GMIC)": "o",
+ "Legacy Standard-Greedy": "s",
+ "Legacy Enhanced-RWS": "^",
+ }
+ _method_order = [
+ "TACTICS Enhanced-TT-TS (GMIC)",
+ "TACTICS Enhanced-RWS (GMIC)",
+ "Legacy Standard-Greedy",
+ "Legacy Enhanced-RWS",
+ ]
+
+ _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white",
+ sharey=True)
+
+ for _panel_idx, _nc in enumerate([2, 3]):
+ _ax = _axes[_panel_idx]
+ _ax.set_facecolor("white")
+
+ _comp_data = _dock_all.filter(pl.col("n_components") == _nc)
+
+ # Aggregate: mean and SE of recovery fraction across libraries and replicates
+ _agg = (
+ _comp_data.group_by(["method", "top_n"])
+ .agg([
+ pl.col("recovery_frac").mean().alias("mean_frac"),
+ pl.col("recovery_frac").std().alias("std_frac"),
+ pl.col("recovery_frac").count().alias("n"),
+ ])
+ .with_columns(
+ (pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac")
+ )
+ .sort(["method", "top_n"])
+ )
+
+ for _method in _method_order:
+ _mdata = _agg.filter(pl.col("method") == _method).sort("top_n")
+ if len(_mdata) == 0:
+ continue
+ _x = _mdata["top_n"].to_numpy()
+ _y = _mdata["mean_frac"].to_numpy() * 100
+ _se = _mdata["se_frac"].to_numpy() * 100
+ _color = _method_colors.get(_method, "#999999")
+ _marker = _method_markers.get(_method, "o")
+
+ _ax.plot(_x, _y, marker=_marker, color=_color, linewidth=2,
+ markersize=7, label=_method, zorder=3)
+ _ax.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se,
+ alpha=0.15, color=_color, zorder=2)
+
+ _libs_in_panel = sorted(
+ _comp_data["library_id"].unique().to_list()
+ )
+ _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax.set_title(
+ f"{_nc}-Component Docking Libraries\n({', '.join(_libs_in_panel)})",
+ fontsize=15, fontweight="bold", color="black",
+ )
+ _ax.set_xticks(sorted(_agg["top_n"].unique().to_list()))
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(axis="both", alpha=0.3, color="grey", zorder=1)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+
+ _axes[0].set_ylabel("% Recovery", fontsize=14, color="black")
+
+ # Legend outside the figure below the panels
+ _handles, _labels = _axes[0].get_legend_handles_labels()
+ _fig.legend(
+ _handles, _labels,
+ loc="lower center", ncol=3, fontsize=12,
+ facecolor="white", edgecolor="black", labelcolor="black",
+ bbox_to_anchor=(0.5, -0.08),
+ )
+
+ _fig.suptitle("Mean % Recovery vs Top-N Threshold (FRED Docking Libraries)",
+ fontsize=15, fontweight="bold", color="black", y=1.01)
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="docking_recovery_vs_topn_FRED.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, pl, plt, recovery_all):
+ """Mean % recovery vs top-N for HYBRID docking libraries (2-comp and 3-comp panels).
+
+ HYBRID-only aggregate parallel to the FRED panel above. Pool: 4 HYBRID docking
+ libraries (adenine_hybrid, amide_hybrid, quinazoline_hybrid, thrombin_hybrid).
+ """
+
+ _DOCK_LIBS = {"adenine_hybrid", "amide_hybrid", "quinazoline_hybrid", "thrombin_hybrid"}
+ _DOCK_NCOMP = {
+ "thrombin_hybrid": 2, "amide_hybrid": 2,
+ "adenine_hybrid": 3, "quinazoline_hybrid": 3,
+ }
+
+ _dock_all = (
+ recovery_all
+ .filter(pl.col("library_id").is_in(_DOCK_LIBS))
+ .with_columns(
+ pl.col("library_id").replace(_DOCK_NCOMP).cast(pl.Int64).alias("n_components")
+ )
+ )
+
+ mo.stop(len(_dock_all) == 0, mo.md("No HYBRID docking library data available."))
+
+ _method_colors = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "#E45756",
+ "TACTICS Enhanced-RWS (GMIC)": "#4C78A8",
+ "Legacy Standard-Greedy": "#76B7B2",
+ "Legacy Enhanced-RWS": "#B07AA1",
+ }
+ _method_markers = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "D",
+ "TACTICS Enhanced-RWS (GMIC)": "o",
+ "Legacy Standard-Greedy": "s",
+ "Legacy Enhanced-RWS": "^",
+ }
+ _method_order = [
+ "TACTICS Enhanced-TT-TS (GMIC)",
+ "TACTICS Enhanced-RWS (GMIC)",
+ "Legacy Standard-Greedy",
+ "Legacy Enhanced-RWS",
+ ]
+
+ _fig, _axes = plt.subplots(1, 2, figsize=(16, 6), facecolor="white", sharey=True)
+
+ for _panel_idx, _nc in enumerate([2, 3]):
+ _ax = _axes[_panel_idx]
+ _ax.set_facecolor("white")
+ _comp_data = _dock_all.filter(pl.col("n_components") == _nc)
+ _agg = (
+ _comp_data.group_by(["method", "top_n"])
+ .agg([
+ pl.col("recovery_frac").mean().alias("mean_frac"),
+ pl.col("recovery_frac").std().alias("std_frac"),
+ pl.col("recovery_frac").count().alias("n"),
+ ])
+ .with_columns((pl.col("std_frac") / pl.col("n").sqrt()).alias("se_frac"))
+ .sort(["method", "top_n"])
+ )
+
+ for _method in _method_order:
+ _mdata = _agg.filter(pl.col("method") == _method).sort("top_n")
+ if len(_mdata) == 0:
+ continue
+ _x = _mdata["top_n"].to_numpy()
+ _y = _mdata["mean_frac"].to_numpy() * 100
+ _se = _mdata["se_frac"].to_numpy() * 100
+ _color = _method_colors.get(_method, "#999999")
+ _marker = _method_markers.get(_method, "o")
+ _ax.plot(_x, _y, marker=_marker, color=_color, linewidth=2,
+ markersize=7, label=_method, zorder=3)
+ _ax.fill_between(_x, _y - 1.96 * _se, _y + 1.96 * _se,
+ alpha=0.15, color=_color, zorder=2)
+
+ _libs_in_panel = sorted(_comp_data["library_id"].unique().to_list())
+ _ax.set_xlabel("Top-N Threshold", fontsize=14, color="black")
+ _ax.set_title(
+ f"{_nc}-Component HYBRID Docking Libraries\n({', '.join(_libs_in_panel)})",
+ fontsize=15, fontweight="bold", color="black",
+ )
+ _ax.set_xticks(sorted(_agg["top_n"].unique().to_list()))
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(axis="both", alpha=0.3, color="grey", zorder=1)
+ for _spine in _ax.spines.values():
+ _spine.set_color("black")
+
+ _axes[0].set_ylabel("% Recovery", fontsize=14, color="black")
+ _handles, _labels = _axes[0].get_legend_handles_labels()
+ _fig.legend(_handles, _labels, loc="lower center", ncol=3, fontsize=12,
+ facecolor="white", edgecolor="black", labelcolor="black",
+ bbox_to_anchor=(0.5, -0.08))
+ _fig.suptitle("Mean % Recovery vs Top-N Threshold (HYBRID Docking Libraries)",
+ fontsize=15, fontweight="bold", color="black", y=1.01)
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="docking_recovery_vs_topn_HYBRID.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 4 — Statistical-Test Convention Reminder
+ """)
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ ## Statistical-test convention reminder
+
+ Significance stars in the per-library figures follow the **scoring-type rule**:
+
+ - **ROCS libraries** (20 libraries × 109 queries × 20 replicates): repeated-measures
+ ANOVA + **Tukey HSD** per Ash et al. 2025. Stars rendered in **black**.
+ - **Docking libraries** (8 libraries × 1 query × 20 replicates: lacks
+ repeated-measures structure, single-query × 20 reps reduces to one-way ANOVA where
+ homoscedasticity is required but empirically violated by bimodal methods): planned
+ **Welch's t-test + Bonferroni-4** on the 4 pre-specified TACTICS-vs-Legacy pairs.
+ Stars rendered in **blue (#4477AA)**.
+
+ Both tests are reported in the analysis parquets; the per-library figures use the
+ one appropriate to the library's scoring type. Cross-test agreement on the planned
+ pairs is in `outputs/full_benchmark_ema_cats/analysis/welch_planned_results.parquet`
+ and `tukey_hsd_results.parquet`.
+ """)
+ return
+
+
+@app.cell
+def _(mo):
+ mo.md(r"""
+ # Section 5 — Budget Sensitivity Analysis (1% → 5%) — DOCK
+
+ Recovery and TACTICS-vs-Legacy gap as a function of library-fraction screening
+ budget for the 8 docking libraries, at the canonical top-100 threshold.
+ Demonstrates that TACTICS dominance is preserved across budgets and is largest
+ at constrained-budget regimes (1–2%). The parallel ROCS sensitivity panel
+ lives in `manuscript_plots_ROCS.py`.
+
+ 2×2 panel grid:
+ - **Top row** (line plots): per-method mean recovery vs budget fraction.
+ - **Bottom row** (stacked bars): Δ(best-TACTICS − best-Legacy) per budget,
+ **decomposed into per-library contributions** (each library's Δ ÷ n_libraries,
+ so the stacked segments sum to the bin mean — annotated on top). Color = base
+ chemistry, summing each chemistry's FRED + HYBRID scorings into one segment.
+ This exposes that the 3-component bin's advantage is carried by **adenine** while
+ the two **quinazoline** variants contribute ≈0 — the source of the
+ counter-intuitive 2-comp ≥ 3-comp ordering at most budgets.
+ - **Left column**: 2-component libraries. **Right column**: 3-component libraries.
+
+ Recovery is averaged per query × replicate within each library, then aggregated
+ across libraries within each (n_comp) cell.
+ """)
+ return
+
+
+@app.cell
+def _(pl, project_root):
+ """Load top-100 recovery from all 5 budget benchmarks into a single dataframe."""
+ _budgets = [
+ (1, project_root / "outputs/full_benchmark_1pct/analysis/recovery_summary.parquet"),
+ (2, project_root / "outputs/full_benchmark_ema_cats/analysis/recovery_summary.parquet"),
+ (3, project_root / "outputs/full_benchmark_3pct/analysis/recovery_summary.parquet"),
+ (4, project_root / "outputs/full_benchmark_4pct/analysis/recovery_summary.parquet"),
+ (5, project_root / "outputs/full_benchmark_5pct/analysis/recovery_summary.parquet"),
+ ]
+ _frames = []
+ for _budget, _path in _budgets:
+ if not _path.exists():
+ continue
+ _sub = (
+ pl.read_parquet(_path)
+ .filter(pl.col("top_n") == 100)
+ .with_columns(pl.lit(_budget).alias("budget_pct"))
+ )
+ _frames.append(_sub)
+ sensitivity_data = pl.concat(_frames)
+ return (sensitivity_data,)
+
+
+@app.cell
+def _(pl, sensitivity_data):
+ """Compute per-(library, method, budget) mean recovery + per-(n_comp, method, budget) aggregates.
+
+ Pipeline:
+ 1. Per (library, query, replicate, method, budget) → already in recovery_summary
+ 2. Mean over (query, replicate) → per (library, method, budget) library-level mean
+ 3. Mean across libraries in each (n_comp, method, budget) cell → aggregate mean + std
+ 4. Per-library Δ = best-T(lib) − best-L(lib); aggregate Δ mean + std across libs
+
+ Returns aggregate dataframes ready for plotting.
+ """
+ _METHOD_SHORT = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "TT-TS",
+ "TACTICS Enhanced-RWS (GMIC)": "T-RWS",
+ "Legacy Enhanced-RWS": "L-RWS",
+ "Legacy Standard-Greedy": "L-Grdy",
+ }
+ _ROCS_NCOMP = {
+ "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
+ "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
+ "amide-suzuki": 3, "betti": 3, "dobener": 3, "groebke-blackburn-bienayme": 3,
+ "mannich": 3, "niementowski": 3, "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
+ }
+ _DOCK_NCOMP = {
+ "thrombin": 2, "thrombin_hybrid": 2, "amide": 2, "amide_hybrid": 2,
+ "adenine": 3, "adenine_hybrid": 3, "quinazoline": 3, "quinazoline_hybrid": 3,
+ }
+
+ # Add kind + n_comp to the data
+ _ncomp_map = {**_ROCS_NCOMP, **_DOCK_NCOMP}
+ _kind_map = {lib: "ROCS" for lib in _ROCS_NCOMP}
+ _kind_map.update({lib: "DOCK" for lib in _DOCK_NCOMP})
+
+ _df = (
+ sensitivity_data
+ .with_columns(pl.col("method").replace_strict(_METHOD_SHORT).alias("m"))
+ .filter(pl.col("library_id").is_in(set(_ncomp_map.keys())))
+ .with_columns([
+ pl.col("library_id").replace(_ncomp_map).cast(pl.Int64).alias("n_comp"),
+ pl.col("library_id").replace(_kind_map).alias("kind"),
+ ])
+ )
+
+ # Step 1: per-library mean across (query, replicate)
+ sens_per_lib = (
+ _df.group_by(["library_id", "kind", "n_comp", "m", "budget_pct"])
+ .agg(pl.col("recovery_frac").mean().alias("rec"))
+ )
+
+ # Step 2: per-method aggregate across libraries within each (kind, n_comp, budget)
+ sens_per_method = (
+ sens_per_lib.group_by(["kind", "n_comp", "m", "budget_pct"])
+ .agg([
+ pl.col("rec").mean().alias("mean_rec"),
+ pl.col("rec").std().alias("std_rec"),
+ pl.col("rec").count().alias("n_libs"),
+ ])
+ .with_columns(
+ (pl.col("std_rec") / pl.col("n_libs").sqrt()).alias("se_rec")
+ )
+ )
+
+ # Step 3: per-library Δ = best-T − best-L
+ _wide = (
+ sens_per_lib.pivot(
+ values="rec", index=["library_id", "kind", "n_comp", "budget_pct"], on="m"
+ )
+ )
+ _delta_per_lib = _wide.with_columns([
+ pl.max_horizontal(["TT-TS", "T-RWS"]).alias("best_T"),
+ pl.max_horizontal(["L-RWS", "L-Grdy"]).alias("best_L"),
+ ]).with_columns(
+ ((pl.col("best_T") - pl.col("best_L")) * 100).alias("delta_pp")
+ )
+ sens_delta_per_lib = _delta_per_lib
+ return sens_delta_per_lib, sens_per_method
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, np, pl, plt, sens_delta_per_lib, sens_per_method):
+ """DOCK sensitivity figure: 2×2 layout (line top, stacked-Δ bars bottom; 2-comp left, 3-comp right).
+
+ Bottom row decomposes each bin's mean Δ (best-TACTICS − best-Legacy) into per-library
+ contributions, colored by base chemistry (FRED + HYBRID for a chemistry are summed into
+ one segment); segments sum to the bin mean (annotated). Makes explicit that the 3-comp
+ bin's advantage is carried by adenine while the quinazolines contribute ~0.
+ """
+ _METHOD_ORDER = ["TT-TS", "T-RWS", "L-RWS", "L-Grdy"]
+ _METHOD_LABELS = {
+ "TT-TS": "TACTICS TT-TS (GMIC)",
+ "T-RWS": "TACTICS RWS (GMIC)",
+ "L-RWS": "Legacy Enhanced-RWS",
+ "L-Grdy": "Legacy Standard-Greedy",
+ }
+ _METHOD_COLORS = {
+ "TT-TS": "#E45756", "T-RWS": "#4C78A8",
+ "L-RWS": "#B07AA1", "L-Grdy": "#76B7B2",
+ }
+ _METHOD_MARKERS = {
+ "TT-TS": "D", "T-RWS": "o", "L-RWS": "^", "L-Grdy": "s",
+ }
+ # Bottom-row decomposition: one stacked segment per base chemistry. A chemistry's
+ # FRED + HYBRID contributions are summed into a single colored segment (no hatch).
+ # Colors are the UNUSED members of the same Tableau-10 palette the method lines
+ # draw from (the lines use blue / orange / red / teal / purple; these are the
+ # remaining green / brown / gold / pink). Same visual family as the top row, but
+ # no color is shared with a method, so the bars are never confused with a line.
+ _CHEM_COLOR = {
+ "adenine": "#59A14F", # Tableau green
+ "quinazoline": "#FF9DA7", # Tableau pink
+ "amide": "#9C755F", # Tableau brown
+ "thrombin": "#EDC949", # Tableau gold
+ }
+
+ def _base(_lib):
+ return _lib[:-7] if _lib.endswith("_hybrid") else _lib
+
+ _budgets = sorted(
+ sens_delta_per_lib.filter(pl.col("kind") == "DOCK")["budget_pct"].unique().to_list()
+ )
+
+ _fig, _axes = plt.subplots(2, 2, figsize=(14, 9), facecolor="white", sharex=True)
+
+ for _col_idx, _nc in enumerate([2, 3]):
+ _ax_line = _axes[0, _col_idx]
+ _ax_bar = _axes[1, _col_idx]
+ _ax_line.set_facecolor("white"); _ax_bar.set_facecolor("white")
+
+ _sub_method = sens_per_method.filter(
+ (pl.col("kind") == "DOCK") & (pl.col("n_comp") == _nc)
+ )
+ _n_libs = int(_sub_method["n_libs"][0]) if _sub_method.height > 0 else 0
+ for _m in _METHOD_ORDER:
+ _md = _sub_method.filter(pl.col("m") == _m).sort("budget_pct")
+ if _md.height == 0: continue
+ _x = _md["budget_pct"].to_numpy()
+ _y = _md["mean_rec"].to_numpy() * 100
+ _se = _md["se_rec"].to_numpy() * 100
+ _ax_line.errorbar(
+ _x, _y, yerr=1.96 * _se, color=_METHOD_COLORS[_m],
+ marker=_METHOD_MARKERS[_m], markersize=7, linewidth=2,
+ capsize=4, capthick=1.2, label=_METHOD_LABELS[_m], zorder=3,
+ )
+ _ax_line.set_ylabel("Mean Top-100 Recovery (%)", fontsize=12, color="black")
+ _ax_line.set_title(f"DOCK {_nc}-Component (n = {_n_libs} libraries)",
+ fontsize=13, fontweight="bold", color="black")
+ _ax_line.grid(alpha=0.3, color="grey", zorder=1)
+ _ax_line.tick_params(colors="black", labelsize=11)
+ for _sp in _ax_line.spines.values(): _sp.set_color("black")
+
+ # Bottom row: per-library Δ decomposed by base chemistry (stacked). A
+ # chemistry's FRED + HYBRID contributions are summed into one segment, then
+ # divided by the number of libraries in the bin so the stacked segments sum
+ # to the bin-mean Δ shown by the top-of-stack annotation.
+ _sub_lib = sens_delta_per_lib.filter(
+ (pl.col("kind") == "DOCK") & (pl.col("n_comp") == _nc)
+ )
+ _n_by_b = {_b: _sub_lib.filter(pl.col("budget_pct") == _b).height
+ for _b in _budgets}
+ _by_chem = {}
+ for _r in _sub_lib.iter_rows(named=True):
+ _c = _base(_r["library_id"])
+ _d = _by_chem.setdefault(_c, {})
+ _d[_r["budget_pct"]] = _d.get(_r["budget_pct"], 0.0) + _r["delta_pp"]
+
+ _pos = np.zeros(len(_budgets))
+ _neg = np.zeros(len(_budgets))
+ for _chem in sorted(_by_chem):
+ _heights = np.array([
+ _by_chem[_chem].get(_b, 0.0) / max(_n_by_b[_b], 1) for _b in _budgets
+ ])
+ _bottoms = np.where(_heights >= 0, _pos, _neg)
+ _ax_bar.bar(_budgets, _heights, bottom=_bottoms, width=0.7,
+ color=_CHEM_COLOR[_chem], edgecolor="black", linewidth=0.6,
+ zorder=3, label=_chem)
+ _pos = _pos + np.where(_heights >= 0, _heights, 0.0)
+ _neg = _neg + np.where(_heights < 0, _heights, 0.0)
+
+ # Net (= bin-mean Δ) annotated above each stack's positive top.
+ for _xi, _top, _mean in zip(_budgets, _pos, _pos + _neg):
+ _ax_bar.text(_xi, _top + 0.3, f"{_mean:+.2f}", ha="center", va="bottom",
+ fontsize=10, fontweight="bold", color="black")
+
+ _ax_bar.axhline(0, color="black", linewidth=0.8, zorder=2)
+ _ax_bar.set_xlabel("Library Fraction (%)", fontsize=12, color="black")
+ _ax_bar.set_ylabel("Δ Best-TACTICS − Best-Legacy (pp)\n(stacked by library)",
+ fontsize=12, color="black")
+ _ax_bar.set_xticks([1, 2, 3, 4, 5])
+ _ax_bar.grid(axis="y", alpha=0.3, color="grey", zorder=1)
+ _ax_bar.tick_params(colors="black", labelsize=11)
+ for _sp in _ax_bar.spines.values(): _sp.set_color("black")
+ # No legend title: a one-word "Library" title sits in a box sized to the
+ # widest entry ("quinazoline"), leaving a white gap beside it regardless of
+ # alignment. The entries are clearly library names (the y-axis already says
+ # "stacked by library"), so the title is dropped.
+ _ax_bar.legend(fontsize=10, facecolor="white", edgecolor="black",
+ labelcolor="black", loc="upper right",
+ handlelength=1.2, handletextpad=0.5, borderpad=0.5,
+ labelspacing=0.35)
+ _yl, _yh = _ax_bar.get_ylim()
+ _ax_bar.set_ylim(_yl, _yh + 0.18 * (abs(_yh - _yl) or 1.0))
+
+ _handles, _labels = _axes[0, 0].get_legend_handles_labels()
+ _fig.legend(_handles, _labels, loc="lower center", ncol=5, fontsize=11,
+ facecolor="white", edgecolor="black", labelcolor="black",
+ bbox_to_anchor=(0.5, -0.04))
+ _fig.suptitle("Budget Sensitivity — Docking Libraries (Top-100 Recovery)",
+ fontsize=14, fontweight="bold", color="black", y=1.00)
+ _fig.tight_layout(rect=[0, 0.04, 1, 0.98])
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="sensitivity_dock.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(mo, pl, sens_delta_per_lib):
+ """Summary table — per-library Δ for all 8 docking libraries across 5 budgets.
+
+ Demonstrates the adenine + quinazoline_hybrid pattern extends to the full 8-library
+ docking set: TACTICS is either better (top rows) or statistically indistinguishable
+ (bottom rows) than the best Legacy method at every budget. The "TACTICS is either
+ better or the same" claim holds on 8/8 libraries at every budget tested.
+ """
+ _DOCK_LIBS_ORDERED = [
+ # TACTICS-dominant regime (consistent positive Δ at most budgets)
+ ("adenine", "FRED", 3),
+ ("adenine_hybrid", "HYBRID", 3),
+ ("amide_hybrid", "HYBRID", 2),
+ ("thrombin_hybrid", "HYBRID", 2),
+ ("thrombin", "FRED", 2),
+ # Saturated / tied regime (Δ near zero, all methods at recovery ceiling)
+ ("amide", "FRED", 2),
+ ("quinazoline", "FRED", 3),
+ ("quinazoline_hybrid", "HYBRID", 3),
+ ]
+ _meta = {l: (s, n) for l, s, n in _DOCK_LIBS_ORDERED}
+ _order = [l for l, _, _ in _DOCK_LIBS_ORDERED]
+
+ _rows = []
+ for _lib in _order:
+ _scoring, _ncomp = _meta[_lib]
+ _r = {"library": _lib, "scoring": _scoring, "n_comp": _ncomp}
+ _sub = sens_delta_per_lib.filter(
+ (pl.col("kind") == "DOCK") & (pl.col("library_id") == _lib)
+ )
+ for _b in [1, 2, 3, 4, 5]:
+ _val = _sub.filter(pl.col("budget_pct") == _b)["delta_pp"]
+ _r[f"delta_{_b}pct"] = round(float(_val[0]), 2) if _val.len() > 0 else None
+ _rows.append(_r)
+ sens_dock_table = pl.DataFrame(_rows)
+
+ _csv_bytes = sens_dock_table.write_csv().encode()
+
+ mo.vstack([
+ mo.md(
+ "### Docking library summary — Δ Best-TACTICS − Best-Legacy (pp) across budgets\n\n"
+ "Libraries ordered top-to-bottom by TACTICS dominance at the canonical 2% budget. "
+ "Rows 1-5 = TACTICS-dominant regime (consistent positive Δ); rows 6-8 = saturated "
+ "regime (Δ near zero, all methods at the recovery ceiling). The 'TACTICS is either "
+ "better or the same' claim holds on 8/8 libraries at every budget tested."
+ ),
+ mo.ui.table(sens_dock_table, selection=None),
+ mo.download(data=_csv_bytes, filename="sensitivity_dock_table.csv",
+ mimetype="text/csv", label="Download CSV"),
+ ])
+ return
+
+
+if __name__ == "__main__":
+ app.run()
diff --git a/tutorials/manuscript_sar_plots.py b/tutorials/manuscript_sar_plots.py
new file mode 100644
index 0000000..668a28e
--- /dev/null
+++ b/tutorials/manuscript_sar_plots.py
@@ -0,0 +1,1085 @@
+"""
+TACTICS Manuscript — Reagent Score Landscape Plots (Docking only)
+
+Heavy plots that read multi-million-row brute-force ground-truth parquets and
+compute per-reagent posterior statistics. Separated from manuscript_plots.py
+to avoid blocking iteration on the recovery/diversity plots in that notebook.
+
+Coverage: 8 docking libraries (4 FRED + 4 HYBRID) for both the single-library
+explorer and the multi-top-N grid.
+
+Run as app: marimo run tutorials/manuscript_sar_plots.py
+Edit mode: marimo edit tutorials/manuscript_sar_plots.py
+"""
+
+import marimo
+
+__generated_with = "0.21.1"
+app = marimo.App(
+ width="full",
+ app_title="TACTICS Manuscript — Reagent Score Landscape",
+)
+
+
+@app.cell
+def _():
+ """Imports."""
+ import io
+ import marimo as mo
+ import sys
+ from pathlib import Path
+
+ import matplotlib.pyplot as plt
+ import numpy as np
+ import polars as pl
+
+ def fig_to_pdf_bytes(fig):
+ """Convert a matplotlib figure to PDF bytes in memory.
+
+ Includes any per-axes legends as bbox_extra_artists so legends placed
+ OUTSIDE the axes (e.g. the M-S map's right-gutter legends) are not
+ clipped by the tight bbox.
+ """
+ from matplotlib.legend import Legend as _Legend
+
+ buf = io.BytesIO()
+ # Collect ALL Legend artists (an axes can hold several via add_artist),
+ # plus any figure-level legend.
+ _extra = []
+ for _ax in fig.axes:
+ _extra.extend(_c for _c in _ax.get_children() if isinstance(_c, _Legend))
+ _extra.extend(_lg for _lg in fig.legends if isinstance(_lg, _Legend))
+ fig.savefig(buf, format="pdf", bbox_inches="tight", dpi=300,
+ bbox_extra_artists=_extra or None)
+ return buf.getvalue()
+
+ try:
+ project_root = Path(__file__).parent.parent.resolve()
+ except NameError:
+ project_root = Path.cwd()
+
+ sys.path.insert(0, str(project_root))
+ sys.path.insert(0, str(project_root / "src"))
+ return fig_to_pdf_bytes, mo, np, pl, plt, project_root
+
+
+@app.cell
+def _(mo):
+ """SAR section header + per-library dropdown (docking libraries only)."""
+ _docking_libs = [
+ "adenine", "adenine_hybrid",
+ "amide", "amide_hybrid",
+ "quinazoline", "quinazoline_hybrid",
+ "thrombin", "thrombin_hybrid",
+ ]
+ sar_library_selector = mo.ui.dropdown(
+ options=_docking_libs,
+ value="adenine",
+ label="Library",
+ )
+ sar_topn = mo.ui.slider(start=25, stop=250, step=25, value=100, label="Top-N")
+ mo.vstack([
+ mo.md(r"""
+ # SAR Landscape Characterization — Reagent Posterior Separation
+
+ Riley (2024) showed that for ROCS, reagents producing top compounds have high
+ means and low variance ("stand out from the crowd"), while for docking, top-hit
+ reagents are statistically indistinguishable from mediocre ones (mean-uninformative).
+ Below: per-reagent (z(mean), z(std)) scatter for the selected docking library,
+ colored by hit count in the chosen top-N.
+ """),
+ mo.hstack([sar_library_selector, sar_topn], gap=2),
+ ])
+ return sar_library_selector, sar_topn
+
+
+@app.cell
+def _(
+ fig_to_pdf_bytes,
+ mo,
+ np,
+ pl,
+ plt,
+ project_root,
+ sar_library_selector,
+ sar_topn,
+):
+ """Reagent posterior separation plot (Riley-style) for the selected docking library.
+
+ For each reagent, computes (mean score, std score) across all products containing
+ that reagent. Colors reagents by how many times they appear in top-N compounds.
+ """
+ from matplotlib.colors import Normalize as _Normalize
+ from matplotlib import cm as _cm
+ import re as _re
+
+ _SCORES_DIR = project_root / "data" / "scores"
+ _lib_id = sar_library_selector.value
+ _TOP_N = sar_topn.value
+
+ def _parse_adenine(code):
+ _m = _re.match(r"(amidine_\d+)_(isocyanide_db_\d+)_(aldehyde_\d+)", code)
+ return _m.groups() if _m else None
+
+ def _parse_thrombin(code):
+ _pts = code.split("_")
+ return (_pts[0], _pts[1] + "_" + _pts[2]) if len(_pts) == 3 else None
+
+ def _parse_split(code, n):
+ _pts = code.split("_")
+ return tuple(_pts) if len(_pts) == n else None
+
+ _CONFIGS = {
+ "adenine": {"components": ["Amidines", "Isocyanides", "Aldehydes"],
+ "parse": _parse_adenine},
+ "adenine_hybrid": {"components": ["Amidines", "Isocyanides", "Aldehydes"],
+ "parse": _parse_adenine},
+ "quinazoline": {"components": ["Aminobenzoics", "Amines", "Acids"],
+ "parse": lambda c: _parse_split(c, 3)},
+ "quinazoline_hybrid": {"components": ["Aminobenzoics", "Amines", "Acids"],
+ "parse": lambda c: _parse_split(c, 3)},
+ "thrombin": {"components": ["Acids", "Dipeptides"],
+ "parse": _parse_thrombin},
+ "thrombin_hybrid": {"components": ["Acids", "Dipeptides"],
+ "parse": _parse_thrombin},
+ "amide": {"components": ["Amines", "Acids"],
+ "parse": lambda c: _parse_split(c, 2)},
+ "amide_hybrid": {"components": ["Amines", "Acids"],
+ "parse": lambda c: _parse_split(c, 2)},
+ }
+
+ _cfg = _CONFIGS[_lib_id]
+ _comp_names = _cfg["components"]
+ _n_comp = len(_comp_names)
+ _parse_fn = _cfg["parse"]
+
+ _score_df = pl.read_parquet(_SCORES_DIR / f"{_lib_id}.parquet")
+ _score_df = _score_df.select(["Product_Code", "Scores"]).filter(pl.col("Scores").is_not_nan())
+
+ _codes = _score_df["Product_Code"].to_list()
+ _scores_arr = _score_df["Scores"].to_numpy()
+ _parsed = [_parse_fn(c) for c in _codes]
+
+ _sorted_idx = np.argsort(_scores_arr) # ascending = best for docking minimize
+ _top_codes = [_codes[i] for i in _sorted_idx[:_TOP_N]]
+ _top_parsed = [_parse_fn(c) for c in _top_codes]
+
+ _cmap = _cm.YlOrRd
+ _global_max_count = 0
+ _plot_data = []
+
+ for _ci in range(_n_comp):
+ _reagent_to_idx = {}
+ for _i, _p in enumerate(_parsed):
+ if _p is not None:
+ _reagent_to_idx.setdefault(_p[_ci], []).append(_i)
+
+ _top_counts = {}
+ for _p in _top_parsed:
+ if _p is not None:
+ _top_counts[_p[_ci]] = _top_counts.get(_p[_ci], 0) + 1
+
+ _unique = sorted(_reagent_to_idx.keys())
+ _means = np.array([float(np.mean(_scores_arr[_reagent_to_idx[r]])) for r in _unique])
+ _stds = np.array([float(np.std(_scores_arr[_reagent_to_idx[r]])) for r in _unique])
+ _counts = np.array([_top_counts.get(r, 0) for r in _unique])
+
+ _hc = _counts[_counts > 0]
+ if len(_hc) > 0:
+ _global_max_count = max(_global_max_count, int(_hc.max()))
+
+ _sig = float(np.var(_means))
+ _noise = float(np.mean(_stds**2)) if len(_stds) > 0 else 1.0
+ _gmic = 0.5 * np.log(1 + _sig / _noise) if _noise > 0 else 0.0
+ _plot_data.append((_ci, _means, _stds, _counts, _unique, _gmic))
+
+ _gs = plt.GridSpec(1, _n_comp + 1, width_ratios=[1] * _n_comp + [0.05], wspace=0.35)
+ _fig = plt.figure(figsize=(7 * _n_comp, 6), facecolor="white")
+ _axes = [_fig.add_subplot(_gs[0, c]) for c in range(_n_comp)]
+ _norm = _Normalize(vmin=1, vmax=max(_global_max_count, 2))
+
+ for _ci, _means, _stds, _counts, _unique, _gmic in _plot_data:
+ _ax = _axes[_ci]
+ _ax.set_facecolor("white")
+
+ _m_mu, _m_sig = np.mean(_means), np.std(_means)
+ _s_mu, _s_sig = np.mean(_stds), np.std(_stds)
+ _zm = (_means - _m_mu) / _m_sig if _m_sig > 0 else _means * 0
+ _zs = (_stds - _s_mu) / _s_sig if _s_sig > 0 else _stds * 0
+
+ _zero = _counts == 0
+ _ax.scatter(_zm[_zero], _zs[_zero], c="#CCCCCC", s=40, alpha=0.6,
+ edgecolors="none", zorder=2)
+
+ _hit = _counts > 0
+ if _hit.any():
+ _ax.scatter(_zm[_hit], _zs[_hit], c=_counts[_hit], cmap=_cmap,
+ norm=_norm, s=60, alpha=0.9, edgecolors="black",
+ linewidths=0.5, zorder=3)
+
+ _ax.set_xlabel("Reagent Mean Score (z)", fontsize=14, color="black")
+ if _ci == 0:
+ _ax.set_ylabel("Reagent Stdev (z)", fontsize=14, color="black")
+ _n_hit = int(_hit.sum())
+ _ax.set_title(
+ f"{_comp_names[_ci]} ({_n_hit}/{len(_unique)}) — GMIC {_gmic:.3f}",
+ fontsize=13, fontweight="bold", color="black",
+ )
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(alpha=0.3, color="grey", zorder=1)
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+
+ _cbar_ax = _fig.add_subplot(_gs[0, _n_comp])
+ _cbar = _fig.colorbar(_cm.ScalarMappable(norm=_norm, cmap=_cmap), cax=_cbar_ax)
+ _cbar.set_label(f"Count in top-{_TOP_N}", fontsize=14, color="black")
+ _cbar.ax.tick_params(colors="black", labelsize=12)
+
+ _fig.suptitle(
+ f"{_lib_id.capitalize()} — Reagent Score Distributions vs Top-{_TOP_N} (Docking)",
+ fontsize=16, fontweight="bold", color="black", y=1.02,
+ )
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename=f"reagent_posterior_separation_{_lib_id}.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(mo):
+ """ROCS SAR section header + per-library dropdown + top-N slider."""
+ _rocs_libs = [
+ "rxn101", "rxn102a", "rxn108b", "rxn111b", "rxn114b",
+ "rxn203", "rxn205", "rxn206", "rxn207", "rxn208",
+ "amide-suzuki", "betti", "dobener", "groebke-blackburn-bienayme",
+ "mannich", "niementowski", "orru", "passerini", "petasis", "poparov",
+ ]
+ sar_rocs_library_selector = mo.ui.dropdown(
+ options=_rocs_libs,
+ value="rxn108b",
+ label="ROCS Library",
+ )
+ sar_rocs_topn = mo.ui.slider(
+ start=25, stop=2500, step=25, value=2500, label="Top-N"
+ )
+ mo.vstack([
+ mo.md(r"""
+ # SAR Landscape — ROCS Libraries (Mean Across All Queries)
+
+ For ROCS-similarity libraries, each library has 109 query molecules. The reagent
+ posterior used here is the per-reagent (mean similarity, std similarity)
+ computed by **pooling all (product × query) similarity scores** across every
+ product containing the reagent and every query.
+
+ This mean-SAR view is paired with the per-library mean diversity used in the
+ Zhao Fig 5 analysis — both averaged across queries — to ensure structural
+ consistency between the diversity-vs-recovery story and the per-library SAR
+ characterization.
+
+ Reagents are coloured by **total hit count in top-N across all queries** (sum
+ over the 109 queries of how many times the reagent appears in that query's
+ top-N ground truth). The hit-count coloring serves as a complementary
+ check: if the reagents flagged as hidden hubs or mean-separable by the
+ mean-SAR view are also consistently hit across queries, the library-level
+ aggregation is a representative summary of the actual enriched building
+ blocks.
+ """),
+ mo.hstack([sar_rocs_library_selector, sar_rocs_topn], gap=2),
+ ])
+ return sar_rocs_library_selector, sar_rocs_topn
+
+
+@app.cell
+def _(
+ fig_to_pdf_bytes,
+ mo,
+ np,
+ pl,
+ plt,
+ project_root,
+ sar_rocs_library_selector,
+ sar_rocs_topn,
+):
+ """Mean SAR landscape for the selected ROCS library.
+
+ Per reagent at each component: pool all (product × query) similarity scores
+ across all products containing that reagent and across all 109 queries.
+ Mean and std define the Riley-style (z(mean), z(std)) point. Colour = sum
+ across queries of per-query top-N hit count.
+ """
+ from matplotlib.colors import Normalize as _Normalize
+ from matplotlib import cm as _cm
+
+ _ROCS_NCOMP = {
+ "rxn101": 2, "rxn102a": 2, "rxn108b": 2, "rxn111b": 2, "rxn114b": 2,
+ "rxn203": 2, "rxn205": 2, "rxn206": 2, "rxn207": 2, "rxn208": 2,
+ "amide-suzuki": 3, "betti": 3, "dobener": 3,
+ "groebke-blackburn-bienayme": 3, "mannich": 3, "niementowski": 3,
+ "orru": 3, "passerini": 3, "petasis": 3, "poparov": 3,
+ }
+
+ _SCORES_DIR = project_root / "data" / "scores"
+ _lib_id = sar_rocs_library_selector.value
+ _TOP_N = sar_rocs_topn.value
+ _n_comp = _ROCS_NCOMP[_lib_id]
+ _comp_names = [f"Component {i + 1}" for i in range(_n_comp)]
+
+ _df = pl.read_parquet(_SCORES_DIR / f"{_lib_id}.parquet")
+ _name_col = "Product_Code" if "Product_Code" in _df.columns else "Name"
+ _query_cols = sorted(c for c in _df.columns if c.startswith("query_"))
+
+ _codes = _df[_name_col].to_list()
+ _parsed = []
+ for _c in _codes:
+ _pts = _c.split("_")
+ _parsed.append(tuple(_pts) if len(_pts) == _n_comp else None)
+
+ # Map reagent → list of product row indices, per component
+ _reagent_to_idx = [{} for _ in range(_n_comp)]
+ for _i, _p in enumerate(_parsed):
+ if _p is None:
+ continue
+ for _ci in range(_n_comp):
+ _reagent_to_idx[_ci].setdefault(_p[_ci], []).append(_i)
+
+ # Materialize a (n_products × n_queries) score matrix
+ _qarrs = [_df[_q].to_numpy() for _q in _query_cols]
+ _qstack = np.column_stack(_qarrs) # shape (n_products, n_queries)
+
+ # Per-query top-N hit counts per reagent per component
+ _hit_per_comp = [{} for _ in range(_n_comp)]
+ for _qi in range(len(_query_cols)):
+ _scores = _qstack[:, _qi]
+ _valid = ~np.isnan(_scores)
+ _valid_idx = np.where(_valid)[0]
+ # Descending sort for similarity (higher is better)
+ _sorted_idx = _valid_idx[np.argsort(-_scores[_valid_idx])][:_TOP_N]
+ for _idx in _sorted_idx:
+ _p = _parsed[_idx]
+ if _p is None:
+ continue
+ for _ci in range(_n_comp):
+ _hit_per_comp[_ci][_p[_ci]] = _hit_per_comp[_ci].get(_p[_ci], 0) + 1
+
+ _cmap = _cm.YlOrRd
+ _global_max_count = 0
+ _plot_data = []
+
+ for _ci in range(_n_comp):
+ _unique = sorted(_reagent_to_idx[_ci].keys())
+ _means = np.empty(len(_unique))
+ _stds = np.empty(len(_unique))
+ for _ri, _r in enumerate(_unique):
+ _idx = _reagent_to_idx[_ci][_r]
+ _block = _qstack[_idx, :]
+ _flat = _block[~np.isnan(_block)]
+ if len(_flat) == 0:
+ _means[_ri] = np.nan
+ _stds[_ri] = np.nan
+ else:
+ _means[_ri] = float(np.mean(_flat))
+ _stds[_ri] = float(np.std(_flat))
+ # Mean hits per query (sum across queries / n_queries). Aligns the
+ # coloring convention with the diversity-vs-recovery analysis, which
+ # also uses per-query means.
+ _counts = np.array([_hit_per_comp[_ci].get(r, 0) for r in _unique], dtype=float) / max(len(_query_cols), 1)
+
+ _hc = _counts[_counts > 0]
+ if len(_hc) > 0:
+ _global_max_count = max(_global_max_count, float(_hc.max()))
+
+ _vmask = ~np.isnan(_means) & ~np.isnan(_stds)
+ if _vmask.sum() > 0:
+ _sig = float(np.var(_means[_vmask]))
+ _noise = float(np.mean(_stds[_vmask] ** 2))
+ _gmic = 0.5 * np.log(1 + _sig / _noise) if _noise > 0 else 0.0
+ else:
+ _gmic = 0.0
+ _plot_data.append((_ci, _means, _stds, _counts, _unique, _gmic))
+
+ _gs = plt.GridSpec(1, _n_comp + 1, width_ratios=[1] * _n_comp + [0.05], wspace=0.35)
+ _fig = plt.figure(figsize=(7 * _n_comp, 6), facecolor="white")
+ _axes = [_fig.add_subplot(_gs[0, c]) for c in range(_n_comp)]
+ _norm = _Normalize(vmin=0.0, vmax=max(_global_max_count, 0.1))
+
+ for _ci, _means, _stds, _counts, _unique, _gmic in _plot_data:
+ _ax = _axes[_ci]
+ _ax.set_facecolor("white")
+
+ _vmask = ~np.isnan(_means) & ~np.isnan(_stds)
+ _vmeans = _means[_vmask]
+ _vstds = _stds[_vmask]
+ _vcounts = _counts[_vmask]
+
+ _m_mu, _m_sig = float(np.mean(_vmeans)), float(np.std(_vmeans))
+ _s_mu, _s_sig = float(np.mean(_vstds)), float(np.std(_vstds))
+ _zm = (_vmeans - _m_mu) / _m_sig if _m_sig > 0 else _vmeans * 0
+ _zs = (_vstds - _s_mu) / _s_sig if _s_sig > 0 else _vstds * 0
+
+ _zero = _vcounts == 0
+ _ax.scatter(_zm[_zero], _zs[_zero], c="#CCCCCC", s=40, alpha=0.6,
+ edgecolors="none", zorder=2)
+ _hit = _vcounts > 0
+ if _hit.any():
+ _ax.scatter(_zm[_hit], _zs[_hit], c=_vcounts[_hit], cmap=_cmap,
+ norm=_norm, s=60, alpha=0.9, edgecolors="black",
+ linewidths=0.5, zorder=3)
+
+ _ax.set_xlabel("Reagent Mean Similarity (z)", fontsize=14, color="black")
+ if _ci == 0:
+ _ax.set_ylabel("Reagent Stdev (z)", fontsize=14, color="black")
+ _n_hit = int(_hit.sum())
+ _ax.set_title(
+ f"{_comp_names[_ci]} ({_n_hit}/{int(_vmask.sum())}) — GMIC {_gmic:.3f}",
+ fontsize=13, fontweight="bold", color="black",
+ )
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(alpha=0.3, color="grey", zorder=1)
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+
+ _cbar_ax = _fig.add_subplot(_gs[0, _n_comp])
+ _cbar = _fig.colorbar(_cm.ScalarMappable(norm=_norm, cmap=_cmap), cax=_cbar_ax)
+ _cbar.set_label(
+ f"Mean hits per query in top-{_TOP_N} (averaged across 109 queries)",
+ fontsize=13, color="black",
+ )
+ _cbar.ax.tick_params(colors="black", labelsize=12)
+
+ _fig.suptitle(
+ f"{_lib_id} — Mean SAR Across Queries vs Top-{_TOP_N} Hit Frequency (ROCS)",
+ fontsize=16, fontweight="bold", color="black", y=1.02,
+ )
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename=f"reagent_posterior_separation_rocs_{_lib_id}_top{_TOP_N}.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, np, plt, project_root):
+ """z* sensitivity of the COVERAGE axis — why the reachability reference is 1.5, not 0.9/1.0.
+
+ The coverage coordinate is x = max(0, z* + z(mean)); a library leaves the origin
+ (is flagged "coverage-failing") once z* exceeds its activation threshold t = -z(mean).
+ Two things behave differently as z* moves:
+ - the RANK ORDER of flagged libraries by severity is invariant (severity is monotone
+ in z(mean)) -- this is what stayed put when z* was swept up from 1.0;
+ - the SET of flagged libraries (those off the origin) GROWS with z* -- this is what
+ actually moves points on the landscape.
+ Lowering z* to 0.9 or 1.0 drops adenine_hybrid back onto the "easy" origin even though
+ greedy genuinely fails it (Legacy Greedy ~ 30 vs TACTICS ~ 95 top-100). The window that
+ flags BOTH coverage libraries (adenine, adenine_hybrid) while keeping the mean-separable
+ libraries (quinazoline ...) at the origin is z* in (1.04, 1.67); 1.5 sits inside it.
+
+ NB: x is the COVERAGE coordinate ONLY (one of three failure modes). A high coverage-
+ reachability does NOT imply greedy-easy -- both amide_hybrid (allocation, high-sigma hub)
+ and thrombin (imbalance, 30:1) sit far-right (mean-reachable hubs) yet are greedy-hard via
+ OTHER modes that live on the main landscape (y-axis / edge). Recovery is explained by the
+ COMBINATION of modes, never this one axis. Colour encodes the OPERATIVE mode (red coverage
+ / blue allocation / orange imbalance / grey none), computed from the three severities -- so
+ a non-grey point far-right is coherent (coverage-reachable, hard via another mode).
+ Source: dominant-hub z(mean) + greedy/TACTICS top-100 recovery, landscape probe JSONs
+ (experiments/_debug/docking_landscape_probe.py).
+ """
+ import json as _json
+ from matplotlib.lines import Line2D as _Line2D
+ from matplotlib.patches import Patch as _Patch
+
+ _PROBE_DIR = project_root / "outputs" / "debug" / "docking_landscape_probe"
+ _MAP_LIBS = [
+ "adenine", "adenine_hybrid", "quinazoline", "quinazoline_hybrid",
+ "thrombin", "thrombin_hybrid", "amide", "amide_hybrid",
+ ]
+ _TT = "TACTICS Enhanced-TT-TS (GMIC)"
+ _RWS = "TACTICS Enhanced-RWS (GMIC)"
+ _LGRD = "Legacy Standard-Greedy"
+ _Z_CANDIDATES = [0.9, 1.0, 1.5]
+
+ _rows = []
+ for _lab in _MAP_LIBS:
+ with open(_PROBE_DIR / f"{_lab}.json") as _fh:
+ _j = _json.load(_fh)
+ _h = _j["dominant_hub"]
+ _zm = float(_h["z_mean"])
+ _m2 = _j["M2_concentration_by_topn"]["100"][f"comp_{_h['component_idx']}"]
+ _counts = list(_j["n_reagents"].values())
+ _pm = _j["M6_search"]["per_method"]
+ _grd = _pm[_LGRD]["topn_recovered_mean"]
+ _tac = max(_pm[_TT]["topn_recovered_mean"], _pm[_RWS]["topn_recovered_mean"])
+ _rows.append({
+ "lab": _lab, "zmean": _zm, "thr": -_zm,
+ "grd": _grd, "tac": _tac, "delta": _tac - _grd,
+ # the three landscape severities (same computation as the landscape cell)
+ "cov": max(0.0, 1.5 + _zm),
+ "alloc": _h["z_std"] * (1.0 - _m2["top1_share"]),
+ "imb": max(_counts) / min(_counts),
+ })
+ _rows.sort(key=lambda _r: _r["thr"]) # ascending activation threshold == descending z(mean)
+
+ # Rank-order invariance: the off-origin libraries at any z* are a prefix of the same
+ # z(mean)-sorted master list -- their relative order never inverts.
+ _master = [_r["lab"] for _r in _rows]
+
+ def _flagged_prefix(_zs):
+ return [_r["lab"] for _r in _rows if _zs + _r["zmean"] > 0]
+
+ _rank_invariant = all(
+ _flagged_prefix(_zs) == _master[:len(_flagged_prefix(_zs))]
+ for _zs in [1.0, 1.25, 1.5, 1.75, 2.0]
+ )
+
+ # Colour = OPERATIVE failure mode, computed from the three landscape severities (NOT
+ # from the recovery magnitude -- that mis-binned thrombin, a 30:1 imbalance library, as
+ # "competitive" merely because its greedy gap is modest). Priority coverage > allocation
+ # > imbalance picks the dominant mode when several co-occur (e.g. adenine is coverage AND
+ # 24:1 imbalanced -> coverage). Thresholds are display choices; the continuous severities
+ # live on the main landscape. Palette matches the landscape: coverage=red, allocation=
+ # blue, imbalance=orange.
+ _COV_T, _ALLOC_T, _IMB_T = 0.30, 4.0, 15.0
+ _C_COV, _C_ALLOC, _C_IMB, _C_EASY = "#B23030", "#2A5DA0", "#E8820E", "#888888"
+
+ def _mech(_r):
+ if _r["cov"] > _COV_T:
+ return "coverage"
+ if _r["alloc"] > _ALLOC_T:
+ return "allocation"
+ if _r["imb"] > _IMB_T:
+ return "imbalance"
+ return "easy"
+
+ _MECH_COLOR = {"coverage": _C_COV, "allocation": _C_ALLOC,
+ "imbalance": _C_IMB, "easy": _C_EASY}
+
+ def _color(_r):
+ return _MECH_COLOR[_mech(_r)]
+
+ # valid z* window: flag the coverage libraries but not the no-failure (easy) ones
+ _win_lo = max(_r["thr"] for _r in _rows if _mech(_r) == "coverage") # 1.04 (adenine_hybrid)
+ _win_hi = min(_r["thr"] for _r in _rows if _mech(_r) == "easy") # 1.67 (quinazoline)
+
+ _fig, _ax = plt.subplots(figsize=(11.5, 6.2), facecolor="white")
+ _ax.set_facecolor("white")
+ for _i, _r in enumerate(_rows):
+ _c = _color(_r)
+ _ax.plot([0, _r["thr"]], [_i, _i], color=_c, lw=2.0, alpha=0.55, zorder=2)
+ _ax.scatter(_r["thr"], _i, s=130, color=_c, edgecolors="black",
+ linewidths=0.8, zorder=3)
+ _ax.annotate(f'{_r["lab"]} (greedy {_r["grd"]:.0f} -> TACTICS {_r["tac"]:.0f})',
+ (_r["thr"], _i), xytext=(8, 0), textcoords="offset points",
+ va="center", ha="left", fontsize=9.5, color="black", zorder=4)
+
+ _ax.axvspan(_win_lo, _win_hi, color="#9CC16A", alpha=0.20, zorder=0)
+ _ax.annotate(f"valid window\nz* in ({_win_lo:.2f}, {_win_hi:.2f})",
+ ((_win_lo + _win_hi) / 2.0, len(_rows) - 0.4), ha="center", va="top",
+ fontsize=9.5, color="#2F6B2F", style="italic", zorder=1)
+
+ # stagger the 0.9/1.0 labels vertically (they are only 0.1 apart on the x-axis)
+ _zlab_y = {0.9: -0.65, 1.0: -1.15, 1.5: -0.65}
+ for _zs in _Z_CANDIDATES:
+ _in_win = _win_lo < _zs < _win_hi
+ _ax.axvline(_zs, ls="--", lw=1.6,
+ color=("#2F6B2F" if _in_win else "#444444"), alpha=0.9, zorder=1)
+ _ax.annotate(f"z*={_zs}", (_zs, _zlab_y.get(_zs, -0.65)), ha="center", va="top",
+ fontsize=10, color=("#2F6B2F" if _in_win else "#444444"),
+ fontweight="bold")
+
+ _ax.set_yticks(np.arange(len(_rows)))
+ _ax.set_yticklabels([])
+ _ax.set_ylim(-1.6, len(_rows) - 0.2)
+ _ax.set_xlim(0, max(_r["thr"] for _r in _rows) * 1.15)
+ _ax.set_xlabel("activation threshold t = -z(mean) of dominant hub "
+ "(library flagged coverage-failing once z* > t)",
+ fontsize=11.5, color="black")
+ _ax.set_title("Coverage-axis z* sensitivity - a library leaves the origin when z* > -z(mean)\n"
+ "0.9 / 1.0 exclude adenine_hybrid (a real greedy-failure); 1.5 is in the valid window",
+ fontsize=12.5, fontweight="bold", color="black")
+ _ax.tick_params(colors="black", labelsize=11)
+ _ax.grid(axis="x", alpha=0.3, color="grey")
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+
+ # Legend: colour = OPERATIVE failure mode (red coverage / blue allocation / orange
+ # imbalance / grey none). A far-right point that is NOT grey is coherent: its hub is
+ # coverage-reachable, but the library is hard via allocation (amide_hybrid) or imbalance
+ # (thrombin). Only x (coverage) is on this axis; allocation/imbalance live on the landscape.
+ _leg_handles = [
+ _Line2D([0], [0], marker="o", ls="", ms=11, mfc=_C_COV, mec="black",
+ label="coverage failure — mean-blind hub (this axis)"),
+ _Line2D([0], [0], marker="o", ls="", ms=11, mfc=_C_ALLOC, mec="black",
+ label="allocation — partner-specific / high-σ hub (landscape y)"),
+ _Line2D([0], [0], marker="o", ls="", ms=11, mfc=_C_IMB, mec="black",
+ label="imbalance — under-observed large component (landscape edge)"),
+ _Line2D([0], [0], marker="o", ls="", ms=11, mfc=_C_EASY, mec="black",
+ label="no failure mode — greedy-competitive"),
+ _Patch(facecolor="#9CC16A", alpha=0.45, edgecolor="none",
+ label=f"valid z* window ({_win_lo:.2f}–{_win_hi:.2f})"),
+ _Line2D([0], [0], ls="--", lw=1.6, color="#444444", label="candidate z* cut"),
+ ]
+ _ax.legend(handles=_leg_handles, loc="lower right", fontsize=8.5, framealpha=0.96,
+ edgecolor="black", borderpad=0.7, labelspacing=0.6)
+ _fig.tight_layout()
+
+ _flag_09 = ", ".join(_r["lab"] for _r in _rows if 0.9 + _r["zmean"] > 0) or "(none)"
+ _flag_10 = ", ".join(_r["lab"] for _r in _rows if 1.0 + _r["zmean"] > 0) or "(none)"
+ _flag_15 = ", ".join(_r["lab"] for _r in _rows if 1.5 + _r["zmean"] > 0) or "(none)"
+ _md = mo.md(
+ f"""
+ **Does swapping z* change the figure?** The *rank order* of flagged libraries is
+ invariant (rank-invariant across z* in [1.0, 2.0]: **{_rank_invariant}**) — that is
+ what stayed fixed when z* was swept. But the *set* of off-origin libraries grows with
+ z*, and that moves points:
+
+ | z* | libraries flagged coverage-failing (off origin) |
+ |----|--------------------------------------------------|
+ | 0.9 | {_flag_09} |
+ | 1.0 | {_flag_10} |
+ | 1.5 | {_flag_15} |
+
+ At **0.9 / 1.0** only adenine is flagged — **adenine_hybrid collapses onto the 'easy'
+ origin** despite greedy failing it (greedy ≈ 30 vs TACTICS ≈ 95). The window that
+ catches both coverage libraries yet excludes the mean-separable ones is
+ **z* ∈ ({_win_lo:.2f}, {_win_hi:.2f})**; 1.5 sits inside it, 0.9 / 1.0 do not.
+
+ **Why are some far-right points still hard?** This axis is *coverage only*, and colour
+ marks the **operative** mode. A mean-reachable hub (high −z(mean), far-right) means
+ greedy is not starved on coverage — but the library can still be hard via the other two
+ modes: **amide_hybrid** (blue) is allocation-hard (partner-specific / high-σ hub);
+ **thrombin** (orange) is imbalance-hard (30:1 → noisy estimates, Legacy RWS breaks).
+ Both sit far-right yet greedy-hard, and TACTICS wins on hub uncertainty / GMIC routing.
+ **Post-hoc statement:** greedy underperforms when the dominant hub triggers **any one —
+ or a combination — of the three modes** (coverage, allocation, imbalance); none
+ triggered ⇒ all methods converge (quinazoline). Outcome = the *combination*, never one
+ coordinate.
+ """
+ )
+
+ mo.vstack([
+ plt.gcf(),
+ _md,
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="coverage_zstar_sensitivity.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, np, plt, project_root):
+ """Docking reagent score landscape: the greedy-failure modes behind a TACTICS advantage.
+
+ POST-HOC ground-truth characterization, one point per docking library, placed by how
+ much it suffers each of the THREE greedy-failure modes that make TACTICS beat greedy
+ (a point at the origin with a thin edge has none -> easy to screen). The COMBINATION of
+ modes determines WHICH method wins (fill colour).
+ x = COVERAGE-failure severity = max(0, 1.5 + z(mean) of the dominant hub).
+ 0 = the hit-holding hub is greedy-reachable (mean-separated past the reachability
+ reference z* = -1.5, ~ top 7% by mean, Phi(-1.5)=0.067); large = a mean-blind hub
+ holds the hits and greedy starves it (adenine). The max() is the severity floor and
+ z* sets where it bites -- the clamp and the constant are ONE mechanism, not two
+ (max(0, z(mean)) with no offset is degenerate: every hit-hub has z(mean) < 0).
+ y = ALLOCATION-failure severity = z(std) × (1 − top-1 share).
+ 0 = greedy can cover the hit mass; large = the hits are partner-specific AND
+ dispersed, so greedy's budget can't cover the spread (amide_hybrid).
+ EDGE (orange, width) = reagent-count IMBALANCE = max/min component reagent count --
+ a search-dynamics mode: a heavily imbalanced library (thrombin 30:1) breaks Legacy
+ RWS's symmetric thermal cycling, so TACTICS wins even with NO hub failure.
+ FILL = which method wins (the combination of failure modes determines it):
+ GREY easy/method-agnostic (Δ_TL<1); PURPLE TACTICS-either; GREEN TT-TS; BLUE RWS.
+ Reading: x away from origin = coverage failure (adenine); up = allocation failure
+ (amide_hybrid); thick edge at origin = imbalance (thrombin); origin + thin edge = easy
+ (quinazoline). FRED→HYBRID arrows = reagent-score shift, same chemistry. Descriptive
+ structural summary (n=8, constructed thresholds), NOT a prospective classifier.
+ """
+ import json as _json
+ from matplotlib.lines import Line2D as _Line2D
+
+ _PROBE_DIR = project_root / "outputs" / "debug" / "docking_landscape_probe"
+ _MAP_LIBS = [
+ "thrombin", "thrombin_hybrid",
+ "adenine", "adenine_hybrid",
+ "amide", "amide_hybrid",
+ "quinazoline", "quinazoline_hybrid",
+ ]
+ _TT = "TACTICS Enhanced-TT-TS (GMIC)"
+ _RWS = "TACTICS Enhanced-RWS (GMIC)"
+ _LRWS = "Legacy Enhanced-RWS"
+ _LGRD = "Legacy Standard-Greedy"
+ _TIE_DELTA = 1.0 # delta_TL below this => easy / method-agnostic (grey)
+ _GAP_DELTA = 1.0 # |TT-TS - RWS| below this => TACTICS-either (purple)
+ _C_TIE, _C_TT, _C_RWS = "#343838", "#349E41", "#4477AA"
+ _C_EITHER = "#471A6E" # purple: TACTICS-either
+ _CAT_COLOR = {"easy": _C_TIE, "either": _C_EITHER, "ttts": _C_TT, "rws": _C_RWS}
+ _CAT_LABEL = {
+ "easy": "easy / method-agnostic (Δ$_{T-L}$ < 1)",
+ "either": "TACTICS (either) (|TT-TS − RWS| < 1)",
+ "ttts": "TT-TS wins",
+ "rws": "RWS wins",
+ }
+
+ def _categorize(_delta_tl, _gap_tr):
+ if _delta_tl < _TIE_DELTA:
+ return "easy"
+ if abs(_gap_tr) < _GAP_DELTA:
+ return "either"
+ return "ttts" if _gap_tr > 0 else "rws"
+
+ _EDGE = "#E8820E" # imbalance edge colour
+
+ def _imb_lw(_ratio):
+ # edge width grows with log imbalance ratio (2:1 thin -> 30:1 thick)
+ _t = (np.log10(_ratio) - np.log10(2.0)) / (np.log10(30.0) - np.log10(2.0))
+ return 1.0 + 3.2 * float(np.clip(_t, 0.0, 1.0))
+
+ _rows = {}
+ for _lab in _MAP_LIBS:
+ with open(_PROBE_DIR / f"{_lab}.json") as _fh:
+ _j = _json.load(_fh)
+ _h = _j["dominant_hub"]
+ _cidx = _h["component_idx"]
+ _m2 = _j["M2_concentration_by_topn"]["100"][f"comp_{_cidx}"]
+ _pm = _j["M6_search"]["per_method"]
+ _tt = _pm[_TT]["topn_recovered_mean"]
+ _rws = _pm[_RWS]["topn_recovered_mean"]
+ _delta = max(_tt, _rws) - max(_pm[_LRWS]["topn_recovered_mean"],
+ _pm[_LGRD]["topn_recovered_mean"])
+ _gap = _tt - _rws
+ _counts = list(_j["n_reagents"].values())
+ _rows[_lab] = {
+ "cov": max(0.0, 1.5 + _h["z_mean"]), # coverage-failure severity
+ "alloc": _h["z_std"] * (1.0 - _m2["top1_share"]), # allocation-failure severity
+ "imb": max(_counts) / min(_counts), # reagent-count imbalance
+ "category": _categorize(_delta, _gap),
+ }
+
+ # Reserve a right-hand gutter for the legends so they never collide with the
+ # plot region.
+ _fig, _ax = plt.subplots(figsize=(13.6, 8.5), facecolor="white")
+ _ax.set_facecolor("white")
+
+ _xs = np.array([_r["cov"] for _r in _rows.values()])
+ _ys = np.array([_r["alloc"] for _r in _rows.values()])
+ _x_hi = float(_xs.max())
+ _y_hi = float(_ys.max())
+ _ax.set_xlim(-0.10 * _x_hi, 1.22 * _x_hi)
+ _ax.set_ylim(-0.06 * _y_hi, 1.18 * _y_hi)
+
+ # Faint failure-mode region guides (axes are severities: origin = no failure).
+ _ax.axhline(0, color="#CCCCCC", lw=0.8, zorder=0)
+ _ax.axvline(0, color="#CCCCCC", lw=0.8, zorder=0)
+ _ax.annotate("easy\n(reachable, coverable, balanced)", (0.02 * _x_hi, 0.45),
+ fontsize=9.5, color="#888888", style="italic", ha="left",
+ va="bottom", zorder=1)
+ _ax.annotate("COVERAGE failure →\n(mean-blind hub)", (0.50 * _x_hi, 0.28),
+ fontsize=10, color="#B23030", style="italic", ha="left",
+ va="bottom", zorder=1)
+ _ax.annotate("↑ ALLOCATION failure\n(partner-specific + dispersed)",
+ (0.0, 0.97 * _y_hi),
+ fontsize=10, color="#2A5DA0", style="italic", ha="left",
+ va="top", zorder=1)
+
+ # Per-label offsets (several libraries cluster at cov=0; spread their labels).
+ _label_off = {
+ "amide_hybrid": (13, 0, "left", "center"),
+ "thrombin": (13, 11, "left", "bottom"),
+ "quinazoline": (13, 1, "left", "center"),
+ "quinazoline_hybrid": (13, -12, "left", "top"),
+ "amide": (13, -1, "left", "center"),
+ "thrombin_hybrid": (10, -11, "left", "top"),
+ "adenine": (13, 2, "left", "center"),
+ "adenine_hybrid": (13, 9, "left", "bottom"),
+ }
+ for _lab, _r in _rows.items():
+ _ax.scatter(_r["cov"], _r["alloc"], s=320,
+ c=_CAT_COLOR[_r["category"]], edgecolors=_EDGE,
+ linewidths=_imb_lw(_r["imb"]), alpha=0.95, zorder=3)
+ _dx, _dy, _hha, _vva = _label_off.get(_lab, (10, 7, "left", "bottom"))
+ _ax.annotate(_lab, (_r["cov"], _r["alloc"]), fontsize=10.5, color="black",
+ xytext=(_dx, _dy), textcoords="offset points", ha=_hha,
+ va=_vva, zorder=4)
+
+ # FRED -> HYBRID reagent-score-shift arrows (neutral SHIFT in the ground-truth
+ # reagent scores; same chemistry, NOT a quality judgment about a scoring function).
+ _pairs = [("amide", "amide_hybrid"), ("thrombin", "thrombin_hybrid"),
+ ("quinazoline", "quinazoline_hybrid"), ("adenine", "adenine_hybrid")]
+ for _a, _b in _pairs:
+ if _a in _rows and _b in _rows:
+ _ax.annotate("", xy=(_rows[_b]["cov"], _rows[_b]["alloc"]),
+ xytext=(_rows[_a]["cov"], _rows[_a]["alloc"]),
+ arrowprops=dict(arrowstyle="->", color="#999999", lw=1.1,
+ alpha=0.5, connectionstyle="arc3,rad=0.18"),
+ zorder=2)
+
+ _ax.set_xlabel("COVERAGE-failure severity max(0, 1.5 + z(mean))\n"
+ "(0 = greedy-reachable hub past z*=−1.5; large = mean-blind hidden hub)",
+ fontsize=12.5, color="black")
+ _ax.set_ylabel("ALLOCATION-failure severity z(std) × (1 − top-1 share)\n"
+ "(0 = greedy can cover; large = partner-specific + dispersed)",
+ fontsize=12, color="black")
+ _ax.set_title("Docking reagent score landscape — three greedy-failure modes behind a TACTICS advantage:\n"
+ "coverage (x) × allocation (y) × imbalance (edge width)",
+ fontsize=13.5, fontweight="bold", color="black")
+ _ax.tick_params(colors="black", labelsize=12)
+ _ax.grid(alpha=0.3, color="grey", zorder=1)
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+
+ # Outcome (fill) + FRED->HYBRID legend, upper-right gutter.
+ _cat_handles = [
+ _Line2D([0], [0], marker="o", ls="", ms=11, mfc=_CAT_COLOR[_c],
+ mec="#888888", label=_CAT_LABEL[_c])
+ for _c in ("easy", "either", "ttts", "rws")
+ ]
+ _cat_handles.append(
+ _Line2D([0], [0], color="#999999", lw=1.3,
+ label="FRED → HYBRID (reagent-score\nshift, same chemistry)")
+ )
+ _leg1 = _ax.legend(handles=_cat_handles, loc="upper left",
+ bbox_to_anchor=(1.02, 1.0), fontsize=10, framealpha=0.95,
+ edgecolor="black", title="Benchmark outcome (fill)",
+ title_fontsize=11, borderaxespad=0.0)
+ _ax.add_artist(_leg1)
+
+ _imb_refs = [2, 10, 30]
+ _imb_handles = [
+ _Line2D([0], [0], marker="o", ls="", mfc="#DDDDDD", mec=_EDGE,
+ mew=_imb_lw(_r), markersize=14, label=f"{_r}:1")
+ for _r in _imb_refs
+ ]
+ _leg2 = _ax.legend(handles=_imb_handles, loc="lower left",
+ bbox_to_anchor=(1.02, 0.0), fontsize=10, framealpha=0.95,
+ edgecolor="black", labelspacing=1.5, borderpad=1.0,
+ borderaxespad=0.0,
+ title="Reagent imbalance (edge)\nmax / min reagent count",
+ title_fontsize=10)
+
+ # Reserve the right gutter via subplots_adjust (NOT tight_layout(rect=...),
+ # which clips out-of-axes legends).
+ _fig.subplots_adjust(left=0.07, right=0.74, top=0.90, bottom=0.10)
+
+ # Display a TIGHT-bbox PNG (legends included as extra artists) so the two
+ # right-gutter legends are never clipped by marimo's canvas crop at this
+ # figure size — plt.gcf() crops to the figure box and cut them off.
+ import io as _io
+ _png = _io.BytesIO()
+ _fig.savefig(_png, format="png", dpi=150, bbox_inches="tight",
+ bbox_extra_artists=[_leg1, _leg2])
+ mo.vstack([
+ mo.image(_png.getvalue()),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="reagent_score_landscape_map.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, np, pl, plt, project_root):
+ """Greedy reach mechanism — coverage failure IS greedy starving the mean-blind hub.
+
+ Companion to the coverage axis, measured from the search shards (Legacy Greedy, search
+ phase only) vs ground-truth reagent z(mean) -- NO recovery/outcome is read, so it is a
+ non-circular mechanism measurement. enrichment = greedy's selection rate of a reagent
+ divided by uniform 1/N_component; >1 = greedy over-samples (reaches) it, <1 = starves it.
+ Greedy's enrichment is monotone in z(mean) and its over/under crossover is z(mean) ~ -0.9.
+ The adenine hub (amidine_003, z(mean) -0.58, holds 49/100 hits) sits at ~0.06x uniform --
+ greedy picks it ~18x LESS than blind chance. The coverage reference z*=-1.5 sits deep in
+ greedy's strong-exploitation region (~2.5x uniform), i.e. where greedy concentrates enough
+ budget to RECOVER a hub -- which is why -1.5 is the recovery-relevant frontier rather than
+ the bare over/under crossover at -0.9.
+ Source: outputs/debug/greedy_reach_calibration/ (experiments/_debug/greedy_reach_calibration.py).
+ """
+ import json as _json
+
+ _CAL = project_root / "outputs" / "debug" / "greedy_reach_calibration"
+ mo.stop(
+ not (_CAL / "reagents.csv").exists(),
+ mo.md("Run `python experiments/_debug/greedy_reach_calibration.py` first "
+ "(writes `outputs/debug/greedy_reach_calibration/`)."),
+ )
+
+ _reag = pl.read_csv(_CAL / "reagents.csv")
+ with open(_CAL / "summary.json") as _fh:
+ _summary = _json.load(_fh)
+ _hubs = _summary["hubs"]
+ _binned = _summary["binned_enrichment"]
+ _cross = _summary["pooled_crossover_mean"]
+
+ _floor = 0.02
+ _z = _reag["z_mean"].to_numpy()
+ _Ec = np.clip(_reag["enrichment"].to_numpy(), _floor, None)
+
+ _fig, _ax = plt.subplots(figsize=(11.5, 6.6), facecolor="white")
+ _ax.set_facecolor("white")
+ _ax.set_yscale("log")
+ _ax.scatter(_z, _Ec, s=6, color="#BBBBBB", alpha=0.25, zorder=1,
+ label="reagents (all components, 8 libraries)")
+ _bz = [_b["z_center"] for _b in _binned]
+ _bE = np.clip([_b["mean_E"] for _b in _binned], _floor, None)
+ _ax.plot(_bz, _bE, color="#222222", lw=2.0, zorder=3, label="binned mean enrichment")
+
+ _ax.axhline(1.0, color="#888888", ls=":", lw=1.4, zorder=2)
+ _ax.annotate("enrichment = 1 (reach / starve boundary)", (2.7, 1.15), fontsize=9,
+ color="#666666", va="bottom", ha="right")
+ _ax.axvline(_cross, color="#E8820E", ls="--", lw=1.6, zorder=2)
+ _ax.annotate(f"greedy crossover\nz(mean) ≈ {_cross:.1f}", (_cross, _floor * 1.5),
+ fontsize=9, color="#B5650E", va="bottom", ha="center")
+ _ax.axvline(-1.5, color="#B23030", ls="-", lw=1.6, alpha=0.85, zorder=2)
+ _ax.annotate("coverage\nreference z*=−1.5", (-1.5, 28.0), fontsize=9,
+ color="#B23030", va="top", ha="center")
+
+ _hub_off = { # fan out the labels of the hubs that cluster near z(mean) ~ -1.4..-1.9
+ "quinazoline_hybrid": (-6, 11, "right"),
+ "quinazoline": (4, -13, "left"),
+ "thrombin_hybrid": (8, 7, "left"),
+ }
+ for _h in _hubs:
+ _hy = max(_h["hub_enrichment"], _floor)
+ _ax.scatter(_h["hub_z_mean"], _hy, s=150, color="#349E41", edgecolors="black",
+ linewidths=0.9, zorder=5)
+ _dx, _dy, _hha = _hub_off.get(_h["library"], (6, 4, "left"))
+ _ax.annotate(_h["library"], (_h["hub_z_mean"], _hy), xytext=(_dx, _dy),
+ textcoords="offset points", fontsize=8.5, color="black",
+ ha=_hha, zorder=6)
+
+ _ax.set_xlabel("ground-truth reagent z(mean) "
+ "(more negative = better mean; docking minimizes)",
+ fontsize=11.5, color="black")
+ _ax.set_ylabel("greedy selection enrichment\n(selection rate ÷ uniform 1/N)",
+ fontsize=11.5, color="black")
+ _ax.set_title("Greedy reach: enrichment is monotone in hub mean — greedy starves the mean-blind hub\n"
+ "(dominant hubs in green; adenine sits at ~0.06× uniform — actively avoided)",
+ fontsize=12.5, fontweight="bold", color="black")
+ _ax.tick_params(colors="black", labelsize=11)
+ _ax.grid(alpha=0.3, color="grey", zorder=0)
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+ _ax.legend(loc="upper right", fontsize=9, framealpha=0.95, edgecolor="black")
+ _fig.tight_layout()
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="greedy_reach_mechanism.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+@app.cell
+def _(fig_to_pdf_bytes, mo, np, plt, project_root):
+ """Ranking heatmap: top-100 recovery for 5 methods × 8 docking landscapes.
+
+ Cell = M6 topn_recovered_mean (annotated). Columns ordered by the
+ best-TACTICS minus best-Legacy gap (largest left), so the libraries where
+ TACTICS helps most appear first (ordering not stated in the title -- the
+ manuscript text covers it). Single heatmap, no rank strip.
+ """
+ import json as _json
+
+ _PROBE_DIR = project_root / "outputs" / "debug" / "docking_landscape_probe"
+ _MAP_LIBS = [
+ "thrombin", "thrombin_hybrid",
+ "adenine", "adenine_hybrid",
+ "amide", "amide_hybrid",
+ "quinazoline", "quinazoline_hybrid",
+ ]
+ _METHODS = [
+ "TACTICS Enhanced-TT-TS (GMIC)",
+ "TACTICS Enhanced-RWS (GMIC)",
+ "Legacy Enhanced-RWS",
+ "Legacy Standard-Greedy",
+ ]
+ _MSHORT = {
+ "TACTICS Enhanced-TT-TS (GMIC)": "TACTICS TT-TS",
+ "TACTICS Enhanced-RWS (GMIC)": "TACTICS RWS",
+ "Legacy Enhanced-RWS": "Legacy RWS",
+ "Legacy Standard-Greedy": "Legacy Greedy",
+ }
+ _TT, _RWS = "TACTICS Enhanced-TT-TS (GMIC)", "TACTICS Enhanced-RWS (GMIC)"
+ _LRWS, _LGRD = "Legacy Enhanced-RWS", "Legacy Standard-Greedy"
+
+ _recov, _delta = {}, {}
+ for _lab in _MAP_LIBS:
+ with open(_PROBE_DIR / f"{_lab}.json") as _fh:
+ _j = _json.load(_fh)
+ _pm = _j["M6_search"]["per_method"]
+ _recov[_lab] = {_m: _pm[_m]["topn_recovered_mean"] for _m in _METHODS}
+ _bt = max(_pm[_TT]["topn_recovered_mean"], _pm[_RWS]["topn_recovered_mean"])
+ _bl = max(_pm[_LRWS]["topn_recovered_mean"], _pm[_LGRD]["topn_recovered_mean"])
+ _delta[_lab] = _bt - _bl
+
+ _libs = sorted(_MAP_LIBS, key=lambda l: _delta[l], reverse=True)
+ _mat = np.array([[_recov[l][_m] for l in _libs] for _m in _METHODS])
+
+ _fig, _ax = plt.subplots(figsize=(14, 5.2), facecolor="white")
+ _ax.set_facecolor("white")
+
+ _im = _ax.imshow(_mat, cmap="viridis", aspect="auto", vmin=0, vmax=100)
+ _ax.set_xticks(range(len(_libs)))
+ _ax.set_xticklabels(_libs, rotation=30, ha="right", rotation_mode="anchor",
+ fontsize=11, color="black")
+ _ax.set_yticks(range(len(_METHODS)))
+ _ax.set_yticklabels([_MSHORT[_m] for _m in _METHODS], fontsize=12, color="black")
+ for _r in range(_mat.shape[0]):
+ for _c in range(_mat.shape[1]):
+ _val = _mat[_r, _c]
+ _txt_c = "white" if _val < 55 else "black"
+ _ax.text(_c, _r, f"{_val:.0f}", ha="center", va="center",
+ fontsize=11, color=_txt_c, fontweight="bold")
+ _ax.set_title("Top-100 recovery (%) — methods × docking landscapes",
+ fontsize=14, fontweight="bold", color="black")
+ _ax.tick_params(colors="black")
+ _cbar = _fig.colorbar(_im, ax=_ax, fraction=0.025, pad=0.015)
+ _cbar.set_label("Top-100 recovery (%)", fontsize=12, color="black")
+ _cbar.ax.tick_params(colors="black", labelsize=10)
+ for _sp in _ax.spines.values():
+ _sp.set_color("black")
+
+ # Footnote: quinazoline's within-noise TT-TS caveat (data unchanged).
+ _fig.text(0.5, -0.06,
+ "quinazoline is method-agnostic (mean-reachable landscape); its "
+ "TT-TS value reflects a single 2% replicate set and is within-noise "
+ "of the others.",
+ ha="center", va="top", fontsize=9, color="#444444", style="italic",
+ wrap=True)
+
+ mo.vstack([
+ plt.gcf(),
+ mo.download(
+ data=fig_to_pdf_bytes(_fig),
+ filename="sar_ranking_heatmap.pdf",
+ mimetype="application/pdf",
+ label="Download as PDF",
+ ),
+ ])
+ return
+
+
+if __name__ == "__main__":
+ app.run()