Add deepcell types 2026-06-23 weights to registry - #8
Conversation
…25-06-09 Stacked on vanvalenlab#8. Three changes to the deepcell-types asset registry: 1. Model: serve the vocab-bundled 2026 checkpoints and default to 2026-06-15. - 2026-06-15 (default) -> deepcell-types_2026-06-15_resmlp.pt, b819a7e0 - 2026-06-23 -> deepcell-types_2026-06-23_resmlp_ptft.pt, 402e94c1 These bundle ct2idx/canonical_channels so deepcell-types predict()'s vocab guard accepts them (the un-bundled 067f558b in vanvalenlab#8 fails it). This matches deepcell-types _model_registry (PR xuefei-wang/deepcell-types#70). 2. Baselines: add a deepcell-types-baselines section + download helper for the comparison baselines (cellsighter, maps [+ _stats], xgboost [+ .remap]), hashes matching the 2026-06-30 retrains in deepcell-types #74. Each baseline maps to a list of assets to carry companion files. Nimbus is omitted (its weights are distributed upstream via nimbus-inference, not re-hosted). 3. Drop the legacy 2025-06-09 CLIP entries: they are incompatible with the current deepcell-types code and have no deepcell-types consumers (both the old and new packages download via their own _auth, not deepcell_auth). Assets must be uploaded to users.deepcell.org before merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oVttaRJScBzjoJDB8ZkpA
… conflicts main advanced (torch-mesmer weights, vanvalenlab#7) after this branch was cut. Resolved by keeping all of main's additions (torch-mesmer manifest section + download_torch_mesmer_model + its __all__ entry) alongside the deepcell-types changes on this branch: - deepcell-types serves the bundled 2026 checkpoints, default 2026-06-15 - new deepcell-types-baselines section + download_deepcell_types_baseline - legacy 2025-06-09 entries dropped Also fixes a latent export bug: download_deepcell_types_baseline is now added to __all__ (this branch predated main introducing __all__, so the function would not have been re-exported via `from ._download import *`). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oVttaRJScBzjoJDB8ZkpA
|
Heads-up @rossbar — I pushed two commits directly onto this PR's branch ( 1. Serve the vocab-bundled 2026 checkpoints; default to
The original 2. Add comparison baselines ( 3. Dropped the legacy Merge/conflict: I also merged current
|
|
We do not want to remove previous versions of the model - these are important for reproducibility/showing progress. |
|
But the old checkpoint requires the users to checkout an old snapshot of the code base in order to use, right? |
Yup! But that's no problem as it's all in the repo - all we have to do is point to a commit hash for users who want to do this (e.g. hubmap) |
Per Ross's review: previous model versions must not be removed — they're needed for reproducibility. Restores the two legacy 2025-06-09 CLIP checkpoints (2025-06-09 and 2025-06-09_public-data-only) to the manifest and the download_deepcell_types_model docstring. Users pair these with the matching historical deepcell-types commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178visUmEvakMZfFV3Aa33P
The version identifier now reflects the pretrain-finetune (ptft) arm. Asset filename unchanged (already deepcell-types_2026-06-23_resmlp_ptft.pt). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178visUmEvakMZfFV3Aa33P
Gotcha, reverted! The PR should be good. |
|
Confirmed — every hash in this PR matches the
Two notes on the two main-model assets:
The four legacy/other entries ( Merging — the deepcell-types side (vanvalenlab/deepcell-types#47) will pin |
deepcell-auth's bundled asset_manifest.yaml is now the single source of truth for deepcell-types asset keys and integrity hashes, so this repo drops its duplicate registry and its auth transport (`utils/_auth.py`). The public API is unchanged: `download_model`, `download_baseline_checkpoint`, `download_training_data`, `list_model_versions` and `list_baseline_names` keep their signatures, `Path`/`list[Path]` return types, the `ValueError` on an unknown identifier, and the pointer to Nimbus' upstream weights. `predict.py` is untouched. Two details worth flagging: * The SSL checkpoint's manifest key is `2026-06-23-ptft`, not `2026-06-23` (vanvalenlab/deepcell-auth#8 renamed it); the mirrored list follows. * The 2025 CLIP checkpoints are in the manifest for reproducibility but cannot be loaded by this code, so `download_model` rejects them instead of handing back a checkpoint that fails at `load_state_dict`. `tests/test_download_delegation.py` reads the packaged manifest offline and fails if the mirrored versions/baselines drift from it, which is what would otherwise surface as a `KeyError` from inside `deepcell_auth` at download time. The `_auth.py` unit tests go with the module they covered. Note: deepcell-auth's `fetch_data` is simpler than the transport removed here -- it hashes only for cache lookup (no post-download integrity check), writes straight to the final path rather than temp-then-replace, hashes whole-file rather than chunked, and sets no request timeouts. Adopting it is the deliberate consolidation trade-off; the safe `extract_archive` was kept locally rather than delegated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz
* build: add the deepcell-auth dependency The shared client's bundled ``asset_manifest.yaml`` becomes the single source of truth for deepcell-types asset keys and integrity hashes, the way vanvalenlab/torch-mesmer already consumes it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz * refactor(utils): move extract_archive into its own module The next commit deletes this repo's download transport in favour of deepcell-auth, but ``extract_archive`` stays: deepcell-auth's extractor calls ``extractall`` without vetting members, so delegating it would drop the zip-slip / tar-traversal / symlink rejection and the member-count and size bounds. Move it (and the archive limits it reads) out of ``_auth`` ahead of that deletion, and split its tests out of ``tests/test_auth.py``. Pure move: no behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz * refactor(utils): delegate model/baseline/data downloads to deepcell-auth deepcell-auth's bundled asset_manifest.yaml is now the single source of truth for deepcell-types asset keys and integrity hashes, so this repo drops its duplicate registry and its auth transport (`utils/_auth.py`). The public API is unchanged: `download_model`, `download_baseline_checkpoint`, `download_training_data`, `list_model_versions` and `list_baseline_names` keep their signatures, `Path`/`list[Path]` return types, the `ValueError` on an unknown identifier, and the pointer to Nimbus' upstream weights. `predict.py` is untouched. Two details worth flagging: * The SSL checkpoint's manifest key is `2026-06-23-ptft`, not `2026-06-23` (vanvalenlab/deepcell-auth#8 renamed it); the mirrored list follows. * The 2025 CLIP checkpoints are in the manifest for reproducibility but cannot be loaded by this code, so `download_model` rejects them instead of handing back a checkpoint that fails at `load_state_dict`. `tests/test_download_delegation.py` reads the packaged manifest offline and fails if the mirrored versions/baselines drift from it, which is what would otherwise surface as a `KeyError` from inside `deepcell_auth` at download time. The `_auth.py` unit tests go with the module they covered. Note: deepcell-auth's `fetch_data` is simpler than the transport removed here -- it hashes only for cache lookup (no post-download integrity check), writes straight to the final path rather than temp-then-replace, hashes whole-file rather than chunked, and sets no request timeouts. Adopting it is the deliberate consolidation trade-off; the safe `extract_archive` was kept locally rather than delegated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz * feat(utils): unpack the shared baseline bundle served by deepcell-auth vanvalenlab/deepcell-auth#12 ships all three baselines in a single `.tar.gz` with a subdirectory per baseline, under one version-scoped top-level directory (the cellsam layout). The motivation is atomicity -- `maps` and `xgboost` each ship a companion file required at inference (`_stats.npz`, `.remap.json`), and as separate assets those could be fetched into inconsistent states -- plus one asset to upload, version and hash instead of five. Bundling also compresses the xgboost booster (1.1 GB of JSON) by ~2.9x, so the baselines go from 1.5 GB to 646 MB. `download_baseline_checkpoint` now unpacks the bundle with this package's path-traversal-safe `extract_archive` -- deliberately not deepcell-auth's, which calls `extractall` unvetted -- and returns only the requested baseline's subdirectory, sorted. The public contract is unchanged: still `list[Path]`, and sorting reproduces the order the loose assets were declared in (weights before companion). The cost of one shared archive is that requesting any single baseline downloads all three. `fetch_data` caches by filename, so the first request pays the transfer and the other two are then served from cache; verified end-to-end against the real bundle (three baselines, one transfer). Extraction is skipped when the bundle directory already exists, matching cellsam. `fetch_data` still re-checks the archive's pinned hash on every call, so a corrupt download is caught; a hand-edited extraction directory is not. A bundle missing the requested baseline's subdirectory, or holding a plain file in its place, raises a readable error instead of NotADirectoryError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz --------- Co-authored-by: xuefei-wang <yuecrew42@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Okay here's the replacement for #5 !
@xuefei-wang can you confirm that the hash for this set of weights is the right one (i.e. the listed hash matches the
md5sumof the weights that should be uploaded)