Ship all deepcell-types baselines in a single .tar.gz - #12
Merged
Conversation
Supersedes the per-baseline bundles from #11 (never uploaded). The three baselines now share one archive, with a subdirectory per baseline under the version-scoped top-level directory. Every baseline name maps to the same asset, so requesting any one downloads all three (646 MB). `fetch_data` caches by filename, so the other two are then served from the cache without a second transfer; deepcell-types unpacks the archive and returns only the requested baseline's subdirectory. `_download.py` is unchanged -- the value is still a list of records, and it still returns a list of local paths of length one. Built deterministically (tar --sort=name, fixed owner/mtime, gzip -n), so the hash is reproducible from the same inputs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz
xuefei-wang
pushed a commit
to xuefei-wang/deepcell-types
that referenced
this pull request
Jul 30, 2026
vanvalenlab/deepcell-auth#12 ships all three baselines in a single `.tar.gz` with a subdirectory per baseline, under one version-scoped top-level directory (the cellsam layout). The motivation is atomicity -- `maps` and `xgboost` each ship a companion file required at inference (`_stats.npz`, `.remap.json`), and as separate assets those could be fetched into inconsistent states -- plus one asset to upload, version and hash instead of five. Bundling also compresses the xgboost booster (1.1 GB of JSON) by ~2.9x, so the baselines go from 1.5 GB to 646 MB. `download_baseline_checkpoint` now unpacks the bundle with this package's path-traversal-safe `extract_archive` -- deliberately not deepcell-auth's, which calls `extractall` unvetted -- and returns only the requested baseline's subdirectory, sorted. The public contract is unchanged: still `list[Path]`, and sorting reproduces the order the loose assets were declared in (weights before companion). The cost of one shared archive is that requesting any single baseline downloads all three. `fetch_data` caches by filename, so the first request pays the transfer and the other two are then served from cache; verified end-to-end against the real bundle (three baselines, one transfer). Extraction is skipped when the bundle directory already exists, matching cellsam. `fetch_data` still re-checks the archive's pinned hash on every call, so a corrupt download is caught; a hand-edited extraction directory is not. A bundle missing the requested baseline's subdirectory, or holding a plain file in its place, raises a readable error instead of NotADirectoryError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz
xuefei-wang
added a commit
to vanvalenlab/deepcell-types
that referenced
this pull request
Jul 30, 2026
* build: add the deepcell-auth dependency The shared client's bundled ``asset_manifest.yaml`` becomes the single source of truth for deepcell-types asset keys and integrity hashes, the way vanvalenlab/torch-mesmer already consumes it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz * refactor(utils): move extract_archive into its own module The next commit deletes this repo's download transport in favour of deepcell-auth, but ``extract_archive`` stays: deepcell-auth's extractor calls ``extractall`` without vetting members, so delegating it would drop the zip-slip / tar-traversal / symlink rejection and the member-count and size bounds. Move it (and the archive limits it reads) out of ``_auth`` ahead of that deletion, and split its tests out of ``tests/test_auth.py``. Pure move: no behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz * refactor(utils): delegate model/baseline/data downloads to deepcell-auth deepcell-auth's bundled asset_manifest.yaml is now the single source of truth for deepcell-types asset keys and integrity hashes, so this repo drops its duplicate registry and its auth transport (`utils/_auth.py`). The public API is unchanged: `download_model`, `download_baseline_checkpoint`, `download_training_data`, `list_model_versions` and `list_baseline_names` keep their signatures, `Path`/`list[Path]` return types, the `ValueError` on an unknown identifier, and the pointer to Nimbus' upstream weights. `predict.py` is untouched. Two details worth flagging: * The SSL checkpoint's manifest key is `2026-06-23-ptft`, not `2026-06-23` (vanvalenlab/deepcell-auth#8 renamed it); the mirrored list follows. * The 2025 CLIP checkpoints are in the manifest for reproducibility but cannot be loaded by this code, so `download_model` rejects them instead of handing back a checkpoint that fails at `load_state_dict`. `tests/test_download_delegation.py` reads the packaged manifest offline and fails if the mirrored versions/baselines drift from it, which is what would otherwise surface as a `KeyError` from inside `deepcell_auth` at download time. The `_auth.py` unit tests go with the module they covered. Note: deepcell-auth's `fetch_data` is simpler than the transport removed here -- it hashes only for cache lookup (no post-download integrity check), writes straight to the final path rather than temp-then-replace, hashes whole-file rather than chunked, and sets no request timeouts. Adopting it is the deliberate consolidation trade-off; the safe `extract_archive` was kept locally rather than delegated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz * feat(utils): unpack the shared baseline bundle served by deepcell-auth vanvalenlab/deepcell-auth#12 ships all three baselines in a single `.tar.gz` with a subdirectory per baseline, under one version-scoped top-level directory (the cellsam layout). The motivation is atomicity -- `maps` and `xgboost` each ship a companion file required at inference (`_stats.npz`, `.remap.json`), and as separate assets those could be fetched into inconsistent states -- plus one asset to upload, version and hash instead of five. Bundling also compresses the xgboost booster (1.1 GB of JSON) by ~2.9x, so the baselines go from 1.5 GB to 646 MB. `download_baseline_checkpoint` now unpacks the bundle with this package's path-traversal-safe `extract_archive` -- deliberately not deepcell-auth's, which calls `extractall` unvetted -- and returns only the requested baseline's subdirectory, sorted. The public contract is unchanged: still `list[Path]`, and sorting reproduces the order the loose assets were declared in (weights before companion). The cost of one shared archive is that requesting any single baseline downloads all three. `fetch_data` caches by filename, so the first request pays the transfer and the other two are then served from cache; verified end-to-end against the real bundle (three baselines, one transfer). Extraction is skipped when the bundle directory already exists, matching cellsam. `fetch_data` still re-checks the archive's pinned hash on every call, so a corrupt download is caught; a hand-edited extraction directory is not. A bundle missing the requested baseline's subdirectory, or holding a plain file in its place, raises a readable error instead of NotADirectoryError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz --------- Co-authored-by: xuefei-wang <yuecrew42@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes the per-baseline bundle shape from #11 (nothing was uploaded under those keys, so no served asset is affected).
The three baselines now share one archive, with a subdirectory per baseline under the version-scoped top-level directory:
models/deepcell-types_baselines_2026-06-30.tar.gz— 646 MB, md5910840c7caf10fea2c60b66061074471. All three baseline names map to it.Trade-off
Requesting any single baseline downloads all three.
fetch_datacaches by filename, so the first request pays the full transfer and the other two are then served from cache with no second download — but a user who only ever wantsmaps(7 MB of checkpoint) still pulls 646 MB.The upside is one asset to upload, version, and hash instead of three, and the same atomicity property that motivated #11:
mapsandxgboosteach ship a companion file required at inference, and those can never be fetched into inconsistent states._download.pyis unchanged — the manifest value is still a list of records, anddownload_deepcell_types_baselinestill returns a list of local paths of length one. deepcell-types unpacks the archive and returns only the requested baseline's subdirectory (vanvalenlab/deepcell-types#47).Verification
Built deterministically (
tar --sort=name --owner=0 --group=0 --numeric-owner --mtime=<fixed>piped throughgzip -n), so the pinned hash is reproducible from the same inputs. Every file extracted from the bundle md5-matches the original loose checkpoint.🤖 Generated with Claude Code
https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz