Ship deepcell-types baselines as one .tar.gz bundle per baseline - #11
Merged
Conversation
Follows the cellsam layout (a single version-scoped top-level directory inside the archive) for the same reason: maps and xgboost each ship a companion file that is required at inference, so weights and companion should not be independently fetchable states. It also cuts transfer size substantially -- the xgboost booster is 1.1 GB of JSON that gzips 2.9x, so the three baselines go from 1.5 GB to 686 MB. The .pt/.pth weights are already-compressed tensor data and are unaffected; that is why the two main-model checkpoints stay bare .pt files, as torch-mesmer does. The value stays a LIST so a baseline can gain a second standalone asset later without a schema change, so `_download.py` needs no change: it still returns a list of local paths, now of length one. deepcell-types unpacks the bundle with its own path-traversal-safe extractor and returns the files inside (vanvalenlab/deepcell-types#47). Bundles are built deterministically (tar --sort=name, fixed owner/mtime, gzip -n), so these hashes are reproducible from the same inputs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz
xuefei-wang
pushed a commit
to xuefei-wang/deepcell-types
that referenced
this pull request
Jul 30, 2026
vanvalenlab/deepcell-auth#11 switched the baselines from loose per-file assets to one `.tar.gz` bundle each, following the cellsam layout: a single version-scoped top-level directory inside the archive. The motivation is atomicity -- `maps` and `xgboost` each ship a companion file required at inference (`_stats.npz`, `.remap.json`), and as separate assets those could be fetched into inconsistent states. Bundling also cuts the three baselines from 1.5 GB to 686 MB, almost all of it the xgboost booster (1.1 GB of JSON, gzips 2.9x). `download_baseline_checkpoint` now unpacks the bundle with this package's path-traversal-safe `extract_archive` -- deliberately not deepcell-auth's, which calls `extractall` unvetted -- and returns the files inside, sorted. The public contract is unchanged: still `list[Path]`, and sorting reproduces the order the loose assets were declared in (weights before companion). Extraction is skipped when the bundle directory already exists, matching cellsam. `fetch_data` still re-checks the archive's pinned hash on every call, so a corrupt download is caught; a hand-edited extraction directory is not. A bundle that unpacks to no directory, an empty one, or a plain file raises a readable error instead of NotADirectoryError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz
xuefei-wang
pushed a commit
that referenced
this pull request
Jul 30, 2026
Supersedes the per-baseline bundles from #11 (never uploaded). The three baselines now share one archive, with a subdirectory per baseline under the version-scoped top-level directory. Every baseline name maps to the same asset, so requesting any one downloads all three (646 MB). `fetch_data` caches by filename, so the other two are then served from the cache without a second transfer; deepcell-types unpacks the archive and returns only the requested baseline's subdirectory. `_download.py` is unchanged -- the value is still a list of records, and it still returns a list of local paths of length one. Built deterministically (tar --sort=name, fixed owner/mtime, gzip -n), so the hash is reproducible from the same inputs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #8/#10, matching the layout
cellSAMalready uses for its multi-file model assets.Why
mapsandxgboosteach ship a companion file that is required at inference (_stats.npzfeature-norm statistics;.remap.jsonlabel remap). As loose per-file assets those are independently fetchable, so a partial fetch leaves a usable-looking weights file with no companion. One bundle per baseline makes weights + companion a single atomic asset — the same reasoncellsam-models_v1.2.tar.gzbundlescellsam_general.pt+cellsam_extra.pt.It also cuts transfer size, because the xgboost booster is 1.1 GB of JSON:
The two main-model checkpoints stay bare
.ptfiles: they are single-file per version, and already-compressed tensor data (1.08×), so there is nothing to gain — same shape astorch-mesmer.Shape
Archive layout follows cellsam's version-scoped top-level directory:
The manifest value stays a list so a baseline can gain a second standalone asset later without a schema change; today every list has exactly one entry. That means no change to
_download.py—download_deepcell_types_baselinestill returns a list of local paths, now of length one. Unpacking happens on the deepcell-types side (vanvalenlab/deepcell-types#47) using its own path-traversal-safe extractor rather than this package's unguardedextractall.2026-06-30is the DCT-sampler retrain generation these checkpoints come from, so a future retrain gets a new key rather than silently replacing these bytes.Verification
Bundles are built deterministically —
tar --sort=name --owner=0 --group=0 --numeric-owner --mtime=<fixed>piped throughgzip -n— so the pinned hashes are reproducible from the same inputs. Confirmed by rebuilding themapsbundle from scratch and getting a byte-identicalf5f0ae6d….Every file extracted from the three bundles md5-matches the original loose checkpoint, and each
asset_hashhere matches the built archive.🤖 Generated with Claude Code
https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz