Adopt deepcell-auth for model/data downloads

[xuefei-wang]

Replaces this repo's inlined asset-download layer with the shared deepcell-auth client, as vanvalenlab/torch-mesmer already does. deepcell-auth's bundled manifest becomes the single source of truth for asset keys and hashes, so this repo drops its duplicate registry and auth transport.

All the upstream pieces have landed: vanvalenlab/deepcell-auth#8 (the deepcell-types + baseline manifest entries), #10 (the download helpers now return the downloaded path), and #12 (all three baselines served in a single .tar.gz). The dependency is pinned to vanvalenlab/deepcell-auth@main.

Changes

• Add the deepcell_auth git dependency; delete utils/_auth.py and the local model/baseline registries. Downloads delegate to deepcell_auth.download_deepcell_types_{model,baseline,data}.
• Public API (download_model, download_baseline_checkpoint, download_training_data, list_model_versions, list_baseline_names) is unchanged — thin adapters preserve the signatures, the Path / list[Path] return types, the ValueError on an unknown identifier, and the Nimbus upstream pointer. predict.py is untouched.
• Keep the path-traversal-safe extract_archive local, moved to utils/_archive.py, since deepcell-auth's extractor calls extractall without vetting members. This is the current hardened version (zip-slip / tar-traversal / symlink rejection plus the member-count and per-member/total size bounds), moved verbatim.
• Unpack the shared baseline bundle. deepcell-auth#12 moved the baselines from loose per-file assets into one .tar.gz with a subdirectory per baseline (cellsam's layout: a single version-scoped top-level directory), so download_baseline_checkpoint now extracts and returns just the requested baseline's subdirectory. The public contract is unchanged — still list[Path], and sorting reproduces the order the loose assets were declared in (weights before companion).
• Add offline tests asserting the mirrored version/baseline lists stay in sync with the bundled manifest, and covering the unpack path (extraction, re-use of an already-unpacked bundle, and malformed bundles).

Two behavioral details

• The SSL checkpoint's manifest key is 2026-06-23-ptft, not 2026-06-23 — deepcell-auth#8 renamed it. The old key would have raised KeyError from inside deepcell_auth at download time.
• The 2025 CLIP checkpoints are in the manifest but not offered here. They are served for reproducibility with the matching historical commit and cannot be loaded by current code, so download_model rejects them rather than handing back a checkpoint that fails at load_state_dict.

Trade-off worth recording

deepcell_auth.fetch_data is simpler than the transport this PR deletes. Relative to utils/_auth.py on master it: hashes only for cache lookup (no post-download integrity check), writes straight to the final path instead of temp-then-os.replace, hashes whole-file instead of chunked, sets no request timeouts, and has no Content-Length cap. Consolidating on the shared client was chosen deliberately over keeping a second transport in this repo; porting the hardening upstream into deepcell-auth would restore it for torch-mesmer and cellsam too, and is the natural follow-up.

Verification

• uv run pytest — 470 passed, 1 skipped (with the [train] extra, matching CI). ruff check clean.

• Drove the real 646 MB bundle through download_baseline_checkpoint with only the network hop stubbed: every extracted file md5-matches the original loose checkpoint, each request returns only its own baseline's files, and all three baselines cost one transfer.

• Offline wiring check with fetch_data stubbed: every asset key and hash the adapters request matches the staged upload set byte-for-byte —

  call	asset key	hash
  download_model()	models/deepcell-types_2026-06-15_resmlp.pt	b819a7e0…
  download_model("2026-06-23-ptft")	models/deepcell-types_2026-06-23_resmlp_ptft.pt	402e94c1…
  download_baseline_checkpoint("cellsighter" | "maps" | "xgboost")	models/deepcell-types_baselines_2026-06-30.tar.gz	910840c7…
  download_training_data()	data/deepcell-types/public_data_v1.1.zip	(unhashed asset)

Why the baselines are bundled and the models are not

maps and xgboost each ship a companion file required at inference (_stats.npz, .remap.json); as separate assets those can be fetched into inconsistent states. One archive also means a single asset to upload, version and hash instead of five, and it cuts the baselines from 1.5 GB to 646 MB — almost all of that the xgboost booster, 1.1 GB of JSON that gzips 2.94×.

The cost is that requesting any single baseline downloads all three. fetch_data caches by filename, so the first request pays the transfer and the other two are then free; a user who only ever wants maps (7 MB of checkpoint) still pulls 646 MB once.

The model checkpoints stay bare .pt files: one file per version, and already-compressed tensor data (measured 1.08×), so a tarball would buy nothing. That is also the shape torch-mesmer uses.

Note on the diff

The branch was rebuilt directly on master (c677b28). Its previous head was based on the fork's lineage, whose merge-base with this repo's squash-merged v0.1.0 history is the pre-release 616d4b5 — so GitHub was showing the whole monorepo merge as the diff and reported 171 conflicts. The rebuilt branch is the same change reapplied on the correct base: 8 files, +341/−799.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sy8eiMVmmQHXRAccKwzAyz