[denniswu28]

Hallmark can now catalogue data that lives on private servers and download it on demand, with every file verified and recorded for commit.

Closes #51. Closes #54.

What this adds

• Servers: SSH and SFTP (OpenSSH 8.0 or later), HTTP(S) with basic or bearer authentication, WebDAV, and S3 (optional pip install hallmark[s3]). Credentials come from typed profiles in a per-user auth.yml, never from the repository or a URL.
• Sources: one catalogue can combine selections from several folders and servers, recorded under sources: in config.yml. Adding a source again updates it:
  • new files are added;
  • files that changed upstream are listed;
  • files gone from the server are removed or kept after asking (--prune / --keep-missing).
• Clashes: paths another source already catalogs are labelled as same content, different content or not comparable. hm add then offers to skip them, keep both under a folder, or abort (--skip-duplicates in scripts). Paths that differ only in letter case get a warning.
• Filename formats (Sam's work from the base branch, merged in):
  • formats belong to the whole catalogue and sort every source's files, each format into its own TSV;
  • files no format matches become file: entries;
  • hm detect suggests formats;
  • when a new format arrives, hm add asks whether to re-sort this source's files, every source's files (all), or only unsorted ones.
• Checksums: taken from published manifests, WebDAV properties, S3, or the server itself (--server-checksum). The checksum_source column records where each came from.
• Downloads: each download runs plan → approval → .hm/objects → worktree → staged SHA-1.
  • Files are fetched into a content-addressed, read-only object store, with size and digest verified.
  • They are hard-linked into the worktree, or copied across filesystems.
  • Each file's SHA-1 is staged for commit.
  • An interrupted download leaves nothing partial, and finished files are still recorded.
  • Outdated files are replaced only after approval, and old versions stay in .hm/objects, so checking out an older commit restores them without a transfer.
  • Edits made in place to a downloaded file are detected.
• Clone: published catalogue snapshots or Git repositories; data is downloaded only after approval.

Changes to existing behaviour

• The column naming a file's source is catalog_source, so {source} stays free as a format field (for example M87).
• Formats no longer select files; --filter does.
• A scan that fails or matches nothing exits with an error and keeps the previous catalogue.
• Downloaded files are read-only links into the object store; copy a file to edit it. Local files are still writable copies, as before.

Testing

• Locally: 1497 tests pass on Python 3.13, including the loopback SSH tests, and 1496 on Python 3.9 with Click 8.1. Sphinx builds with -W.
• CI: all 19 jobs pass, including Python 3.9 to 3.14 on Ubuntu, 3.9 and 3.14 on macOS, Rocky Linux 8 and 9 OpenSSH, S3 against moto, the lowest supported dependencies, and pre-release dependencies. Combined coverage is 99%, and Codecov reports 99.35%; the S3 job's coverage now counts toward it.
• demo_private_data.ipynb and demo_add.ipynb run as tests.
• Manual end-to-end runs of hm against local HTTP servers covered:
  • the duplicate prompt;
  • replacing a file that changed upstream, then restoring the old version from .hm/objects on a branch switch;
  • hm detect --add followed by download and status.

Also fixed

Edge-case tests found three older bugs, fixed here with regression tests:

• --prefix ./ (or .) failed with "Remote path cannot be empty".
• restore --staged . did nothing on a branch with no commits.
• A local repository in a folder whose name contains braces, such as run_{id}, globbed a sibling folder's files.

Not in this PR

• hm gc for objects no commit references.
• Mirrors with automatic fallback.
• Restoring a single file: entry with restore --staged (restore .hm/config.yml instead).
• The notebook-style Python API (hm.clone("eht/2026:science-release-1")), planned as a separate branch.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 99.52951% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
mod/hallmark/repo/dothm.py	93.4%	7 Missing and 2 partials ⚠️
mod/hallmark/transport/auth.py	98.6%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!