[denniswu28]

Summary

Hallmark can now download private datasets through existing SSH access and build catalogs from an explicit SSH dataset root.

• Add SSH/SFTP transports with strict host-key checking, literal-path handling, bounded parallel transfers, and process cleanup. Preserve atomic downloads, checksum verification, and the existing download result mapping.
• Add host-bound local authentication profiles and CLI/API options for remote authentication and dataset sources. Catalogs store profile names; credentials remain local.
• Make server-side listing and bounded SHA-256 hashing explicit opt-ins. Indexed downloads also work with SFTP-only accounts.

Requires OpenSSH 9.6+; SSH catalog building additionally requires a POSIX shell and Python 3 on the server.

Review guide

• Core implementation and tests: start with transport/, then downloader/builder integration, then configuration and CLI wiring.
• Separate Sphinx warning fixes.
• CI now covers dev pushes and pull requests, including the Ubuntu/macOS SSH integration jobs.

Validation

• 642 pytest tests passed, including 16 real loopback OpenSSH tests.
• 250 EHT2026 VM-backed acceptance cases: 181 passes, 69 expected rejections, zero unexpected failures. Covered all 12 CLI commands, every option, 246 representative option pairs, checksum failures, authentication/trust failures, and destination preservation.
• Ruff and the strict Sphinx build passed.
• Package source hashes match the tested implementation; the subsequent change only enables CI for dev.

[denniswu28]

Final pre-review of b1e93035d7320aafbf6d8cbda46f7f998e226e1a, focused on SSH/SFTP private-data support and cross-platform behavior.

Recommendation: keep this PR in draft until these two issues are fixed.

1. [P1] Nested SSH manifests can silently lose checksum protection.
   An entry for sub/item_1.dat in run/export.sha256sums resolves to root-level sub/item_1.dat instead of run/sub/item_1.dat. The checksum is discarded—or assigned to the wrong file if that root-level path exists. Using a local transport fixture, I reproduced a successful download of modified content because the generated catalog had dropped its checksum.

   Resolve SSH manifest entries relative to the manifest directory, and add regression tests for both missing-target and root-level collision cases. [Affected code](

   hallmark/mod/hallmark/repo_builder.py

   Lines 902 to 907 in b1e9303

   	directory = str(Path(path).parent)
   	directory = "" if directory == "." else directory + "/"
   	for checksum, filename in matches:
   	target = _resolve_manifest_path(filename, directory)
   	if target in result:
   	_record_checksum(result, target, algorithm, checksum)

   ).

2. [P2] CRLF checksum manifests break HTTP catalog builds.
   The new parser includes the carriage return in the filename, which path validation rejects as a control character. An otherwise identical HTTP build succeeds with LF and fails with CRLF. The base parser accepted CRLF.

   Normalize record endings while preserving intentional filename whitespace, and add a regression test through the HTTP catalog-building path. [Affected parser](

   hallmark/mod/hallmark/repo_builder.py

   Lines 50 to 51 in b1e9303

   	# regex to match lines in a checksum file, capturing the checksum and filename
   	_SUMS_LINE_RE = re.compile(r"^([0-9a-fA-F]{8,}) [ *](.+)$", re.MULTILINE)

   ).

Validation: 276 focused existing tests passed locally. All eight [CI jobs](https://github.com/lanisrc/hallmark/actions/runs/35171364747) passed; Ubuntu and macOS each report 103 transport tests passed without skips. Windows SSH is explicitly unsupported.

The implementation adds host-bound local profiles, strict host-key checking, bounded transfers, and explicit opt-ins for remote listing and hashing. I found no additional substantive blockers in authentication policy, command quoting, atomic downloads, or process cleanup.

[denniswu28]

Raw dataset discovery now uses hallmark init PATH --from URL. hallmark clone copies existing Git catalogs or published HTTP/SFTP catalog snapshots. Catalog hosting and data hosting remain independent, so a catalog on GitHub or another metadata server can point to data elsewhere.

Changes

• Preserve plain local initialization and existing .hm layouts. Remote initialization reads listings and published checksums, supports filters and backend options, and preserves an existing destination directory without .hm.
• Preserve complete catalogs and Git history when cloning. Clone filters select optional downloads and require --with-download; --download remains supported. Raw dataset URLs receive guidance to use init --from.
• Expose DataBackend, generic HTTP/SSH and CyVerse implementations, named registration, and installed plugins through hallmark.backends. Include a documented backend that routes logical catalog paths across multiple servers.
• Keep metadata authentication separate from data authentication. Offline download plans retain the selected source, backend settings, profile reference, destination, and checksums. Transfers require approval and share checksum verification, cancellation, concurrency control, and atomic publication.
• Retain legacy builder support with deprecation guidance pointing to init --from.
• Align README and Sphinx examples with current APIs while preserving their hierarchy and named-person use cases. Include both scientific workflow files, restore their links, and simplify the private-data demo. The four scientific examples each select one input and demonstrate branching through a small analysis-settings file; detailed scientific processing remains prose.

Validation

• 95 CLI regression tests passed on Python 3.9 and 3.13 for the documentation revision.
• All four scientific CLI workflows and the private-data demo executed against tiny local HTTP/SFTP fixtures on both Python versions. Checks covered published checksums, approval, refusal, empty selections, configuration changes, catalog cloning, settings commits, and recovery.
• All 11 notebook code cells executed from both the repository root and demo/ on both Python versions, including worktree isolation and unchanged input checksums. Separate Python 3.13 checks verified refusal and transfer-failure stopping behavior in the notebook, and transfer-failure stopping behavior in the CLI.
• Bash and Python 3.9 syntax, notebook structure and cleared outputs, Ruff, strict Sphinx, local links/rendered documentation, and git diff --check passed.

Validation used disposable local services without contacting research servers or downloading public datasets. These fixture checks do not validate scientific processing or public-server availability. Remote CI is reported separately from local validation.

[denniswu28]

Following review feedback, remote datasets are now cataloged with hallmark add instead of hallmark init --from, so remote data follows the same workflow as local data:

hallmark init eht && cd eht
hallmark add 'https://archive.example.org/2026MOVIE/ER2/{src}_{day}.h5'
hallmark commit -m 'Add remote EHT data'
hallmark download --all

add splits the URL at the first path segment that contains a {field}. The part before it is recorded as the template's source. The matching files are cataloged with their sizes and published checksums, without being downloaded. I checked the command design against git init, add, clone and remote, and against git-annex, DataLad and DVC. The choices below follow those conventions where they apply.

Changes

• init is local-only again. init --from, --release, --collection, --filter and --format, all added earlier in this PR, are removed, as is Repo.init(source=...). dev never had them, so there are no aliases. hallmark sources NAME still prints release and collection URLs, and templates under a release URL inherit that release's backend.
• add URL/TEMPLATE:
  • It lists only the directories the template can match.
  • A field never spans /, and a file is cataloged only if the template re-creates its name exactly.
  • Re-running the same add syncs the template with the server, including removals, as git add <pathspec> has since Git 2.0.
  • A template that matches nothing is an error, and add -n previews the matches.
• The URL is stored on the template's data entry, not as an implicit data remote; git only creates remotes through clone and remote add. Entries under remote: are explicit mirrors, selected with download --remote NAME.
• Several templates per branch, local and remote mixed, for example .dat files in the worktree beside .h5 files on a server.
  • Each template has its own TSV. The first keeps data.tsv, so single-template repositories are unchanged.
  • hallmark rm --cached TEMPLATE stops tracking one and keeps its files.
  • Behavior change: a new local template is now added beside the existing ones instead of replacing the catalog.
• Remote templates are catalog-only.
  • Commit stores no objects for them.
  • Checkout never removes or restores downloaded copies.
  • Status summarizes their changes per template, and never reports absent or downloaded copies, similar to paths outside a sparse checkout.
• Downloads have a source per file. Each template's files come from its URL, so one plan can span several servers, and the plan lists each source with its file count.
• New hallmark ls-remote URL lists a directory and prints suggested add commands using the existing filename-format detector. It is read-only and works outside a repository.
• --filter is renamed --include on download and clone, and filter= becomes include= in Python. git clone --filter selects objects for a partial clone, which is a different meaning.
• Fixed an existing status bug that printed an empty "Changes not staged for commit:" heading.

Review guide

1. Store each template's catalog in its own TSV: state, dothm, repo_config and repo_manifest, with no behavior change.
2. Status, commit and checkout across templates
3. Accumulating templates, rm --cached and add -n
4. A download source per file
5. URL templates in add: the new repo_remote.py, template matching in discovery.py, and a directory-pruning hook in the HTTP and SSH backends.
6. ls-remote, and removal of init --from
7. Smaller commits:
   • --include rename
   • status fix
   • docs and demos
   • Python 3.9 test fix

Validation

• Tests: 993 pass on both Python 3.9 (Click 8.1.8) and Python 3.13. Both runs include 108 transport tests against a loopback OpenSSH server. Ruff and the strict Sphinx build pass.
• CI: the run on a30ede2 failed only on Python 3.9. One new CLI test assumed Click's stdout excludes stderr, but Click 8.1 mixes them. 53e85f5 fixes the test; the other seven jobs, including both SSH jobs, passed.
• End-to-end run against two local HTTP servers:
  • ls-remote and add -n;
  • two remote templates from different servers, and a template that matched nothing;
  • a commit that stored no objects, then a download from two sources;
  • a re-add that dropped a file removed upstream;
  • a checkout round trip with a local template, and rm --cached.
• Demos: the four scientific CLI workflows and all 11 notebook cells ran against local fixtures that mirror the EHT, DESI and export paths. The private-data demo ran against a loopback SFTP-only export.

No research servers were contacted and no public datasets were downloaded.

Known limits:

• The local layout mirrors the template: the URL up to the first templated segment becomes the root, so include a directory in the template to keep it locally.
• After switching branches, download again to refresh a stale copy.
• ls-remote lists the whole tree below its URL.

[denniswu28]

Superseded