[denniswu28]

Summary

Brings the remote-data path of ram/clean-up in line with the team's updated flowchart (Phase 1: private-data workflow). This replaces porting feat/private-data (#52) commit by commit. Ram's restructure and the flowchart agree on the model (hm add <url-or-pattern> stages a catalogue, one pattern per branch, and clone prompts then downloads), so this builds on Ram's code and ports by hand only a few pieces of #52 (2eb1635's URL checks and no-match error, 7da898c's unknown-path check). Stacked on ram/clean-up; Ram's branch is untouched.

One commit per flowchart rule, each with tests and docs (10 commits):

1. Find the repository from any folder inside it. Repo.find; CLI download paths are relative to the current folder. The CLI runs hm add . (and ./, .., or the absolute root) from the worktree root, so subfolder use cannot expose hm add . from a subfolder drops staged files outside it #53.
2. Explain damaged repositories and stop without writing. Missing or corrupt state files raise a clear DothmError, and opening a repository never writes.
3. Keep credentials out of remote catalogue and clone URLs. Passwords, HTTP usernames, query strings and fragments are rejected before any network access, and errors never echo the URL. Clone sources with HTTP credentials are refused too; previously the token was saved in .hm/.git/config.
4. Report remote scans that match nothing, and keep the previous catalogue.
5. Refuse remote patterns that would drop catalogued files. There is no force option; a broader pattern is accepted.
6. Select downloads by catalogued paths and folders. Unknown paths and empty folders are rejected before any server is contacted. --filter/--fmt alone is a usage error.
7. Skip verified files and refuse downloads that would replace different contents. Existing files with the same checksum are skipped. Without a checksum, a file whose catalogued size matches is skipped as "present, not verified", so interrupted downloads from manifest-less SSH exports can resume. A different size or different contents is a conflict, and then nothing is downloaded. Sizes are compared before hashing, unknown digests are typed by length, and the hashing pool is capped.
8. Clone into a default folder and refuse unsafe destinations. [DIRECTORY] is optional and an empty folder is accepted; nested or overlapping destinations are refused. A failed clone removes only what it created and never follows a symlink.
9. Refuse to clone repositories that track local files. "Nothing was created" (MVP rule), including any parent folders it made; empty repositories still clone. Source branch names are validated, so option-like names such as --output=x cannot reach git.
10. Copy branch pointers and validate the catalogue when cloning. Every branch is copied and validated, and an incomplete clone says how to retry.

Decisions for the team

• An existing file with no catalogue checksum is skipped when its catalogued size matches, and is a conflict when the size differs or no size is recorded.
• A local catalogue with a mirror remote_url counts as "tracks local files" and cannot be cloned.
• --dry-run exits 0 even when it reports unknown paths or conflicts.
• CLI download paths are relative to the current folder.
• A file that appears at its destination mid-download fails alone; the others continue.
• download --tsv is kept for published catalogues with several tables.

Open questions

1. Demo server checksums. The second download in each demo notebook assumes the shared SSH demo server publishes checksums; otherwise existing files become conflicts. The notebooks were not run (they need the private server).
2. Git clone credentials. Cloning from https://token@… stores the token in .hm/.git/config; commit 3 covers data URLs only.
3. Detached HEAD sources. Cloning a source on a directly selected commit was not changed; it belongs with Phase 2.

Not in this PR (Phase 2/3, for follow-up branches): no automatic first commit, hm config user.*, hm branch <name>, detached checkout, add by path, restore --staged, catalogue browsing (undecided).

Known limitations (not changed here)

• The mid-transfer guard is check-then-replace, so a file created in that instant can still be overwritten.
• hm set-config --fmt can set a pattern that catalogued files do not fit (local pattern rules are Phase 3).
• Validating every branch on clone is per-row: about 5 s per 100k rows.
• A source on a detached commit clones as detached, but status/branches then fail (Phase 2).
• hm init inside an existing repository creates a nested one.
• The Repo.add(".") API bug from a subfolder (hm add . from a subfolder drops staged files outside it #53) remains; only the CLI is guarded.

Testing

• pytest: 756 passed, 18 skipped (baseline 622/18). With HALLMARK_RUN_SSH_TESTS=1: 774 passed (baseline 640). Each commit passes pytest and ruff on its own.
• An independent code review's findings were fixed in the commits they concern, each with a regression test: resume without checksums, clone credentials, clone rollback scope, option-like branch names, root-equivalent hm add inputs, and dangling .hm links.
• Sphinx -W and ruff pass.
• Merged with Automate CI coverage, chaos and survey workflow tests #59 (CI, chaos and survey tests): 836 passed on Python 3.13 and 835 on 3.9 with SSH enabled, 7 strict expected failures and no unexpected passes. The survey and fault-injection suites hold under these changes.
• CI on GitHub runs here once Automate CI coverage, chaos and survey workflow tests #59's trigger change (pull requests into any branch) is in the base. Until then this was verified locally as above.

Supersedes #52.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[denniswu28]

This is still waiting on clean up of the original repo, and several design choices will need to be reconciled. Mostly is the pattern recognition and how to deal with multiple patterns.

[denniswu28]

Retiring this draft. Work has moved from ram/clean-up to codex/function-atomic-replay-20260928. The commits stay on feat/private-data-v2 for anything worth porting to the new base.