Feat/private data

.github/workflows/ci.yml

@@ -5,9 +5,9 @@ name: CI
 
 on:
   push:
-    branches: [ "main" ]
+    branches: [ "main", "dev" ]
   pull_request:
-    branches: [ "main" ]
+    branches: [ "main", "dev" ]
 
 jobs:
   build:
@@ -35,3 +35,24 @@ jobs:
     - name: Test with pytest
       run: |
         pytest
+
+  ssh-transport:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v3
+      with:
+        python-version: "3.9"
+    - name: Install Linux SSH server
+      if: runner.os == 'Linux'
+      run: sudo apt-get update && sudo apt-get install -y openssh-server
+    - name: Install Hallmark and tests
+      run: python -m pip install -e . pytest
+    - name: Exercise disposable OpenSSH server and client
+      env:
+        HALLMARK_RUN_SSH_TESTS: "1"
+      run: python -m pytest test/test_ssh_integration.py test/test_transport.py -q

.gitignore

@@ -17,5 +17,10 @@ __pycache__/
 # Ignore Sphinx build output by default
 _build/
 
-/demo/
-/cli_test_repos/
+/demo/*
+!/demo/private_data_demo.md
+!/demo/multi_server_backend.py
+!/demo/scientific_workflows_cli.md
+!/demo/scientific_workflows_python.ipynb
+/cli_test_repos/
+doc/temp.md

README.rst

@@ -46,7 +46,7 @@ Features:
 -------------
 
 Examples of using ``ParaFrame`` with Python API or Command Line Interface (CLI)
-can be found in the Jupyter Notebook tutorials in the ``demos`` folder.
+can be found in the Jupyter Notebook tutorials in the ``demo`` folder.
 
 ``Installation``
 -----------------
@@ -69,4 +69,76 @@ Or install from source for development purposes::
 ..  _hallmark: https://github.com/l6a/hallmark
 ..  _yukon:    https://github.com/l6a/yukon
 ..  _banyan:   https://github.com/l6a/banyan
-..  _eht:      https://eventhorizontelescope.org
+..  _eht:      https://eventhorizontelescope.org
+
+Private data remotes
+--------------------
+
+|hallmark|_ can catalog and download data products stored on HTTP(S), SSH, and
+SFTP servers. A data source specifies where the files are stored, while a Git
+remote is used to share the history of the data index.
+For a private server, |hallmark|_ uses OpenSSH 9.6 or newer with a trusted
+host key and key-based authentication that does not require a prompt.
+
+Add remote files to a repository with a URL template, then select files to
+download::
+
+    hallmark init eht
+    cd eht
+    hallmark add 'https://archive.example.org/2026MOVIE/ER2/{src}_{day}.h5'
+    hallmark commit -m 'Add remote EHT data'
+    hallmark download --all --dry-run
+    hallmark download --all
+
+``add`` lists the remote directory and catalogs the matching files, with their
+sizes and published checksums, without downloading dataset contents. The URL
+up to the first path segment containing a ``{field}`` is recorded as the
+template's source; the rest is the filename template, and its fields become
+catalog columns. Adding the same URL template again syncs the catalog with the
+server, including files that were removed. A branch can track several
+templates, local or remote, such as ``{run}.dat`` files in the worktree and
+``.h5`` files on a server; ``hallmark rm --cached TEMPLATE`` stops tracking
+one. Remote files are committed as catalog entries, not as local objects.
+
+Use ``hallmark ls-remote URL`` to list a directory and print suggested
+templates, and ``hallmark add -n 'URL/TEMPLATE'`` to preview the matches.
+CyVerse and ordinary browsable HTTPS directories, including DESI, use the same
+workflow. SSH listing also works with SFTP-only accounts; no server shell or
+Python is required.
+
+Every dataset transfer requires approval. Python callers can inspect
+``repo.plan_download()`` and then execute ``repo.download(plan, approved=True)``.
+CLI users can run ``hallmark download --interactive`` to choose path globs or
+all cataloged files, review recorded sizes, then download, change the selection,
+or skip. ``init`` and ``add`` never offer downloads. CLI cloning copies the
+complete catalog, then displays a download plan and requests confirmation by
+default. Use ``clone --interactive`` for the chooser, or ``clone --no-download``
+to copy only the catalog. Without a terminal, clone keeps the catalog, skips
+downloading, and prints commands to use later.
+SSH and SFTP use standard ``~/.ssh/config`` aliases and the SSH agent.
+Use ``hallmark sources desi`` to list the release and collection URLs of a
+named source for use with ``ls-remote`` and ``add``. Download selection belongs
+to the separate ``download --include`` command.
+The older remote ``build`` command is deprecated in favor of ``add``.
+
+Use ``hallmark clone CATALOG PATH`` for an existing Git-hosted ``.hm`` or a
+published HTTP/SFTP catalog snapshot. Catalogs can live on GitHub or another
+server while their data sources point elsewhere. Git clones preserve the full
+catalog and its history. ``clone --include`` narrows the planned downloads
+without removing catalog entries. No payloads transfer until the user approves
+the plan.
+
+The public ``DataBackend`` interface supports generic HTTP/SSH, CyVerse, and
+registered plugins for collaboration-specific APIs and multiple data servers.
+See `Data backends <doc/backends.rst>`_ for configuration and an extension example.
+
+Examples and configuration details are described in
+`Private data over SSH and SFTP <doc/private_data.rst>`_.
+
+For a sequential CLI walkthrough, see the
+`private data demo <demo/private_data_demo.md>`_. It includes catalog
+initialization, SSH configuration and explicit download approval.
+
+The `scientific CLI workflows <demo/scientific_workflows_cli.md>`_ and
+`scientific Python notebook <demo/scientific_workflows_python.ipynb>`_
+follow four examples using EHT, DESI, Roman–Rubin and local DES data.