Automate CI coverage, chaos and survey workflow tests

.github/workflows/ci.yml

@@ -1,79 +1,155 @@
-# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# Lints, tests every supported Python version on Linux and macOS, and checks
+# the built distributions and documentation.
 # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
 
 name: CI
 
 on:
   push:
     branches: [ "main", "dev" ]
+  # Pull requests into any branch, including stacked feature branches.
   pull_request:
-    branches: [ "main", "dev" ]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
-  build:
+  lint:
+    runs-on: ubuntu-24.04
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        persist-credentials: false
+    - uses: actions/setup-python@v7
+      with:
+        python-version: "3.13"
+    - name: Install ruff
+      run: python -m pip install ruff
+    - name: Lint with ruff
+      run: ruff check .
 
-    runs-on: ubuntu-latest
+  # Every job runs the whole suite, including the loopback OpenSSH tests, so
+  # each Python version and both operating systems exercise every module.
+  test:
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 25
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9", "3.10","3.11","3.12","3.13","3.14"]
-
+        # Pinned because Python 3.9 builds may not exist for newer images.
+        os: [ubuntu-24.04]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
+        include:
+        - os: macos-latest
+          python-version: "3.9"
+        - os: macos-latest
+          python-version: "3.14"
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v7
+      with:
+        persist-credentials: false
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v3
+      uses: actions/setup-python@v7
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install dependencies
+    - name: Install the OpenSSH server
+      if: runner.os == 'Linux'
+      run: sudo apt-get update && sudo apt-get install -y --no-install-recommends openssh-server
+    - name: Install Hallmark and test tools
       run: |
         python -m pip install --upgrade pip
-        python -m pip install ruff pytest
-        python -m pip install -e .
-    - name: Lint with ruff
-      run: |
-        ruff check --select E,F --exclude ./demo
+        python -m pip install -e . --group dev
     - name: Test with pytest
+      env:
+        HALLMARK_RUN_SSH_TESTS: "1"
+        COVERAGE_FILE: .coverage.${{ runner.os }}-py${{ matrix.python-version }}
+      run: python -m pytest --cov --cov-report= --timeout=600 -ra --durations=15
+    - name: Upload coverage data
+      uses: actions/upload-artifact@v7
+      with:
+        name: coverage-${{ runner.os }}-py${{ matrix.python-version }}
+        path: .coverage.${{ runner.os }}-py${{ matrix.python-version }}
+        include-hidden-files: true
+        if-no-files-found: error
+        retention-days: 7
+
+  # Combines coverage from every test job. Some branches only run on one
+  # Python version or operating system, so no single job sees the total.
+  coverage:
+    needs: test
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+      # Lets Codecov verify uploads with OIDC instead of a stored token.
+      id-token: write
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        persist-credentials: false
+    - uses: actions/setup-python@v7
+      with:
+        python-version: "3.13"
+    - name: Install coverage
+      run: python -m pip install coverage
+    - uses: actions/download-artifact@v8
+      with:
+        pattern: coverage-*
+        merge-multiple: true
+    - name: Combine and report coverage
       run: |
-        pytest
+        coverage combine
+        coverage report
+        coverage report --format=markdown >> "$GITHUB_STEP_SUMMARY"
+        coverage xml
+        coverage html
+    - name: Upload the combined report
+      uses: actions/upload-artifact@v7
+      with:
+        name: coverage-report
+        path: |
+          coverage.xml
+          htmlcov/
+    # Codecov needs its GitHub App installed on the organization. Until then
+    # the upload fails without failing CI; the floor below is the gate.
+    # Pull requests from forks cannot request OIDC tokens and upload tokenless.
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
+      with:
+        files: coverage.xml
+        disable_search: true
+        fail_ci_if_error: false
+        use_oidc: ${{ !github.event.pull_request.head.repo.fork }}
+    # The floor lives here rather than in pyproject.toml because pytest-cov
+    # would apply it to each partial run.
+    - name: Enforce the coverage floor
+      run: coverage report --fail-under=90
 
   distribution-and-docs:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v7
       with:
         fetch-depth: 0
-    - uses: actions/setup-python@v5
+        persist-credentials: false
+    - uses: actions/setup-python@v7
       with:
-        python-version: "3.12"
-    - name: Build source and wheel distributions
+        python-version: "3.13"
+    - name: Build and check source and wheel distributions
       run: |
-        python -m pip install build
+        python -m pip install build twine
         python -m build
+        python -m twine check --strict dist/*
     - name: Install the wheel and documentation tools
       run: python -m pip install dist/*.whl sphinx
-    - name: Check the installed CLI
-      run: hm --help
+    - name: Check both installed commands
+      run: |
+        hm --help
+        hallmark --help
       working-directory: ${{ runner.temp }}
     - name: Build documentation with warnings as errors
       run: python -m sphinx -b html -W --keep-going doc "${{ runner.temp }}/hallmark-docs"
-
-  ssh-transport:
-    runs-on: ${{ matrix.os }}
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ubuntu-latest, macos-latest]
-    steps:
-    - uses: actions/checkout@v4
-    - uses: actions/setup-python@v3
-      with:
-        python-version: "3.9"
-    - name: Install Linux SSH server
-      if: runner.os == 'Linux'
-      run: sudo apt-get update && sudo apt-get install -y openssh-server
-    - name: Install Hallmark and tests
-      run: python -m pip install -e . pytest
-    - name: Exercise disposable OpenSSH server and client
-      env:
-        HALLMARK_RUN_SSH_TESTS: "1"
-      run: python -m pytest test/test_ssh_integration.py test/test_transport.py -q

.github/workflows/nightly.yml

@@ -0,0 +1,113 @@
+# Runs the chaos tests with many more random operations and fresh seeds,
+# and downloads a few small files from public survey archives.
+
+name: Nightly
+
+on:
+  schedule:
+  - cron: "23 6 * * *"
+  workflow_dispatch:
+    inputs:
+      steps:
+        description: Random operations per seed
+        default: "2000"
+      seeds:
+        description: Comma-separated seeds (blank derives them from the run)
+        default: ""
+  # Scheduled and manual runs only start from the default branch, so pull
+  # requests that change these tests or this workflow run it too.
+  pull_request:
+    paths:
+    - .github/workflows/nightly.yml
+    - test/chaos.py
+    - test/chaos_child.py
+    - test/http_server.py
+    - test/survey.py
+    - test/test_chaos_*.py
+    - test/test_survey*.py
+
+permissions:
+  contents: read
+
+concurrency:
+  group: nightly-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  chaos:
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+        - os: ubuntu-24.04
+          python-version: "3.9"
+          lane: 1
+        - os: ubuntu-24.04
+          python-version: "3.14"
+          lane: 2
+        - os: macos-latest
+          python-version: "3.14"
+          lane: 3
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        persist-credentials: false
+    - uses: actions/setup-python@v7
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install the OpenSSH server
+      if: runner.os == 'Linux'
+      run: sudo apt-get update && sudo apt-get install -y --no-install-recommends openssh-server
+    - name: Install Hallmark and test tools
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install -e . --group dev
+    # Inputs reach the shell through the environment, never inline.
+    - name: Choose seeds and steps
+      env:
+        INPUT_SEEDS: ${{ inputs.seeds }}
+        INPUT_STEPS: ${{ inputs.steps }}
+        RUN_ID: ${{ github.run_id }}
+        LANE: ${{ matrix.lane }}
+        EVENT: ${{ github.event_name }}
+      run: |
+        seeds="${INPUT_SEEDS:-${RUN_ID}${LANE}1,${RUN_ID}${LANE}2,${RUN_ID}${LANE}3}"
+        if [ "$EVENT" = pull_request ]; then default_steps=300; else default_steps=2000; fi
+        steps="${INPUT_STEPS:-$default_steps}"
+        [[ "$seeds" =~ ^[0-9,]+$ && "$steps" =~ ^[0-9]+$ ]] || { echo "Invalid seeds or steps"; exit 1; }
+        echo "HALLMARK_CHAOS_SEED=$seeds" >> "$GITHUB_ENV"
+        echo "HALLMARK_CHAOS_STEPS=$steps" >> "$GITHUB_ENV"
+        echo "Chaos seeds: \`$seeds\`, steps per seed: $steps" >> "$GITHUB_STEP_SUMMARY"
+    - name: Run the chaos tests
+      env:
+        HALLMARK_RUN_SSH_TESTS: "1"
+      run: python -m pytest -m chaos -ra --timeout=1800 --basetemp="$RUNNER_TEMP/pytest"
+    - name: Keep failed repositories for inspection
+      if: failure()
+      uses: actions/upload-artifact@v7
+      with:
+        name: chaos-${{ runner.os }}-py${{ matrix.python-version }}
+        path: ${{ runner.temp }}/pytest
+        include-hidden-files: true
+        retention-days: 14
+
+  survey-live:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 20
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        persist-credentials: false
+    - uses: actions/setup-python@v7
+      with:
+        python-version: "3.13"
+    - name: Install Hallmark and test tools
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install -e . --group dev
+    - name: Download small files from public survey archives
+      env:
+        HALLMARK_RUN_LIVE_TESTS: "1"
+      run: python -m pytest -m survey_live -ra --timeout=600

.gitignore

@@ -7,6 +7,8 @@ venv/
 .pytest_cache/
 .ruff_cache/
 .coverage
+.coverage.*
+coverage.xml
 htmlcov/
 build/
 dist/

README.rst

@@ -3,6 +3,10 @@
 
     Reproducibility is the |hallmark|_ of the scientific method.
 
+..  image:: https://codecov.io/gh/lanisrc/hallmark/graph/badge.svg?branch=main
+    :target: https://codecov.io/gh/lanisrc/hallmark
+    :alt: Coverage
+
 Modern science has become so complex that many science projects rely
 on multiple software packages to work in unison, resulting in networks of
 data products along the analyses.
@@ -135,8 +139,8 @@ version, operating system, and a minimal example without credentials or private
 
 From a development checkout, install the validation tools and run::
 
-    python -m pip install -e . pytest ruff sphinx
-    python -m pytest
+    python -m pip install -e . --group dev --group lint --group doc
+    python -m pytest --cov
     ruff check --select E,F mod test
     python -m sphinx -b html -W doc doc/_build
 

codecov.yml

@@ -0,0 +1,25 @@
+# Codecov reports are informational; CI enforces the coverage floor in
+# .github/workflows/ci.yml from the combined report it uploads.
+codecov:
+  notify:
+    after_n_builds: 1
+coverage:
+  precision: 1
+  round: down
+  status:
+    project:
+      default:
+        target: auto
+        threshold: 1%
+        informational: true
+    patch:
+      default:
+        target: 80%
+        informational: true
+comment:
+  layout: "condensed_header, diff, flags, files"
+  after_n_builds: 1
+ignore:
+  - "test/**"
+  - "demo/**"
+  - "doc/**"

doc/CONTRIBUTING.md

@@ -28,6 +28,15 @@
 
 * If the implementation of a feature or fix is less clear than the desired outcome, write unit or end-to-end tests first to define the expected functionality. Then write the code to make those tests pass.
 * If the implementation is as clear as (or clearer than) the goal, implement the solution first. Then write tests to ensure the solution covers edge cases and realistic user scenarios. The tests may reveal places where the implementation needs to be fixed, expanded, or simplified.
+* Run `pytest --cov` to see which lines the tests miss, or `pytest --cov --cov-report=html` for a browsable report in `htmlcov/`. CI runs the whole suite, including the loopback SSH tests, on every supported Python version, combines coverage from every job, and fails if the total drops below the floor set in `.github/workflows/ci.yml`. Codecov comments on each pull request with the coverage of its changed lines; those comments are informational.
+
+## Fault Injection and Survey Workflow Tests
+
+
+* `pytest -m survey` runs end-to-end workflows from `doc/usecase.rst` against scaled-down DESI-like and EHT-like releases on a loopback HTTP server, and over a loopback SSH server when `HALLMARK_RUN_SSH_TESTS=1`. These tests refuse any request that would leave the machine.
+* `pytest -m chaos` injects network faults, full disks, I/O errors and process kills, and runs seeded random operation sequences (the "monkey") against repository invariants. Set `HALLMARK_CHAOS_SEED` (comma-separated) and `HALLMARK_CHAOS_STEPS` to explore further; a failing monkey prints the command that reproduces it.
+* `HALLMARK_RUN_LIVE_TESTS=1 pytest -m survey_live` downloads a few megabytes from DESI DR1 and the EHT release on CyVerse. CI runs it, and the chaos tests with many more steps, every night.
+* When these tests expose a bug, reduce it to a small deterministic test marked `xfail(strict=True)` with the reason and issue, and fix it on its own branch. Strict expected failures fail CI once the bug is fixed, so the fix must remove the marker.
 
 ## Data Management
 

doc/usecase.rst

@@ -357,5 +357,5 @@ SSH configuration and known-hosts files.
 They do not connect to a lab server or change the user's SSH trust files.
 Enabling the tests requires the OpenSSH tools to be installed; missing
 tools cause a test failure.
-CI installs the server and runs these tests separately from the
-Python-version matrix.
+CI installs the server and runs the complete suite, these tests included,
+in every Linux and macOS test job.