Skip to content

Add coverage, Codecov and dependency-bounds CI to the codex branch - #61

Merged
denniswu28 merged 11 commits into
codex/function-atomic-replay-20260928from
ci/coverage-dependency-bounds
Oct 12, 2026
Merged

denniswu28 merged 11 commits into
codex/function-atomic-replay-20260928from
ci/coverage-dependency-bounds

Conversation

@denniswu28

@denniswu28 denniswu28 commented Oct 10, 2026 •

Copy link
Copy Markdown

Summary

This branch adds coverage with Codecov, a broader CI test matrix, and a weekly dependency-bounds workflow. It's stacked on codex/function-atomic-replay-20260928. CI has never run on this line of branches, because ci.yml only triggered for main and dev.

Taken from #59 (cherry-picked with -x, CI plumbing only)

  • One test matrix: Python 3.9–3.14 on Ubuntu, plus 3.9 and 3.14 on macOS. Every job installs sshd and runs the whole suite with HALLMARK_RUN_SSH_TESTS=1. This replaces the separate build and ssh-transport jobs.
  • Broader triggers: CI now runs on pull requests into any branch, and on manual dispatch.
  • Lint and packaging: a separate lint job, plus twine check --strict on the built distributions.
  • Coverage: branch coverage of the hallmark package from every job, combined in a coverage job with a 90% floor. It's uploaded to Codecov through OIDC, and codecov.yml marks Codecov's statuses informational.
  • Python 3.9 test fix: the Click 8.1 CliRunner handling in test_cli.py.
  • Not ported: the chaos and survey suites and nightly.yml stay in Automate CI coverage, chaos and survey workflow tests #59. Once this merges, Automate CI coverage, chaos and survey workflow tests #59 can drop these five commits.

New here

  • The Codecov upload works now. On Automate CI coverage, chaos and survey workflow tests #59 the action never downloaded its CLI from cli.codecov.io, and fail_ci_if_error: false hid that, so Codecov never received a report. The CLI now comes from PyPI (use_pypi: true). A failed upload marks its step failed without failing the job. The first upload from this PR went through OIDC, and Codecov now lists the repository as active.
  • Coverage is reported even when tests fail. Before, one failing test skipped the combined report, the Codecov upload and the floor. The test jobs still fail on their own.
  • Dependency lower bounds in pyproject.toml: click>=8.1, GitPython>=3.1.30, numpy>=1.21, pandas>=1.5, parse>=1.19, pyyaml>=5.4, requests>=2.26, tqdm>=4.60.
    • Before this there were no bounds, so pip could install pandas 1.3, where to_csv(lineterminator=...) raises a TypeError in repo/dothm.py.
    • These are tested floors, not proven minimums.
  • New Dependencies workflow (.github/workflows/dependencies.yml):
    • When it runs: weekly, on demand, and on PRs that change pyproject.toml or the workflow.
    • lowest job (Python 3.9): installs the oldest versions pyproject.toml allows, resolved with uv pip compile --resolution lowest-direct.
    • pre-release job (Python 3.14 and 3.15): installs the newest releases, including pre-releases.
    • Both jobs: run the whole suite with SSH and list the installed versions in the job summary.
    • Caveat: the weekly schedule only starts once the workflow is on main.
  • Coverage floor 89%: the combined total on this PR's first run was 90.45%, so the floor is set one point below the whole-number total, following Automate CI coverage, chaos and survey workflow tests #59's rule.
  • Authors: adds Tianrui (Dennis) Wu and Arshia Akhtarkavan to pyproject.toml.

Existing failures (CI is red, not caused by this PR)

The base branch tip (a449faa) has 27 failing tests, plus 2 more with Click ≥ 8.5 (every job except Python 3.9). That's 27 failures on the 3.9 jobs and 29 on the others. 326d4fb (the codex branch before the merge) passes all 674. These failures are reported here and not fixed, to keep this PR to CI.

From ee6cbf5 (Add file and folder staging with staged restore), 7:

  • test_demo_notebooks.py: demo_cli.ipynb, demo_python.ipynb
  • test_hallmark.py:
    • test_checkout_allows_return_to_branch_when_target_files_already_match
    • test_checkout_rebuilds_worktree_for_branch_specific_nested_fmt
    • test_checkout_rejects_directory_at_target_file_path
    • test_checkout_rejects_symlink_destination_escape
    • test_manifest_entries_empty_when_config_has_no_data_fmt

First failing at the merge a449faa, 20:

  • test_cli.py: test_cli_add_dot_and_explicit_paths, test_cli_set_config_and_add_dot
  • test_hallmark.py:
    • test_checkout_remote_branch
    • test_data_tsv_and_worktree_reconstruction
    • test_load_head_state_without_commits_uses_current_config_and_empty_data
    • test_manifest_frame_normalizes_missing_and_integral_float_values
    • test_repo_add_dot_from_nested_directory_uses_path_components
    • test_repo_add_dot_removes_deleted_files_from_manifest
    • test_repo_add_dot_replaces_manifest_with_current_tree
    • test_repo_add_parse_failure_preserves_existing_format
    • test_repo_add_paths_is_not_supported_yet
    • test_repo_add_pattern_keeps_deleted_manifest_rows
    • test_repo_add_pattern_replaces_manifest_when_fmt_changes
    • test_repo_add_persists_only_sha1_and_path
    • test_repo_add_result_has_expected_length
    • test_repo_add_result_paths_match_standard_files
    • test_repo_status_does_not_walk_dothm_directory
    • test_state_replace_empty_uses_incoming_schema
    • test_state_update_and_replace_share_data_normalization
    • test_state_update_static_format_keeps_only_latest_row

With Click 8.5.0 (released stable), 2 more:

  • test_local_add.py::test_add_has_no_force_or_all_option[--force] and [--all]. Line 397 expects No such option: --force, but Click 8.5 prints No such option '--force'.. The pre-release job caught this first.

Most of them expect the old manifest layout (no path column) or hit No file pattern set after staging by path.

Testing

  • ruff check . passes; actionlint passes on ci.yml and dependencies.yml.
  • Full suite locally (Python 3.13, SSH tests on): 27 failed, 735 passed, 5 xfailed. These are exactly the 27 above.
  • Branch coverage locally: 90.24%.
  • CI on this PR: lint, distribution-and-docs and coverage pass. All 8 test jobs ran the whole suite with SSH and failed only on the tests listed above.
    • The coverage job ran despite the failures: 90.45% combined, uploaded to Codecov.
  • Dependencies on this PR: lowest hit only the 27; pre-release (3.14 and 3.15, pandas 3.1.0rc0, Click 8.5.0) hit the 27 plus the 2 Click failures.
    • Click also warns that CliRunner.isolated_filesystem, used in test_cli.py, is removed in Click 9.0.
  • Reproduced the lowest job's steps locally on Python 3.9. It resolved click 8.1.0, GitPython 3.1.30, numpy 1.21.0, pandas 1.5.0, parse 1.19.0, pyyaml 5.4, requests 2.26.0 and tqdm 4.60.0. pip check was clean, and the run had the same 27 failures and no others. With pandas 1.3 there were extra lineterminator failures.

🤖 Generated with Claude Code

Configure coverage.py to measure the hallmark package with branch
coverage and repository-relative paths, so data from separate CI jobs on
Linux and macOS can be combined. Add pytest-cov and pytest-timeout to the
dev dependency group and document `pytest --cov` for local runs.

(cherry picked from commit 2852809)
Replace the separate build and ssh-transport jobs with one test matrix
that installs the OpenSSH server and runs the complete suite with
HALLMARK_RUN_SSH_TESTS=1 on Python 3.9-3.14 (Linux) and 3.9/3.14 (macOS).
Run CI for pull requests into any branch so stacked feature branches are
tested, lint once in its own job, update the actions to their current
majors, and check the built distributions with twine.

Move the ruff exclusions to the top-level table so `ruff check .` no
longer parses the shell cells in the demo notebooks.

(cherry picked from commit efb2bc9)
Click 8.1, the newest release for Python 3.9, makes CliRunner read
exhausted input as an empty line instead of raising EOFError, so an empty
answer to the clone download prompt declines rather than aborts there.
Real terminals abort on end of input with both versions; only the test
harness differs. Expect the empty-line behaviour when Click is older than
8.2 so the Python 3.9 CI job tests the same outcomes.

(cherry picked from commit aa6f596)
Each test job writes coverage data under its OS and Python version and
uploads it. A coverage job combines them, writes the report to the job
summary, uploads XML and HTML reports, and fails below 90%. Some branches
only run on one Python version or operating system, so no single job
sees the total. The floor lives in the workflow rather than in
pyproject.toml because pytest-cov would apply it to each partial run.

(cherry picked from commit a4c4ccf)
Upload the combined coverage.xml from the coverage job, verified with
OIDC so no repository secret is needed; pull requests from forks upload
tokenless. The upload cannot fail CI, and codecov.yml marks Codecov's
statuses informational, so the floor in the workflow remains the only
gate. Add a coverage badge and describe the coverage workflow in the
contribution guide.

The Codecov GitHub App must be installed on the lanisrc organization for
uploads to be accepted.

(cherry picked from commit 015f035)
Both CI runs on #59 failed to download the Codecov CLI from
cli.codecov.io, and fail_ci_if_error: false hid that, so Codecov has
never received a report. Install the CLI from PyPI instead, and let an
upload error fail the step but not the job, so a rejected or failed
upload is visible while the in-workflow floor stays the only gate.
The coverage job needed every test job to succeed, so one failing test
skipped the combined report, the Codecov upload and the floor. Upload
each job's coverage data and combine it unless the run was cancelled.
The test jobs still fail on their own.
The dependencies had no lower bounds, so pip could install pandas 1.3,
whose DataFrame.to_csv() has no lineterminator argument; staging then
fails with a TypeError in Dothm. Require pandas 1.5, which added it, and
bound the other dependencies at releases that pass the suite on Python
3.9 when installed at exactly these versions. They are tested floors,
not proven minimums.
Add a Dependencies workflow that runs weekly, on demand, and on pull
requests that change pyproject.toml or the workflow. One job installs
the oldest versions pyproject.toml allows on Python 3.9, resolved with
uv's lowest-direct strategy so the test tools stay current. The other
installs the newest releases, including pre-releases, on Python 3.14 and
3.15. Both run the whole suite with the loopback SSH tests and list the
installed versions in the job summary.
@codecov-commenter

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

The first CI run on this branch combined 90.45% branch coverage across
all eight test jobs, with 27-29 tests failing in each. A floor of 90
would leave under half a point of margin, so use #59's rule of one
point below the whole-number total: 89.
@denniswu28
denniswu28 force-pushed the ci/coverage-dependency-bounds branch from 34dff32 to 983f357 Compare October 10, 2026 22:01
@denniswu28
denniswu28 marked this pull request as ready for review October 10, 2026 22:02
@denniswu28
denniswu28 merged commit 78dff8c into codex/function-atomic-replay-20260928 Oct 12, 2026
3 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants