Repository navigation
Add coverage, Codecov and dependency-bounds CI to the codex branch - #61
Merged
denniswu28 merged 11 commits intoOct 12, 2026
Merged
denniswu28 merged 11 commits into
denniswu28 merged 11 commits into
Conversation
Configure coverage.py to measure the hallmark package with branch coverage and repository-relative paths, so data from separate CI jobs on Linux and macOS can be combined. Add pytest-cov and pytest-timeout to the dev dependency group and document `pytest --cov` for local runs. (cherry picked from commit 2852809)
Replace the separate build and ssh-transport jobs with one test matrix that installs the OpenSSH server and runs the complete suite with HALLMARK_RUN_SSH_TESTS=1 on Python 3.9-3.14 (Linux) and 3.9/3.14 (macOS). Run CI for pull requests into any branch so stacked feature branches are tested, lint once in its own job, update the actions to their current majors, and check the built distributions with twine. Move the ruff exclusions to the top-level table so `ruff check .` no longer parses the shell cells in the demo notebooks. (cherry picked from commit efb2bc9)
Click 8.1, the newest release for Python 3.9, makes CliRunner read exhausted input as an empty line instead of raising EOFError, so an empty answer to the clone download prompt declines rather than aborts there. Real terminals abort on end of input with both versions; only the test harness differs. Expect the empty-line behaviour when Click is older than 8.2 so the Python 3.9 CI job tests the same outcomes. (cherry picked from commit aa6f596)
Each test job writes coverage data under its OS and Python version and uploads it. A coverage job combines them, writes the report to the job summary, uploads XML and HTML reports, and fails below 90%. Some branches only run on one Python version or operating system, so no single job sees the total. The floor lives in the workflow rather than in pyproject.toml because pytest-cov would apply it to each partial run. (cherry picked from commit a4c4ccf)
Upload the combined coverage.xml from the coverage job, verified with OIDC so no repository secret is needed; pull requests from forks upload tokenless. The upload cannot fail CI, and codecov.yml marks Codecov's statuses informational, so the floor in the workflow remains the only gate. Add a coverage badge and describe the coverage workflow in the contribution guide. The Codecov GitHub App must be installed on the lanisrc organization for uploads to be accepted. (cherry picked from commit 015f035)
Both CI runs on #59 failed to download the Codecov CLI from cli.codecov.io, and fail_ci_if_error: false hid that, so Codecov has never received a report. Install the CLI from PyPI instead, and let an upload error fail the step but not the job, so a rejected or failed upload is visible while the in-workflow floor stays the only gate.
The coverage job needed every test job to succeed, so one failing test skipped the combined report, the Codecov upload and the floor. Upload each job's coverage data and combine it unless the run was cancelled. The test jobs still fail on their own.
The dependencies had no lower bounds, so pip could install pandas 1.3, whose DataFrame.to_csv() has no lineterminator argument; staging then fails with a TypeError in Dothm. Require pandas 1.5, which added it, and bound the other dependencies at releases that pass the suite on Python 3.9 when installed at exactly these versions. They are tested floors, not proven minimums.
Add a Dependencies workflow that runs weekly, on demand, and on pull requests that change pyproject.toml or the workflow. One job installs the oldest versions pyproject.toml allows on Python 3.9, resolved with uv's lowest-direct strategy so the test tools stay current. The other installs the newest releases, including pre-releases, on Python 3.14 and 3.15. Both run the whole suite with the loopback SSH tests and list the installed versions in the job summary.
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment Thanks for integrating Codecov - We've got you covered ☂️ |
The first CI run on this branch combined 90.45% branch coverage across all eight test jobs, with 27-29 tests failing in each. A floor of 90 would leave under half a point of margin, so use #59's rule of one point below the whole-number total: 89.
denniswu28
force-pushed
the
ci/coverage-dependency-bounds
branch
from
October 10, 2026 22:01
34dff32 to
983f357
Compare
denniswu28
marked this pull request as ready for review
October 10, 2026 22:02
denniswu28
merged commit Oct 12, 2026
78dff8c
into
codex/function-atomic-replay-20260928
3 of 14 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch adds coverage with Codecov, a broader CI test matrix, and a weekly dependency-bounds workflow. It's stacked on
codex/function-atomic-replay-20260928. CI has never run on this line of branches, becauseci.ymlonly triggered formainanddev.Taken from #59 (cherry-picked with
-x, CI plumbing only)HALLMARK_RUN_SSH_TESTS=1. This replaces the separatebuildandssh-transportjobs.twine check --stricton the built distributions.hallmarkpackage from every job, combined in acoveragejob with a 90% floor. It's uploaded to Codecov through OIDC, andcodecov.ymlmarks Codecov's statuses informational.CliRunnerhandling intest_cli.py.nightly.ymlstay in Automate CI coverage, chaos and survey workflow tests #59. Once this merges, Automate CI coverage, chaos and survey workflow tests #59 can drop these five commits.New here
cli.codecov.io, andfail_ci_if_error: falsehid that, so Codecov never received a report. The CLI now comes from PyPI (use_pypi: true). A failed upload marks its step failed without failing the job. The first upload from this PR went through OIDC, and Codecov now lists the repository as active.pyproject.toml:click>=8.1,GitPython>=3.1.30,numpy>=1.21,pandas>=1.5,parse>=1.19,pyyaml>=5.4,requests>=2.26,tqdm>=4.60.to_csv(lineterminator=...)raises aTypeErrorinrepo/dothm.py.Dependenciesworkflow (.github/workflows/dependencies.yml):pyproject.tomlor the workflow.lowestjob (Python 3.9): installs the oldest versionspyproject.tomlallows, resolved withuv pip compile --resolution lowest-direct.pre-releasejob (Python 3.14 and 3.15): installs the newest releases, including pre-releases.main.pyproject.toml.Existing failures (CI is red, not caused by this PR)
The base branch tip (
a449faa) has 27 failing tests, plus 2 more with Click ≥ 8.5 (every job except Python 3.9). That's 27 failures on the 3.9 jobs and 29 on the others.326d4fb(the codex branch before the merge) passes all 674. These failures are reported here and not fixed, to keep this PR to CI.From
ee6cbf5(Add file and folder staging with staged restore), 7:test_demo_notebooks.py:demo_cli.ipynb,demo_python.ipynbtest_hallmark.py:test_checkout_allows_return_to_branch_when_target_files_already_matchtest_checkout_rebuilds_worktree_for_branch_specific_nested_fmttest_checkout_rejects_directory_at_target_file_pathtest_checkout_rejects_symlink_destination_escapetest_manifest_entries_empty_when_config_has_no_data_fmtFirst failing at the merge
a449faa, 20:test_cli.py:test_cli_add_dot_and_explicit_paths,test_cli_set_config_and_add_dottest_hallmark.py:test_checkout_remote_branchtest_data_tsv_and_worktree_reconstructiontest_load_head_state_without_commits_uses_current_config_and_empty_datatest_manifest_frame_normalizes_missing_and_integral_float_valuestest_repo_add_dot_from_nested_directory_uses_path_componentstest_repo_add_dot_removes_deleted_files_from_manifesttest_repo_add_dot_replaces_manifest_with_current_treetest_repo_add_parse_failure_preserves_existing_formattest_repo_add_paths_is_not_supported_yettest_repo_add_pattern_keeps_deleted_manifest_rowstest_repo_add_pattern_replaces_manifest_when_fmt_changestest_repo_add_persists_only_sha1_and_pathtest_repo_add_result_has_expected_lengthtest_repo_add_result_paths_match_standard_filestest_repo_status_does_not_walk_dothm_directorytest_state_replace_empty_uses_incoming_schematest_state_update_and_replace_share_data_normalizationtest_state_update_static_format_keeps_only_latest_rowWith Click 8.5.0 (released stable), 2 more:
test_local_add.py::test_add_has_no_force_or_all_option[--force]and[--all]. Line 397 expectsNo such option: --force, but Click 8.5 printsNo such option '--force'.. Thepre-releasejob caught this first.Most of them expect the old manifest layout (no
pathcolumn) or hitNo file pattern setafter staging by path.Testing
ruff check .passes;actionlintpasses onci.ymlanddependencies.yml.lint,distribution-and-docsandcoveragepass. All 8 test jobs ran the whole suite with SSH and failed only on the tests listed above.coveragejob ran despite the failures: 90.45% combined, uploaded to Codecov.lowesthit only the 27;pre-release(3.14 and 3.15, pandas 3.1.0rc0, Click 8.5.0) hit the 27 plus the 2 Click failures.CliRunner.isolated_filesystem, used intest_cli.py, is removed in Click 9.0.lowestjob's steps locally on Python 3.9. It resolved click 8.1.0, GitPython 3.1.30, numpy 1.21.0, pandas 1.5.0, parse 1.19.0, pyyaml 5.4, requests 2.26.0 and tqdm 4.60.0.pip checkwas clean, and the run had the same 27 failures and no others. With pandas 1.3 there were extralineterminatorfailures.🤖 Generated with Claude Code