Skip to content

T6: run the Verify suite five times every night on Linux and macOS, and name every flaky test - #124

Merged
SwayamInSync merged 2 commits into
mainfrom
ci/stability-nightly
Oct 29, 2026
Merged

SwayamInSync merged 2 commits into
mainfrom
ci/stability-nightly

Conversation

@SwayamInSync

Copy link
Copy Markdown
Owner

Plan item T6 (stability CI), for exit criterion 3 (steady verdicts): nothing measured whether the Verify suite gives the same results run after run, and flaky tests showed up only as red marks on unrelated pull requests (confirmation_invariants.cpp on #105 and #110).

With this PR

cppverify-stability.yml runs every night (04:17 UTC) and on demand:

  • Linux (ubuntu-24.04, the release configuration with cvc5 1.4.2, ccache from CI's cache) and macOS (macos-14, setup.sh, as CI's macOS job);
  • the Verify suite five times on each, at the parallelism CI uses, each run's results kept as an artifact (lit -o);
  • cppverify-stability.py compares the runs and fails the job when a test is not PASS, UNSUPPORTED, or XFAIL in some run, naming it with its result in each run, as flaky (results differ) or failing (the same failure every time):
2 tests, 3 runs: 1 flaky, 0 failing in every run

Flaky:
  Clang :: Verify/dump_ir.cpp: PASS PASS FAIL

(the script on two real runs of two tests and a third with one result changed by hand).

Seeds are not part of it: cpp-verify does not expose the solver's random seed, and a test that checks a counterexample's values would change with it although its verdict does not.

Not tested here

A workflow can be started by hand only once it is on the default branch, so its first run will be after the merge; it reuses the build steps of cppverify-differential.yml (Linux) and CI's macOS job unchanged.

@SwayamInSync
SwayamInSync merged commit 9e66610 into main Oct 29, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant