Skip to content

chore(conformance): fail on any failing variant; re-baseline at 257,288/257,288 - #2736

Merged
vieiralucas merged 4 commits into
mainfrom
chore/conformance-strict-rebaseline
Oct 17, 2026
Merged

vieiralucas merged 4 commits into
mainfrom
chore/conformance-strict-rebaseline

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Every implemented service now passes every variant on the refreshed AWS models, so the check enforces that.

Test plan

  • cargo +stable clippy -p fakecloud-conformance --all-targets -D warnings, fmt
  • conformance bin tests (112), lib tests (106), including every_failing_operation_is_a_check_failure
  • full local probe on main: 7610/7610 ops, 257288/257288 variants
  • bash scripts/check-doc-counts.sh OK; generate-operations-index.sh no diff
  • CI conformance-check runs the strict check against the new baseline

Summary by cubic

Conformance checks now fail on any failing or unchecked variant, making the 100% pass requirement enforceable in CI. The baseline is refreshed to 257,288 passing variants across 106 services.

  • The per-service baseline ratchet has no default flake margin; FAKECLOUD_CONFORMANCE_FLAKE_MARGIN can relax local runs.
  • The sweep server startup timeout is now 60 seconds to accommodate cold debug builds.
  • Updated published conformance counts to match the refreshed baseline.

Written for commit 3795ed0. Summary will update on new commits.

View guided diff Turn on auto-fix

- check reports every operation with a failing variant as a failure; the
  per-service ratchet keeps catching a service that loses variants, now with
  no default flake margin (FAKECLOUD_CONFORMANCE_FLAKE_MARGIN still relaxes
  local runs).
- The sweep server gets 60s to start (a cold debug binary missed 5s).
Every implemented service passes every variant on the refreshed models
(7,610 probed operations; RDS runtime-dependent variants re-probed against
a Docker-backed server). Variant counts synced across README, AGENTS.md,
docs, landing pages and llms.txt; check-doc-counts passes.
- Drop the overall ratchet: it could never fire before a per-service one.
- An operation with no variants counts as unchecked, so it fails the check.
- Status lines and the check/step descriptions name the new failure rule.
@vieiralucas
vieiralucas merged commit 91a2665 into main Oct 17, 2026
157 checks passed
@vieiralucas
vieiralucas deleted the chore/conformance-strict-rebaseline branch October 17, 2026 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant