ci: gate PR merges on tri-comparison validated-precision regression - #2139
Merged
Conversation
Adds tests/tools/tri_comparison_chart.py's --ci/--regenerate mode, gated on GitGalaxy's own ledger-validated precision (never a raw disagreement count) -- recall is deliberately excluded, since the chart's own docstring already explains why that ratio's cross-tool denominator isn't trustworthy to rank on. Two new workflows mirror the existing tree-sitter-accuracy split: tri-comparison-audit.yml (PR-time, blocking, measures only) and tri-comparison-history.yml (push-to-main, regenerates chart/ledger/report, opens an auto-merged PR only when something actually changed). Both hard-fail if ctags isn't really Universal Ctags, per the PR #2111 incident. Updates ci-push-checklist and tree-sitter-accuracy-sweep skills, the tri-comparison README, and CLAUDE.md's Differential Scan section to reflect the new gate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the tri-comparison system (GitGalaxy vs. tree-sitter vs. ctags) CI-enforced for any PR
touching core-engine parsing logic, instead of skill/human-driven only. Two parts, following the
already-proven
tree-sitter-accuracy-audit.yml/-history.ymlsplit rather than inventing a newmechanism:
tests/tools/tri_comparison_chart.py --ci/--regenerate: baseline-gated regression checkon GitGalaxy's own validated precision (
func_precision/class_precision, read afterapply_verified_adjustments()applies any ledger verdict — never a raw, unvalidateddisagreement count, per this repo's "comparative-correctness claims require verification" rule).
New per-language baseline files,
tests/tri_comparison_baseline_<lang>.json, populatedincrementally via
--regenerate(none bundled in this PR — that's a deliberate follow-up oncethis scaffolding lands, not bulk-generated here). Recall is deliberately not gated — the
chart's own module docstring already explains why that ratio's cross-tool denominator isn't
trustworthy enough to rank on (a bug in either other tool can inflate it without GitGalaxy
doing anything wrong); gating CI on it would reintroduce the exact problem the chart already
designed around.
.github/workflows/tri-comparison-audit.yml(PR-time, blocking, same core-engine pathstree-sitter-accuracy-audit.ymlalready uses) runs the--cicheck. It only ever measures —it never requires a contributor to regenerate and commit the chart/ledger/report themselves.
.github/workflows/tri-comparison-history.yml(push-to-main companion) does thatregeneration automatically after a relevant change lands, opening an auto-merged PR only when
the numbers actually moved (overwrite-in-place files, so "no diff" already is the adaptivity —
no separate dedup check needed the way the tree-sitter history CSV needs one).
ctags --versiondoesn't printUniversal Ctagsbefore doinganything else, addressing the PR Fix #1892: COBOL func_start regex reserved word shield #2111 (2026-08-22) incident where a shadowed
ctagsbinarysilently degraded every language's comparison with no error anywhere in the run.
docs/self_scan/tri_comparison_README.md(new "CI enforcement" section),CLAUDE.md'sDifferential Scan section, and the
ci-push-checklist/tree-sitter-accuracy-sweepskills toreflect that the chart/ledger regen is no longer a manual pre-push step, and that a sweep fix
touching a baselined language now needs
tri_comparison_chart.py --all --ciclean too.Verified end-to-end locally (real
ctags, reallanguage-cruciblecorpus, realtree-sitter-language-pack): generated a real cpp baseline (func_precision: 99.71%,class_precision: 100%— matches the already-documented cpp numbers), confirmed--cipassesclean against it, then simulated a regression by editing the baseline and confirmed
--cifailswith the correct diff message. That test-generated baseline file was removed before committing,
per the "start empty, populate incrementally" decision above.
Test plan
tests/tools/tri_comparison_chart.py --languages cpp --regenerateproduces correct realnumbers against the pinned language-crucible corpus.
--cipasses clean against a matching baseline.--cifails with a correct, readable diff against a deliberately-inflated baseline, andrecovers cleanly once reverted.
python tests/tools/audit_check.py(ruff/mypy/dead-key/ast-accuracy) — clean except 2pre-existing mypy findings in
gitgalaxy/core/network_risk_sensor.py, unrelated to thischange (not touched by this PR).
yaml.safe_load).tri_comparison_baseline_*.json,so
tri-comparison-audit.ymlshould report "0 languages checked" / pass trivially — firstreal population of a baseline is the deliberate follow-up mentioned above).
🤖 Generated with Claude Code