Skip to content

ci: gate PR merges on tri-comparison validated-precision regression - #2139

Merged
squid-protocol merged 1 commit into
mainfrom
tri-comparison-ci-gate
Aug 22, 2026
Merged

ci: gate PR merges on tri-comparison validated-precision regression#2139
squid-protocol merged 1 commit into
mainfrom
tri-comparison-ci-gate

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Summary

Makes the tri-comparison system (GitGalaxy vs. tree-sitter vs. ctags) CI-enforced for any PR
touching core-engine parsing logic, instead of skill/human-driven only. Two parts, following the
already-proven tree-sitter-accuracy-audit.yml/-history.yml split rather than inventing a new
mechanism:

  • tests/tools/tri_comparison_chart.py --ci/--regenerate: baseline-gated regression check
    on GitGalaxy's own validated precision (func_precision/class_precision, read after
    apply_verified_adjustments() applies any ledger verdict — never a raw, unvalidated
    disagreement count, per this repo's "comparative-correctness claims require verification" rule).
    New per-language baseline files, tests/tri_comparison_baseline_<lang>.json, populated
    incrementally via --regenerate (none bundled in this PR — that's a deliberate follow-up once
    this scaffolding lands, not bulk-generated here). Recall is deliberately not gated — the
    chart's own module docstring already explains why that ratio's cross-tool denominator isn't
    trustworthy enough to rank on (a bug in either other tool can inflate it without GitGalaxy
    doing anything wrong); gating CI on it would reintroduce the exact problem the chart already
    designed around.
  • .github/workflows/tri-comparison-audit.yml (PR-time, blocking, same core-engine paths
    tree-sitter-accuracy-audit.yml already uses) runs the --ci check. It only ever measures —
    it never requires a contributor to regenerate and commit the chart/ledger/report themselves.
  • .github/workflows/tri-comparison-history.yml (push-to-main companion) does that
    regeneration automatically after a relevant change lands, opening an auto-merged PR only when
    the numbers actually moved (overwrite-in-place files, so "no diff" already is the adaptivity —
    no separate dedup check needed the way the tree-sitter history CSV needs one).
  • Both workflows hard-fail if ctags --version doesn't print Universal Ctags before doing
    anything else, addressing the PR Fix #1892: COBOL func_start regex reserved word shield #2111 (2026-08-22) incident where a shadowed ctags binary
    silently degraded every language's comparison with no error anywhere in the run.
  • Updates docs/self_scan/tri_comparison_README.md (new "CI enforcement" section), CLAUDE.md's
    Differential Scan section, and the ci-push-checklist/tree-sitter-accuracy-sweep skills to
    reflect that the chart/ledger regen is no longer a manual pre-push step, and that a sweep fix
    touching a baselined language now needs tri_comparison_chart.py --all --ci clean too.

Verified end-to-end locally (real ctags, real language-crucible corpus, real
tree-sitter-language-pack): generated a real cpp baseline (func_precision: 99.71%,
class_precision: 100% — matches the already-documented cpp numbers), confirmed --ci passes
clean against it, then simulated a regression by editing the baseline and confirmed --ci fails
with the correct diff message. That test-generated baseline file was removed before committing,
per the "start empty, populate incrementally" decision above.

Test plan

  • tests/tools/tri_comparison_chart.py --languages cpp --regenerate produces correct real
    numbers against the pinned language-crucible corpus.
  • --ci passes clean against a matching baseline.
  • --ci fails with a correct, readable diff against a deliberately-inflated baseline, and
    recovers cleanly once reverted.
  • python tests/tools/audit_check.py (ruff/mypy/dead-key/ast-accuracy) — clean except 2
    pre-existing mypy findings in gitgalaxy/core/network_risk_sensor.py, unrelated to this
    change (not touched by this PR).
  • Both new workflow YAML files parse cleanly (yaml.safe_load).
  • CI itself (this PR doesn't touch any baselined language's tri_comparison_baseline_*.json,
    so tri-comparison-audit.yml should report "0 languages checked" / pass trivially — first
    real population of a baseline is the deliberate follow-up mentioned above).

🤖 Generated with Claude Code

Adds tests/tools/tri_comparison_chart.py's --ci/--regenerate mode, gated on
GitGalaxy's own ledger-validated precision (never a raw disagreement count) --
recall is deliberately excluded, since the chart's own docstring already
explains why that ratio's cross-tool denominator isn't trustworthy to rank on.

Two new workflows mirror the existing tree-sitter-accuracy split:
tri-comparison-audit.yml (PR-time, blocking, measures only) and
tri-comparison-history.yml (push-to-main, regenerates chart/ledger/report,
opens an auto-merged PR only when something actually changed). Both hard-fail
if ctags isn't really Universal Ctags, per the PR #2111 incident.

Updates ci-push-checklist and tree-sitter-accuracy-sweep skills, the
tri-comparison README, and CLAUDE.md's Differential Scan section to reflect
the new gate.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol
squid-protocol merged commit 47954ab into main Aug 22, 2026
29 checks passed
@squid-protocol
squid-protocol deleted the tri-comparison-ci-gate branch August 22, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant