Repository navigation
Fail a pull request whose benchmark charts drift from their history - #495
Merged
Merged
Conversation
…303) Nothing checked that the committed SVGs under docs/benchmarks/ are what scripts/benchmark-history.cs draws from the committed history JSON, so a hand-edited history or a renderer change could leave the README charts wrong without any check failing. verify-charts.yml runs on pull requests that touch the histories, the charts, the renderer or the recorder workflow. It redraws every chart and fails on any difference, naming each drifting file. The render loop moves out of benchmark-history.yml into scripts/render-benchmark-charts.sh, and both workflows run it. The guard reads the subject table from benchmark-history.yml's env, so the guard and the recorder can't drift apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vfJW2sHrLz9iC1cJzLLZy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vfJW2sHrLz9iC1cJzLLZy
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #303
What changed
.github/workflows/verify-charts.yml. It runs on pull requests that touchdocs/benchmarks/**,scripts/benchmark-history.cs, the new render script, or either workflow. It redraws all three SVG pairs from the committed history JSON and fails on any byte difference. Untracked output counts as a difference too. Each drifting file gets an::error file=…::annotation, followed by agit diff --stat.scripts/render-benchmark-charts.sh. The render loop moves out ofbenchmark-history.ymlinto this script, and both workflows run it. That is the triage's "reuse the render step". The invocation is unchanged:dotnet run scripts/benchmark-history.cs -- render --subject … --history … --out …, with stdin from/dev/null.SUBJECTSfrombenchmark-history.yml'senvwithyq, which ships on the hosted Ubuntu images. That env stays the only place the table is written.Verification
Run locally.
yqwas swapped for an equivalent Python read of the same key, and the rest is the workflow's own step scripts:main, all three charts redraw byte-identical and the check passes (exit 0). This is the "passes on main" criterion.meanNsinstrings-history.json, the check fails (exit 1). It namesstrings-performance.svgandstrings-performance-dark.svg, and only those.benchmark-history.yml, so the new workflow runs on this PR itself.Note
#494 (for #320) also edits
benchmark-history.yml, in its trigger and job header. That is a different region from the render step this PR replaces, so the two should merge cleanly in either order.Generated by Claude Code