Fail a pull request whose benchmark charts drift from their history

.github/workflows/benchmark-history.yml

@@ -60,6 +60,7 @@ env:
   BENCHMARK_JOB: short
   # One line per chart: name|history|chart|filter. An environment variable cannot hold an array,
   # and three steps need the same three triples, so this is the one place they are written.
+  # verify-charts.yml reads this value from here too, so it stays the only copy.
   #
   # The quantities filter names one operation per panel. The strings and paths filters name whole
   # classes instead, so they measure more than they draw: ingest stores every row and only render
@@ -203,19 +204,10 @@ jobs:
             done <<< "$SUBJECTS"
           done
 
+      # The same script verify-charts.yml runs, so the charts this commits are the ones it checks.
       - name: Redraw the charts
         shell: bash
-        run: |
-          set -euo pipefail
-          while IFS='|' read -r subject history chart filter; do
-            [ -n "$subject" ] || continue
-            if [ ! -f "$history" ]; then
-              echo "No history for $subject yet; nothing to draw."
-              continue
-            fi
-            dotnet run scripts/benchmark-history.cs -- render \
-              --subject "$subject" --history "$history" --out "$chart" </dev/null
-          done <<< "$SUBJECTS"
+        run: bash scripts/render-benchmark-charts.sh
 
       - name: Commit the history and the chart
         shell: bash

.github/workflows/verify-charts.yml

@@ -0,0 +1,67 @@
+name: Verify Benchmark Charts
+
+# The benchmark SVGs under docs/benchmarks/ are committed output, drawn from the history JSON
+# beside them. This job redraws every chart from the committed history and fails if any byte
+# differs, so a hand-edited history or a renderer change can't leave the README charts showing
+# something the data does not say (#303).
+#
+# It redraws through scripts/render-benchmark-charts.sh, the step benchmark-history.yml runs, and
+# reads the subject table from benchmark-history.yml's env, so the guard and the recorder cannot
+# drift apart.
+
+on:
+  pull_request:
+    paths:
+      - "docs/benchmarks/**"
+      - "scripts/benchmark-history.cs"
+      - "scripts/render-benchmark-charts.sh"
+      - ".github/workflows/benchmark-history.yml"
+      - ".github/workflows/verify-charts.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: verify-charts-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  verify-charts:
+    name: Benchmark charts match their history
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup .NET SDK
+        uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: "10.0.x"
+
+      - name: Redraw every chart from its committed history
+        shell: bash
+        run: |
+          set -euo pipefail
+          # yq ships on the GitHub-hosted Ubuntu images.
+          SUBJECTS=$(yq '.env.SUBJECTS' .github/workflows/benchmark-history.yml)
+          export SUBJECTS
+          bash scripts/render-benchmark-charts.sh
+
+      - name: Fail if a committed chart differs
+        shell: bash
+        run: |
+          set -euo pipefail
+          # Untracked files count too: a chart the history draws but nobody committed is drift.
+          drift=$(git status --porcelain -- docs/benchmarks/)
+          if [ -n "$drift" ]; then
+            echo "::error::These benchmark charts differ from what their committed history draws:"
+            while read -r _ file; do
+              echo "::error file=$file::$file is stale"
+            done <<< "$drift"
+            git diff --stat -- docs/benchmarks/
+            echo "Run 'bash scripts/render-benchmark-charts.sh' with SUBJECTS set from benchmark-history.yml, and commit the result." >&2
+            exit 1
+          fi
+          echo "Every benchmark chart matches its history."

CLAUDE.md

@@ -409,7 +409,9 @@ redraws the chart the README shows. Four things about it are not guessable from
   version across every package.
 - **The histories and the SVG files are committed output that a bot pushes.** A local `render` must be
   diffed before commit rather than assumed. The quantities chart in particular is a regression test:
-  a change to the renderer that moves a byte in it has broken something.
+  a change to the renderer that moves a byte in it has broken something. `verify-charts.yml` enforces
+  this on pull requests: it redraws every chart through `scripts/render-benchmark-charts.sh`, the step
+  the recorder runs, and fails on any difference.
 - **Nothing here touches an internal member.** The `InternalsVisibleTo` that would expose one names
   only the test assembly, and a benchmark built on internals could only ever measure the working copy,
   never a published package — which would make the release history impossible to backfill.

scripts/render-benchmark-charts.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Redraws every benchmark chart from its committed history.
+#
+# The one render invocation both workflows run: benchmark-history.yml after it records a point,
+# and verify-charts.yml to check that the committed charts are what the committed histories draw.
+# Sharing it is what keeps the guard and the recorder from drifting apart (#303).
+#
+# Reads the subject table from $SUBJECTS, one `name|history|chart|filter` line per subject, as
+# benchmark-history.yml declares it. The filter is not used here.
+set -euo pipefail
+
+: "${SUBJECTS:?SUBJECTS must hold the subject table from benchmark-history.yml}"
+
+while IFS='|' read -r subject history chart _filter; do
+  [[ -n "$subject" ]] || continue
+  if [[ ! -f "$history" ]]; then
+    echo "No history for $subject yet; nothing to draw."
+    continue
+  fi
+  # stdin from /dev/null: a process that drained the here-string would end the loop early and
+  # still exit zero, silently skipping the remaining subjects.
+  dotnet run scripts/benchmark-history.cs -- render \
+    --subject "$subject" --history "$history" --out "$chart" </dev/null
+done <<< "$SUBJECTS"