[barborico]

Summary

Adds npm run visual-diff, a local command that renders every route in src/App.tsx on the current branch and on its merge base with origin/main. It then lists the pages whose screenshots changed, the pages that are new, and console, network, or render errors that only the branch has. It exits non-zero only on new errors, and it adds no CI workflow and stores no baseline images.

Urgency: low - one week
Expected review effort: <15 min, assuming focus on the tool's shape and documentation rather than reading the script logic itself (IMO not worth scrutiny)

Motivation

Frontend dependency PRs such as the MUI 9 / MUI X 9 upgrade (#706) pass tsc and vitest, but nothing checks the rendered pages, and clicking through about 25 pages by hand on every upgrade isn't practical. Rendering both sides on the same machine in one run avoids stored baselines and the font differences between operating systems that make checked-in screenshots brittle.

Validation

Confidence: medium

Run on macOS (Node 22, Playwright 1.64 with Chromium):

• With no frontend changes it reported no diffs on 10 consecutive runs. Before the 100-pixel noise tolerance was added, about 1 in 10 pages flagged 4–49 pixels of glyph antialiasing noise per run.
• A deliberate mt: 4 → mt: 6 on the main Container flagged every page with a readable 16px-shift diff, and the run exited 0.
• A temporary new /about route was reported as a new page, and a temporary console.error in Home was reported as a branch-only error with exit code 1.
• A Vite dev server on port 3000 kept serving throughout a run. git status, the checkout's build/, and .claude/.api-port were unchanged afterwards.
• Ctrl-C during the base install, during the build, and during rendering exited 130 each time, with no leftover worktree, temp directory, uvicorn, Chromium, or npm process.
• --base HEAD~3 (whose merge base has a different uv.lock) built its own virtualenv and worked.
• First run against a new merge base took about 35s. A cached run took about 23s. Both had warm npm and uv caches and Chromium already downloaded.
• Ran against the MUI 9 upgrade (Upgrade to MUI 9 and MUI X 9 #706) with this branch cherry-picked onto it. It surfaced real changes: shrunken link icons and shorter accordion rows on Home (a codemod turned fontSize="small" into sx={{fontSize: 'small'}}), EmptyListEntry's "None" turning from grey to black, nav chevrons moving to the drawer edge, and Chip labels shifting by about 1px. Its one false "new error" (a pre-existing warning whose React component stack changed) led to the second commit, which compares warnings without their stacks.
• Try it on Linux

Risks

None for the app itself: the change adds a dev-only tool and the @playwright/test devDependency. The tool's main limitation is that the seed data and the spec's route table need a case added whenever a page is added. A spec test fails the run when src/App.tsx has an uncovered <Route>.

Guidance for Reviewers

Design choices worth a look:

• Noise tolerance. Two launches of headless Chromium on the same page still antialias a few glyphs differently, even with GPU off, sRGB forced, hinting and subpixel positioning off, and transitions disabled. Pages are allowed up to 100 differing pixels; the largest noise seen was 49. A one-character text change could fall under that. Retrying the branch's screenshot was the alternative, but it doesn't help when the base sample is the outlier.
• Development-mode, unminified builds on both sides, so React and MUI warnings survive and their component names match across builds.
• Isolation. Each side's backend runs from a scratch directory with a minimal environment, in a virtualenv built from its own uv.lock without dev dependencies. Neither the checkout's .env nor plugins installed in its .venv can leak in. The first attempt failed exactly this way: an example plugin installed in .venv required Google credentials.
• One spec, one seed, both from the branch. The seed sticks to long-stable columns, so a base older than a model change to those columns would fail to seed.

Detailed Description of Changes

• scripts/visual-diff/run.js is the orchestrator. It computes the merge base, adds a temporary git worktree, runs npm ci --ignore-scripts and builds the base on a cache miss (cache key is the commit), builds the branch from the working tree into a temp directory, seeds and starts both backends, serves each build with a small static server that proxies /api, runs Playwright twice, and prints the summary. Cache lives under ${XDG_CACHE_HOME:-~/.cache}/access-visual-diff.
• scripts/visual-diff/seed.py and fixtures.json hold the fixed data set. Timestamps are offsets from midnight UTC of the run, and the browser clock is frozen relative to the same point, so relative dates match.
• scripts/visual-diff/pages.spec.ts visits each route, records errors, waits for spinners and skeletons to clear, grows the viewport to <main>'s scroll height (the app scrolls inside <main>, so fullPage alone stops at one screenful), and takes the screenshot. A route that renders the not-found page on the base writes no expectation, so it reports as new.
• Docs are in scripts/visual-diff/README.md, including what to update as pages and models change. The testing guide and the top-level README each have a short pointer to it. The frontend guide and CLAUDE.md's common mistakes tell authors to add a page case and seed data with every new route.
• Already surfaced on main: request detail pages fire /api/audit/... with empty ids (404s), and several list pages nest <a>/<td>/<button> invalidly. These appear on both sides, so they aren't reported as new.

🤖 Generated with Claude Code

[Jeff-Rowell]

This is great! IMO I'd drop the "side by side" screen shot viewer or make the side by side images larger. I found I just toggled between "Actual" and "Expected" or used the slider more.