Skip to content

update-snapshot: --stability self-diff pre-check, update-compare reports unstable shots as not comparable - #299

Merged
jasperf merged 1 commit into
mainfrom
add/update-snapshot-stability
Nov 17, 2026
Merged

jasperf merged 1 commit into
mainfrom
add/update-snapshot-stability

Conversation

@jasperf

@jasperf jasperf commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

This PR gives update-snapshot a --stability self-diff pre-check and teaches update-compare to report unstable shots as not comparable, resolving #294. A page that changes on every load — rotating testimonials, "latest posts", WooCommerce's ORDER BY RAND() related products — false-flags every before/after diff in an update round, and proving it was the rotation (not the update) meant diffing two captures of the unchanged page by hand. The pre-check does that inside the run.

update-snapshot --stability:

  • Every page is captured twice immediately after another; the two shots are diffed with the same metric update-compare uses (ImageMagick 7 AE as a fraction of all pixels, fuzz 15%, heights padded to one canvas first). It therefore needs magick, and fails fast with an install hint when that is missing.
  • Shots over --stability-threshold (default 0.5%, matching --threshold's default) print ⚠ <page> (<viewport>): UNSTABLE on its own (x% self-diff) — likely rotating content, mask it or expect false flags and get unstable: true plus selfDiffPct in manifest.json. The second capture is temporary (dot-prefixed, deleted after the diff), so it never enters a compare.
  • The run summary counts unstable shots; stability problems alone never fail a capture run.

update-compare:

  • Before flagging a shot over the threshold, it looks up the shot in either side's manifest.json (via python3 when present, silently as before otherwise). A shot marked unstable is reported ? not comparable (unstable on its own: x% self-diff, rotating content) with its own amber row style, and does not count as flagged or fail the compare (0 …, 1 not comparable → exit 0). A stable shot still flagged is a real regression candidate, exactly as before.

Docs:

  • scripts/updates/README.md documents the flag, the stability walk-through, and the starter mask list the issue asked for (section.related.products for Woo related products, plus the usual rotating blocks).
  • Catalog regenerated; CHANGELOG.md entry under [Unreleased]/Added.

Verification (run on this branch):

  • node --check clean on update-snapshot.js; /bin/bash -n update-compare.sh (system bash 3.2) clean; shellcheck at warning level clean; go test ./... passes.
  • End-to-end smoke test against a local server with a static page and a rotating one (a block at a random position per load):
    • --stability run: static page stable, rotating page UNSTABLE on its own (1.689% self-diff); manifest.json carries unstable/selfDiffPct.
    • Two runs compared with update-compare: ✓ home-desktop 0.000%, ? rotating-desktop 1.873% — not comparable: unstable on its own (1.689% self-diff, rotating content), summary 0 of 2 shots over 0.5%, 1 not comparable, exit code 0 (before this change that would exit 1).
    • Compare page HTML carries the unstable row class and the not-comparable meta note.

Files changed (5):

  • scripts/updates/update-snapshot.js
  • scripts/updates/update-compare.sh
  • scripts/updates/README.md
  • go/internal/catalog/catalog.json
  • CHANGELOG.md

A page that changes on every load - rotating testimonials, Woo's
ORDER BY RAND related products - false-flags every before/after diff,
and the only way to prove it was the rotation was diffing two
captures of the unchanged page by hand. --stability captures each
page twice in the same run and diffs the pair with the same metric
update-compare uses; over-threshold shots print an UNSTABLE hint and
are marked unstable/selfDiffPct in manifest.json.

update-compare reads those marks and reports the shot as not
comparable instead of flagging it, so a rotating page no longer fails
a round's eye test. Catalog regenerated; README documents the check
and a starter mask list.

Resolves #294.
@jasperf
jasperf merged commit 7fe46d7 into main Nov 17, 2026
2 checks passed
@jasperf
jasperf deleted the add/update-snapshot-stability branch November 17, 2026 04:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant