Repository navigation
update-snapshot: --stability self-diff pre-check, update-compare reports unstable shots as not comparable - #299
Merged
Conversation
A page that changes on every load - rotating testimonials, Woo's ORDER BY RAND related products - false-flags every before/after diff, and the only way to prove it was the rotation was diffing two captures of the unchanged page by hand. --stability captures each page twice in the same run and diffs the pair with the same metric update-compare uses; over-threshold shots print an UNSTABLE hint and are marked unstable/selfDiffPct in manifest.json. update-compare reads those marks and reports the shot as not comparable instead of flagging it, so a rotating page no longer fails a round's eye test. Catalog regenerated; README documents the check and a starter mask list. Resolves #294.
This was referenced Nov 17, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR gives
update-snapshota--stabilityself-diff pre-check and teachesupdate-compareto report unstable shots as not comparable, resolving #294. A page that changes on every load — rotating testimonials, "latest posts", WooCommerce'sORDER BY RAND()related products — false-flags every before/after diff in an update round, and proving it was the rotation (not the update) meant diffing two captures of the unchanged page by hand. The pre-check does that inside the run.update-snapshot --stability:update-compareuses (ImageMagick 7 AE as a fraction of all pixels, fuzz 15%, heights padded to one canvas first). It therefore needsmagick, and fails fast with an install hint when that is missing.--stability-threshold(default 0.5%, matching--threshold's default) print⚠ <page> (<viewport>): UNSTABLE on its own (x% self-diff) — likely rotating content, mask it or expect false flagsand getunstable: trueplusselfDiffPctinmanifest.json. The second capture is temporary (dot-prefixed, deleted after the diff), so it never enters a compare.update-compare:manifest.json(via python3 when present, silently as before otherwise). A shot marked unstable is reported? not comparable (unstable on its own: x% self-diff, rotating content)with its own amber row style, and does not count as flagged or fail the compare (0 …, 1 not comparable→ exit 0). A stable shot still flagged is a real regression candidate, exactly as before.Docs:
scripts/updates/README.mddocuments the flag, the stability walk-through, and the starter mask list the issue asked for (section.related.productsfor Woo related products, plus the usual rotating blocks).CHANGELOG.mdentry under[Unreleased]/Added.Verification (run on this branch):
node --checkclean onupdate-snapshot.js;/bin/bash -n update-compare.sh(system bash 3.2) clean;shellcheckat warning level clean;go test ./...passes.--stabilityrun: static page stable, rotating pageUNSTABLE on its own (1.689% self-diff);manifest.jsoncarriesunstable/selfDiffPct.update-compare:✓ home-desktop 0.000%,? rotating-desktop 1.873% — not comparable: unstable on its own (1.689% self-diff, rotating content), summary0 of 2 shots over 0.5%, 1 not comparable, exit code 0 (before this change that would exit 1).unstablerow class and the not-comparable meta note.Files changed (5):
scripts/updates/update-snapshot.jsscripts/updates/update-compare.shscripts/updates/README.mdgo/internal/catalog/catalog.jsonCHANGELOG.md