diff --git a/CHANGELOG.md b/CHANGELOG.md index 905d189..ca5dd33 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **`--stability`: `update-snapshot` proves which pages rotate before a round flags them.** A page that changes on every load — rotating testimonials, "latest posts", WooCommerce's `ORDER BY RAND()` related products — false-flags every before/after diff, and proving it was the rotation (not the update) meant diffing two captures of the unchanged page by hand. `--stability` does that inside the run: every page is captured twice immediately after another, the two shots are diffed with the same metric `update-compare` uses (ImageMagick AE, fuzz 15, `--stability-threshold`, default 0.5% — so it needs `magick` like update-compare), and shots over the threshold print `UNSTABLE on its own (x% self-diff)` and get `unstable: true` plus `selfDiffPct` in `manifest.json`. `update-compare` reads those marks — from either side's manifest — and reports such shots as `? not comparable (unstable on its own)` instead of failing the compare, with their own row style on the page. The updates README gained the stability walk-through and a starter mask list (`section.related.products` first) (#294). + - **`checkout-check` walks stores whose shop page has no product grid.** Some production stores make the shop landing a curated or order-form page; step 1 then failed with `no products listed` and every later step was skipped, even though the store was perfectly walkable via its category archives and products. When the grid selector finds nothing, step 1 now falls back to the first `product-category` archive the page links to — and when that archive lists products it acts as the shop for the remaining steps — then to any `a[href*="/product/"]` link on the page. `--product`/`--category` stay the explicit overrides and only become necessary when the page has neither (`SKIP` note as before). Step 2 and 3 also resolve discovered relative `href`s against the base URL, which the fake-store test showed they silently mishandled before. `--runs` and everything else unchanged; the walk stays read-only (#293). - **`--max-file-size` and single-file targets for the malware scanners.** Both `scanner-targeted.php` and `scanner-general.php` skipped pattern-scanning on any file over a hardcoded 5MB cap, and checksum verification doesn't cover every large file — a custom theme asset or a plugin outside the verified list got neither check, with no way to work around it: the cap wasn't a flag, and the scanners only accepted a directory as a target. Both now take a `--max-file-size=20MB`-style flag (plain bytes or a KB/MB/GB size), and `wp-ops malware-scan` passes it through as `--max-file-size` for remote scans (the streamed-source invocation stays unchanged; the scanners read the flag from their own argv). The path can now also be a single file: it is pattern-scanned as asked with no size cap and no checksum filtering, so a known-large, non-core file can be checked in isolation without raising the cap for a whole tree (#230). diff --git a/go/internal/catalog/catalog.json b/go/internal/catalog/catalog.json index a40e59d..27d7c32 100644 --- a/go/internal/catalog/catalog.json +++ b/go/internal/catalog/catalog.json @@ -2087,10 +2087,25 @@ "default": "60", "description": "Seconds per capture before giving up", "raw": "--timeout optional {60} Seconds per capture before giving up" + }, + { + "name": "--stability", + "required_raw": "optional", + "required": false, + "description": "Capture each page twice, flag pages that differ from themselves", + "raw": "--stability optional {} Capture each page twice, flag pages that differ from themselves" + }, + { + "name": "--stability-threshold", + "required_raw": "optional", + "required": false, + "default": "0.5", + "description": "Percent of self-changed pixels that marks a page unstable", + "raw": "--stability-threshold optional {0.5} Percent of self-changed pixels that marks a page unstable" } ], "examples": [ - "wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt" + "wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt --stability" ], "manifest_category": "monitoring", "platform": "any", diff --git a/scripts/updates/README.md b/scripts/updates/README.md index e41c826..bb2548d 100644 --- a/scripts/updates/README.md +++ b/scripts/updates/README.md @@ -9,7 +9,7 @@ Two commands: | Command | What it does | |---|---| | `wp-ops update-snapshot` | Screenshot a page list at desktop and 390px mobile, into one directory per run | -| `wp-ops update-compare` | Diff two runs, write a self-contained compare folder (`index.html` + images), exit 1 if anything changed more than the threshold | +| `wp-ops update-compare` | Diff two runs, write a self-contained compare folder (`index.html` + images), exit 1 if anything changed more than the threshold (shots marked unstable on their own are reported not comparable, no fail) | Both run on your machine, against any URL: a local `.test` copy, staging or production. Neither one changes the site. @@ -71,9 +71,34 @@ expected result is 0.000% on every shot. | `--mask=` | | Extra CSS selectors to hide (`visibility:hidden`, layout unchanged) | | `--no-default-mask` | | Don't hide the Contact Form 7 CAPTCHA | | `--timeout=60` | `60` | Seconds per capture before it is skipped and marked failed | +| `--stability` | off | Capture every page twice and diff it against itself; shots over the threshold get `unstable: true` + `selfDiffPct` in `manifest.json` | +| `--stability-threshold=0.5` | `0.5` | % of self-changed pixels that marks a shot unstable | Writes `-.png` per shot and `manifest.json` (URL, HTTP status, page size, error per shot). Exits 1 if a capture failed or a page returned 400+. +`--stability` additionally needs ImageMagick 7 (`magick`), like `update-compare`. + +### Stability pre-check (`--stability`) + +A page that changes on every load — rotating testimonials, a "latest posts" +block, WooCommerce's related products — shows up as a false "change" in every +before/after diff, and you only find out by diffing two captures of the +unchanged page by hand. `--stability` does that as part of the run: every page +is captured twice immediately after another, the two captures are diffed with +the same metric `update-compare` uses, and anything over the threshold is +printed as `UNSTABLE on its own (x% self-diff)` and marked in the manifest. +`update-compare` then reports those shots as **not comparable** instead of +failing the round. Use it on the before-run (and the after-run, if you want the +mark on both sides of the compare). + +Starter mask list for the usual suspects — mask them with `--mask` (or confirm +with `--stability` first) instead of chasing them through a round: + +- `section.related.products`, `.related.products` — WooCommerce related products, + `ORDER BY RAND()`: every product-page shot on a stock Woo install rotates. +- `.wpcf7-captchac, img.wpcf7-captcha` — CF7 CAPTCHA (masked by default). +- Rotating testimonials, "latest posts" widgets, countdown timers, carousels — + anything that animates or re-orders per load. `update-compare ` diff --git a/scripts/updates/update-compare.sh b/scripts/updates/update-compare.sh index 3b70b1b..46516e5 100755 --- a/scripts/updates/update-compare.sh +++ b/scripts/updates/update-compare.sh @@ -10,8 +10,14 @@ # to a client as-is. Shots of different heights are padded to the same size # before diffing, so a page that grew still gets a diff instead of a skip. # -# Exit code: 0 when every shot is under --threshold, 1 when any shot is over -# it or missing from . +# Shots that update-snapshot --stability marked "unstable" in manifest.json +# (they differ from themselves between two captures: rotating testimonials, +# WooCommerce's ORDER BY RAND related products) are reported as not +# comparable instead of flagged — a rotating page over the threshold proves +# nothing about the update, so it no longer fails the compare. +# +# Exit code: 0 when every comparable shot is under --threshold, 1 when any +# comparable shot is over it or missing from . # # Usage: # ./update-compare.sh [options] @@ -65,9 +71,31 @@ mkdir -p "$OUT/before" "$OUT/after" "$OUT/diff" html_escape() { sed -e 's/&/\&/g' -e 's//\>/g' <<<"$1"; } +# The self-diff update-snapshot --stability recorded for a shot, when that +# run marked it unstable. Prints it when found, nothing otherwise; python3 is +# checked first so a machine without it simply reports as before. +manifest_self_diff() { + local name="$1" manifest="$2" + [[ -f "$manifest" ]] || return 0 + command -v python3 >/dev/null 2>&1 || return 0 + python3 - "$manifest" "$name" <<'PY' 2>/dev/null || true +import json, sys +try: + with open(sys.argv[1]) as fh: + data = json.load(fh) +except Exception: + sys.exit(0) +for shot in data.get('shots', []): + if shot.get('file') == sys.argv[2] and shot.get('unstable'): + print(shot.get('selfDiffPct', '')) + break +PY +} + rows_flag="" rows_ok="" flagged=0 +unstable=0 total=0 shopt -s nullglob for before_file in "$BEFORE"/*.png; do @@ -103,12 +131,22 @@ for before_file in "$BEFORE"/*.png; do pct="$(awk -v f="$fraction" 'BEGIN { printf "%.3f", f * 100 }')" over="$(awk -v a="$pct" -v t="$THRESHOLD" 'BEGIN { print (a > t) ? 1 : 0 }')" + # A shot whose own run proved it rotates is not comparable, not flagged. + self_diff="$(manifest_self_diff "$name" "$BEFORE/manifest.json")" + [[ -z "$self_diff" ]] && self_diff="$(manifest_self_diff "$name" "$AFTER/manifest.json")" + note="" [[ "$size_b" != "$size_a" ]] && note=" (size $size_b → $size_a)" if [[ "$over" == 1 ]]; then - flagged=$((flagged + 1)); mark="✗"; cls="flag" + if [[ -n "$self_diff" ]]; then + unstable=$((unstable + 1)); mark="?"; cls="unstable" + note="$note — not comparable: unstable on its own (${self_diff}% self-diff, rotating content)" + else + flagged=$((flagged + 1)); mark="✗"; cls="flag" + fi else mark="✓"; cls="ok" + [[ -n "$self_diff" ]] && note="$note (unstable on its own: ${self_diff}% self-diff)" fi printf "%s %-32s %7s%%%s\n" "$mark" "${name%.png}" "$pct" "$note" n="$(html_escape "$name")" @@ -123,6 +161,11 @@ if [[ "$total" == 0 ]]; then echo "Error: no PNG files in $BEFORE"; exit 1 fi +meta_extra="" +if [[ "$unstable" -gt 0 ]]; then + meta_extra=", $unstable not comparable (unstable on their own — rotating content)" +fi + cat > "$OUT/index.html" < @@ -133,15 +176,16 @@ h1{font-size:22px;margin:0 0 4px}p.meta{color:#52606d;margin:0 0 20px} table{border-collapse:collapse;width:100%}th,td{border-top:1px solid #e4e7eb;padding:10px;vertical-align:top;text-align:left} td:first-child{width:180px;font-weight:600}td small{font-weight:400;color:#52606d} tr.flag td:first-child{color:#c81e1e}tr.flag{background:#fff5f5} +tr.unstable td:first-child{color:#b46b00}tr.unstable{background:#fff8ec} img{width:100%;max-height:600px;object-fit:cover;object-position:top;border:1px solid #e4e7eb}

$(html_escape "$TITLE")

-

$total shots, $flagged over ${THRESHOLD}% changed (fuzz ${FUZZ}%). Before: $(html_escape "$BEFORE") · After: $(html_escape "$AFTER") · $(date '+%Y-%m-%d %H:%M')

+

$total shots, $flagged over ${THRESHOLD}% changed (fuzz ${FUZZ}%)${meta_extra}. Before: $(html_escape "$BEFORE") · After: $(html_escape "$AFTER") · $(date '+%Y-%m-%d %H:%M')

$rows_flag$rows_ok
PageBeforeAfterChanged pixels
HTML echo "" -echo "$flagged of $total shots over ${THRESHOLD}% → $OUT/index.html" +echo "$flagged of $total shots over ${THRESHOLD}%${meta_extra} → $OUT/index.html" [[ "$flagged" == 0 ]] diff --git a/scripts/updates/update-snapshot.js b/scripts/updates/update-snapshot.js index 90a455a..3a5bc21 100755 --- a/scripts/updates/update-snapshot.js +++ b/scripts/updates/update-snapshot.js @@ -32,6 +32,11 @@ * --mask= Extra CSS selectors to hide before capture * --no-default-mask Don't hide the CF7 CAPTCHA * --timeout=60 Seconds per capture before giving up (default: 60) + * --stability Capture every page twice and diff it against itself, so self-rotating + * content (Woo's ORDER BY RAND related products, rotating testimonials) is + * flagged "unstable" in manifest.json before it can false-flag an update + * round. Needs ImageMagick 7 (magick), like update-compare. + * --stability-threshold=0.5 Percent of self-changed pixels that marks a page unstable * * Requires Playwright: `npm install && npx playwright install chromium` in * this directory (scripts/updates/). @@ -49,11 +54,14 @@ * @flag --mask optional {.cookie-button} Extra CSS selectors to hide before capture * @flag --no-default-mask optional {} Don't hide the Contact Form 7 CAPTCHA * @flag --timeout optional {60} Seconds per capture before giving up - * @example wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt + * @flag --stability optional {} Capture each page twice, flag pages that differ from themselves + * @flag --stability-threshold optional {0.5} Percent of self-changed pixels that marks a page unstable + * @example wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt --stability */ const fs = require('fs'); const path = require('path'); +const { spawnSync } = require('child_process'); // Look for Playwright next to this script first (a wp-ops checkout after // `npm install` here), then in the current project (a site that already has @@ -75,6 +83,7 @@ const VIEWPORTS = { mobile: { width: 390, height: 844 }, }; const DEFAULT_MASK = ['img.wpcf7-captcha', '.wpcf7-captchac']; +const STABILITY_FUZZ = 15; // Same per-pixel tolerance update-compare uses. function parseArgs(argv) { const base = argv[0]; @@ -104,7 +113,15 @@ function parseArgs(argv) { ...(has('no-default-mask') ? [] : DEFAULT_MASK), ...(flag('mask') ? flag('mask').split(',').map((s) => s.trim()).filter(Boolean) : []), ]; - return { base: base.replace(/\/+$/, ''), out, pages, viewports, mask, timeout: (+flag('timeout') || 60) * 1000 }; + const stability = has('stability'); + if (stability && spawnSync('magick', ['-version']).error) { + console.error('Error: --stability needs ImageMagick 7 (magick), like update-compare. brew install imagemagick'); + process.exit(1); + } + return { + base: base.replace(/\/+$/, ''), out, pages, viewports, mask, timeout: (+flag('timeout') || 60) * 1000, + stability, stabilityThreshold: +flag('stability-threshold') || 0.5, + }; } function usage(msg) { @@ -126,6 +143,29 @@ function withTimeout(promise, ms, label) { ]).finally(() => clearTimeout(t)); } +// How many percent of pixels differ between two captures of the same +// unchanged page. Same metric as update-compare: pad both to the same canvas +// (full-page shots of a rotating section can differ in height), then IM7's +// AE as a fraction of all pixels at fuzz 15%. +function selfDiffPct(a, b) { + const size = (f) => { + const r = spawnSync('magick', ['identify', '-format', '%wx%h', f]); + if (r.status !== 0) throw new Error(`magick identify failed for ${f}`); + return r.stdout.toString(); + }; + const [bw, bh] = size(a).split('x').map(Number); + const [aw, ah] = size(b).split('x').map(Number); + const w = Math.max(bw, aw), h = Math.max(bh, ah); + const diff = spawnSync('magick', [ + '(', a, '-background', 'white', '-extent', `${w}x${h}`, ')', + '(', b, '-background', 'white', '-extent', `${w}x${h}`, ')', + '-fuzz', `${STABILITY_FUZZ}%`, '-metric', 'AE', '-compare', + '-format', '%[distortion]', 'info:', + ]); + if (diff.status !== 0) throw new Error(`magick compare failed (${(diff.stderr || '').toString().split('\n')[0]})`); + return +(diff.stdout.toString()) * 100; +} + async function capture(browser, url, vp, file, mask) { const context = await browser.newContext({ viewport: VIEWPORTS[vp] }); const page = await context.newPage(); @@ -188,11 +228,33 @@ async function capture(browser, url, vp, file, mask) { failed++; console.log(`✗ ERR ${vp.padEnd(7)} ${p.path} ${row.error}`); } + // Stability pre-check (#294): capture the same page again immediately + // and diff it against itself. Anything over the threshold changes on + // every load — rotating content — and cannot be trusted in a + // before/after comparison, no matter what the update did. + if (opts.stability && !row.error && row.status < 400) { + const second = path.join(opts.out, `.stability-${p.name}-${vp}.png`); + try { + await withTimeout(capture(browser, url, vp, second, opts.mask), opts.timeout, url); + const selfPct = selfDiffPct(file, second); + row.selfDiffPct = +selfPct.toFixed(3); + row.unstable = selfPct > opts.stabilityThreshold; + if (row.unstable) { + console.log(` ⚠ ${p.name} (${vp}): UNSTABLE on its own (${row.selfDiffPct}% self-diff) — likely rotating content, mask it or expect false flags`); + } + } catch (e) { + row.stabilityError = e.message.split('\n')[0]; + console.log(` ⚠ ${p.name} (${vp}): stability check failed: ${row.stabilityError}`); + } finally { + fs.rmSync(second, { force: true }); + } + } results.push(row); } } await browser.close(); + const unstable = results.filter((r) => r.unstable).length; fs.writeFileSync(path.join(opts.out, 'manifest.json'), JSON.stringify({ base: opts.base, captured_at: new Date().toISOString(), @@ -201,5 +263,10 @@ async function capture(browser, url, vp, file, mask) { shots: results, }, null, 2)); console.log(`\n${results.length - failed}/${results.length} captured in ${Math.round((Date.now() - started) / 1000)}s → ${opts.out}`); + if (opts.stability) { + console.log(unstable + ? `${unstable} shot(s) unstable on its own (rotating content) — see the unstable/selfDiffPct marks in manifest.json, mask the rotating sections or expect false flags` + : 'All shots stable against themselves (captured twice, 0 over threshold).'); + } process.exit(failed ? 1 : 0); })().catch((e) => { console.error(`Error: ${e.message}`); process.exit(1); });