Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **`--stability`: `update-snapshot` proves which pages rotate before a round flags them.** A page that changes on every load — rotating testimonials, "latest posts", WooCommerce's `ORDER BY RAND()` related products — false-flags every before/after diff, and proving it was the rotation (not the update) meant diffing two captures of the unchanged page by hand. `--stability` does that inside the run: every page is captured twice immediately after another, the two shots are diffed with the same metric `update-compare` uses (ImageMagick AE, fuzz 15, `--stability-threshold`, default 0.5% — so it needs `magick` like update-compare), and shots over the threshold print `UNSTABLE on its own (x% self-diff)` and get `unstable: true` plus `selfDiffPct` in `manifest.json`. `update-compare` reads those marks — from either side's manifest — and reports such shots as `? not comparable (unstable on its own)` instead of failing the compare, with their own row style on the page. The updates README gained the stability walk-through and a starter mask list (`section.related.products` first) (#294).

- **`checkout-check` walks stores whose shop page has no product grid.** Some production stores make the shop landing a curated or order-form page; step 1 then failed with `no products listed` and every later step was skipped, even though the store was perfectly walkable via its category archives and products. When the grid selector finds nothing, step 1 now falls back to the first `product-category` archive the page links to — and when that archive lists products it acts as the shop for the remaining steps — then to any `a[href*="/product/"]` link on the page. `--product`/`--category` stay the explicit overrides and only become necessary when the page has neither (`SKIP` note as before). Step 2 and 3 also resolve discovered relative `href`s against the base URL, which the fake-store test showed they silently mishandled before. `--runs` and everything else unchanged; the walk stays read-only (#293).

- **`--max-file-size` and single-file targets for the malware scanners.** Both `scanner-targeted.php` and `scanner-general.php` skipped pattern-scanning on any file over a hardcoded 5MB cap, and checksum verification doesn't cover every large file — a custom theme asset or a plugin outside the verified list got neither check, with no way to work around it: the cap wasn't a flag, and the scanners only accepted a directory as a target. Both now take a `--max-file-size=20MB`-style flag (plain bytes or a KB/MB/GB size), and `wp-ops malware-scan` passes it through as `--max-file-size` for remote scans (the streamed-source invocation stays unchanged; the scanners read the flag from their own argv). The path can now also be a single file: it is pattern-scanned as asked with no size cap and no checksum filtering, so a known-large, non-core file can be checked in isolation without raising the cap for a whole tree (#230).
Expand Down
17 changes: 16 additions & 1 deletion go/internal/catalog/catalog.json
Original file line number Diff line number Diff line change
Expand Up @@ -2087,10 +2087,25 @@
"default": "60",
"description": "Seconds per capture before giving up",
"raw": "--timeout optional {60} Seconds per capture before giving up"
},
{
"name": "--stability",
"required_raw": "optional",
"required": false,
"description": "Capture each page twice, flag pages that differ from themselves",
"raw": "--stability optional {} Capture each page twice, flag pages that differ from themselves"
},
{
"name": "--stability-threshold",
"required_raw": "optional",
"required": false,
"default": "0.5",
"description": "Percent of self-changed pixels that marks a page unstable",
"raw": "--stability-threshold optional {0.5} Percent of self-changed pixels that marks a page unstable"
}
],
"examples": [
"wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt"
"wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt --stability"
],
"manifest_category": "monitoring",
"platform": "any",
Expand Down
27 changes: 26 additions & 1 deletion scripts/updates/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Two commands:
| Command | What it does |
|---|---|
| `wp-ops update-snapshot` | Screenshot a page list at desktop and 390px mobile, into one directory per run |
| `wp-ops update-compare` | Diff two runs, write a self-contained compare folder (`index.html` + images), exit 1 if anything changed more than the threshold |
| `wp-ops update-compare` | Diff two runs, write a self-contained compare folder (`index.html` + images), exit 1 if anything changed more than the threshold (shots marked unstable on their own are reported not comparable, no fail) |

Both run on your machine, against any URL: a local `.test` copy, staging or
production. Neither one changes the site.
Expand Down Expand Up @@ -71,9 +71,34 @@ expected result is 0.000% on every shot.
| `--mask=<selectors>` | | Extra CSS selectors to hide (`visibility:hidden`, layout unchanged) |
| `--no-default-mask` | | Don't hide the Contact Form 7 CAPTCHA |
| `--timeout=60` | `60` | Seconds per capture before it is skipped and marked failed |
| `--stability` | off | Capture every page twice and diff it against itself; shots over the threshold get `unstable: true` + `selfDiffPct` in `manifest.json` |
| `--stability-threshold=0.5` | `0.5` | % of self-changed pixels that marks a shot unstable |

Writes `<name>-<viewport>.png` per shot and `manifest.json` (URL, HTTP status,
page size, error per shot). Exits 1 if a capture failed or a page returned 400+.
`--stability` additionally needs ImageMagick 7 (`magick`), like `update-compare`.

### Stability pre-check (`--stability`)

A page that changes on every load — rotating testimonials, a "latest posts"
block, WooCommerce's related products — shows up as a false "change" in every
before/after diff, and you only find out by diffing two captures of the
unchanged page by hand. `--stability` does that as part of the run: every page
is captured twice immediately after another, the two captures are diffed with
the same metric `update-compare` uses, and anything over the threshold is
printed as `UNSTABLE on its own (x% self-diff)` and marked in the manifest.
`update-compare` then reports those shots as **not comparable** instead of
failing the round. Use it on the before-run (and the after-run, if you want the
mark on both sides of the compare).

Starter mask list for the usual suspects — mask them with `--mask` (or confirm
with `--stability` first) instead of chasing them through a round:

- `section.related.products`, `.related.products` — WooCommerce related products,
`ORDER BY RAND()`: every product-page shot on a stock Woo install rotates.
- `.wpcf7-captchac, img.wpcf7-captcha` — CF7 CAPTCHA (masked by default).
- Rotating testimonials, "latest posts" widgets, countdown timers, carousels —
anything that animates or re-orders per load.

`update-compare <before-dir> <after-dir>`

Expand Down
54 changes: 49 additions & 5 deletions scripts/updates/update-compare.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,14 @@
# to a client as-is. Shots of different heights are padded to the same size
# before diffing, so a page that grew still gets a diff instead of a skip.
#
# Exit code: 0 when every shot is under --threshold, 1 when any shot is over
# it or missing from <after-dir>.
# Shots that update-snapshot --stability marked "unstable" in manifest.json
# (they differ from themselves between two captures: rotating testimonials,
# WooCommerce's ORDER BY RAND related products) are reported as not
# comparable instead of flagged — a rotating page over the threshold proves
# nothing about the update, so it no longer fails the compare.
#
# Exit code: 0 when every comparable shot is under --threshold, 1 when any
# comparable shot is over it or missing from <after-dir>.
#
# Usage:
# ./update-compare.sh <before-dir> <after-dir> [options]
Expand Down Expand Up @@ -65,9 +71,31 @@ mkdir -p "$OUT/before" "$OUT/after" "$OUT/diff"

html_escape() { sed -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g' <<<"$1"; }

# The self-diff update-snapshot --stability recorded for a shot, when that
# run marked it unstable. Prints it when found, nothing otherwise; python3 is
# checked first so a machine without it simply reports as before.
manifest_self_diff() {
local name="$1" manifest="$2"
[[ -f "$manifest" ]] || return 0
command -v python3 >/dev/null 2>&1 || return 0
python3 - "$manifest" "$name" <<'PY' 2>/dev/null || true
import json, sys
try:
with open(sys.argv[1]) as fh:
data = json.load(fh)
except Exception:
sys.exit(0)
for shot in data.get('shots', []):
if shot.get('file') == sys.argv[2] and shot.get('unstable'):
print(shot.get('selfDiffPct', ''))
break
PY
}

rows_flag=""
rows_ok=""
flagged=0
unstable=0
total=0
shopt -s nullglob
for before_file in "$BEFORE"/*.png; do
Expand Down Expand Up @@ -103,12 +131,22 @@ for before_file in "$BEFORE"/*.png; do
pct="$(awk -v f="$fraction" 'BEGIN { printf "%.3f", f * 100 }')"
over="$(awk -v a="$pct" -v t="$THRESHOLD" 'BEGIN { print (a > t) ? 1 : 0 }')"

# A shot whose own run proved it rotates is not comparable, not flagged.
self_diff="$(manifest_self_diff "$name" "$BEFORE/manifest.json")"
[[ -z "$self_diff" ]] && self_diff="$(manifest_self_diff "$name" "$AFTER/manifest.json")"

note=""
[[ "$size_b" != "$size_a" ]] && note=" (size $size_b → $size_a)"
if [[ "$over" == 1 ]]; then
flagged=$((flagged + 1)); mark="✗"; cls="flag"
if [[ -n "$self_diff" ]]; then
unstable=$((unstable + 1)); mark="?"; cls="unstable"
note="$note — not comparable: unstable on its own (${self_diff}% self-diff, rotating content)"
else
flagged=$((flagged + 1)); mark="✗"; cls="flag"
fi
else
mark="✓"; cls="ok"
[[ -n "$self_diff" ]] && note="$note (unstable on its own: ${self_diff}% self-diff)"
fi
printf "%s %-32s %7s%%%s\n" "$mark" "${name%.png}" "$pct" "$note"
n="$(html_escape "$name")"
Expand All @@ -123,6 +161,11 @@ if [[ "$total" == 0 ]]; then
echo "Error: no PNG files in $BEFORE"; exit 1
fi

meta_extra=""
if [[ "$unstable" -gt 0 ]]; then
meta_extra=", $unstable not comparable (unstable on their own — rotating content)"
fi

cat > "$OUT/index.html" <<HTML
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
Expand All @@ -133,15 +176,16 @@ h1{font-size:22px;margin:0 0 4px}p.meta{color:#52606d;margin:0 0 20px}
table{border-collapse:collapse;width:100%}th,td{border-top:1px solid #e4e7eb;padding:10px;vertical-align:top;text-align:left}
td:first-child{width:180px;font-weight:600}td small{font-weight:400;color:#52606d}
tr.flag td:first-child{color:#c81e1e}tr.flag{background:#fff5f5}
tr.unstable td:first-child{color:#b46b00}tr.unstable{background:#fff8ec}
img{width:100%;max-height:600px;object-fit:cover;object-position:top;border:1px solid #e4e7eb}
</style></head><body>
<h1>$(html_escape "$TITLE")</h1>
<p class="meta">$total shots, $flagged over ${THRESHOLD}% changed (fuzz ${FUZZ}%). Before: $(html_escape "$BEFORE") · After: $(html_escape "$AFTER") · $(date '+%Y-%m-%d %H:%M')</p>
<p class="meta">$total shots, $flagged over ${THRESHOLD}% changed (fuzz ${FUZZ}%)${meta_extra}. Before: $(html_escape "$BEFORE") · After: $(html_escape "$AFTER") · $(date '+%Y-%m-%d %H:%M')</p>
<table><thead><tr><th>Page</th><th>Before</th><th>After</th><th>Changed pixels</th></tr></thead><tbody>
$rows_flag$rows_ok
</tbody></table></body></html>
HTML

echo ""
echo "$flagged of $total shots over ${THRESHOLD}% → $OUT/index.html"
echo "$flagged of $total shots over ${THRESHOLD}%${meta_extra} → $OUT/index.html"
[[ "$flagged" == 0 ]]
71 changes: 69 additions & 2 deletions scripts/updates/update-snapshot.js
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@
* --mask=<a,b> Extra CSS selectors to hide before capture
* --no-default-mask Don't hide the CF7 CAPTCHA
* --timeout=60 Seconds per capture before giving up (default: 60)
* --stability Capture every page twice and diff it against itself, so self-rotating
* content (Woo's ORDER BY RAND related products, rotating testimonials) is
* flagged "unstable" in manifest.json before it can false-flag an update
* round. Needs ImageMagick 7 (magick), like update-compare.
* --stability-threshold=0.5 Percent of self-changed pixels that marks a page unstable
*
* Requires Playwright: `npm install && npx playwright install chromium` in
* this directory (scripts/updates/).
Expand All @@ -49,11 +54,14 @@
* @flag --mask optional {.cookie-button} Extra CSS selectors to hide before capture
* @flag --no-default-mask optional {} Don't hide the Contact Form 7 CAPTCHA
* @flag --timeout optional {60} Seconds per capture before giving up
* @example wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt
* @flag --stability optional {} Capture each page twice, flag pages that differ from themselves
* @flag --stability-threshold optional {0.5} Percent of self-changed pixels that marks a page unstable
* @example wp-ops update-snapshot http://example.test --out=snapshots/before --pages-file=pages.txt --stability
*/

const fs = require('fs');
const path = require('path');
const { spawnSync } = require('child_process');

// Look for Playwright next to this script first (a wp-ops checkout after
// `npm install` here), then in the current project (a site that already has
Expand All @@ -75,6 +83,7 @@ const VIEWPORTS = {
mobile: { width: 390, height: 844 },
};
const DEFAULT_MASK = ['img.wpcf7-captcha', '.wpcf7-captchac'];
const STABILITY_FUZZ = 15; // Same per-pixel tolerance update-compare uses.

function parseArgs(argv) {
const base = argv[0];
Expand Down Expand Up @@ -104,7 +113,15 @@ function parseArgs(argv) {
...(has('no-default-mask') ? [] : DEFAULT_MASK),
...(flag('mask') ? flag('mask').split(',').map((s) => s.trim()).filter(Boolean) : []),
];
return { base: base.replace(/\/+$/, ''), out, pages, viewports, mask, timeout: (+flag('timeout') || 60) * 1000 };
const stability = has('stability');
if (stability && spawnSync('magick', ['-version']).error) {
console.error('Error: --stability needs ImageMagick 7 (magick), like update-compare. brew install imagemagick');
process.exit(1);
}
return {
base: base.replace(/\/+$/, ''), out, pages, viewports, mask, timeout: (+flag('timeout') || 60) * 1000,
stability, stabilityThreshold: +flag('stability-threshold') || 0.5,
};
}

function usage(msg) {
Expand All @@ -126,6 +143,29 @@ function withTimeout(promise, ms, label) {
]).finally(() => clearTimeout(t));
}

// How many percent of pixels differ between two captures of the same
// unchanged page. Same metric as update-compare: pad both to the same canvas
// (full-page shots of a rotating section can differ in height), then IM7's
// AE as a fraction of all pixels at fuzz 15%.
function selfDiffPct(a, b) {
const size = (f) => {
const r = spawnSync('magick', ['identify', '-format', '%wx%h', f]);
if (r.status !== 0) throw new Error(`magick identify failed for ${f}`);
return r.stdout.toString();
};
const [bw, bh] = size(a).split('x').map(Number);
const [aw, ah] = size(b).split('x').map(Number);
const w = Math.max(bw, aw), h = Math.max(bh, ah);
const diff = spawnSync('magick', [
'(', a, '-background', 'white', '-extent', `${w}x${h}`, ')',
'(', b, '-background', 'white', '-extent', `${w}x${h}`, ')',
'-fuzz', `${STABILITY_FUZZ}%`, '-metric', 'AE', '-compare',
'-format', '%[distortion]', 'info:',
]);
if (diff.status !== 0) throw new Error(`magick compare failed (${(diff.stderr || '').toString().split('\n')[0]})`);
return +(diff.stdout.toString()) * 100;
}

async function capture(browser, url, vp, file, mask) {
const context = await browser.newContext({ viewport: VIEWPORTS[vp] });
const page = await context.newPage();
Expand Down Expand Up @@ -188,11 +228,33 @@ async function capture(browser, url, vp, file, mask) {
failed++;
console.log(`✗ ERR ${vp.padEnd(7)} ${p.path} ${row.error}`);
}
// Stability pre-check (#294): capture the same page again immediately
// and diff it against itself. Anything over the threshold changes on
// every load — rotating content — and cannot be trusted in a
// before/after comparison, no matter what the update did.
if (opts.stability && !row.error && row.status < 400) {
const second = path.join(opts.out, `.stability-${p.name}-${vp}.png`);
try {
await withTimeout(capture(browser, url, vp, second, opts.mask), opts.timeout, url);
const selfPct = selfDiffPct(file, second);
row.selfDiffPct = +selfPct.toFixed(3);
row.unstable = selfPct > opts.stabilityThreshold;
if (row.unstable) {
console.log(` ⚠ ${p.name} (${vp}): UNSTABLE on its own (${row.selfDiffPct}% self-diff) — likely rotating content, mask it or expect false flags`);
}
} catch (e) {
row.stabilityError = e.message.split('\n')[0];
console.log(` ⚠ ${p.name} (${vp}): stability check failed: ${row.stabilityError}`);
} finally {
fs.rmSync(second, { force: true });
}
}
results.push(row);
}
}
await browser.close();

const unstable = results.filter((r) => r.unstable).length;
fs.writeFileSync(path.join(opts.out, 'manifest.json'), JSON.stringify({
base: opts.base,
captured_at: new Date().toISOString(),
Expand All @@ -201,5 +263,10 @@ async function capture(browser, url, vp, file, mask) {
shots: results,
}, null, 2));
console.log(`\n${results.length - failed}/${results.length} captured in ${Math.round((Date.now() - started) / 1000)}s → ${opts.out}`);
if (opts.stability) {
console.log(unstable
? `${unstable} shot(s) unstable on its own (rotating content) — see the unstable/selfDiffPct marks in manifest.json, mask the rotating sections or expect false flags`
: 'All shots stable against themselves (captured twice, 0 over threshold).');
}
process.exit(failed ? 1 : 0);
})().catch((e) => { console.error(`Error: ${e.message}`); process.exit(1); });
Loading