v1.41.0 — @harperfast/prerender-browser

@harperfast/prerender-browser 1.41.0

New: userAgentSuffix (default '', off)

A product token appended, after one space, to every User-Agent a render sends: the navigation, every subresource, navigator.userAgent inside the page, and the document prefetch. The origin can then tell render traffic from real visitors. Today the bypass token identifies renders to the CDN, but the origin's client-side analytics never sees that header.

{ "userAgentSuffix": "HarperRender/1.0" }

• A device profile without a userAgent gets Chrome's own UA plus the token.
• Client hints: on a profile without a userAgent, this turns Chrome's UA into an override, and Chrome then sends no Sec-CH-UA* headers and an empty navigator.userAgentData, exactly as for a profile that sets userAgent itself. Profiles that already set one see no difference.
• Keep a real-browser UA as the base (sites gate third-party tags on it), keep prerender out of the token, and compare a sample of renders with and without it before enabling.

Changed: the document prefetch follows scoped overrides

prefetchHeaders now builds from the config the render resolves, so a device UA, userAgentSuffix or extraHeaders that a scoped override sets reaches the prefetch exactly as it reaches the page. Before, an override that changed a device's userAgent or extraHeaders reached the page but not its prefetch.

No other changes since v1.40.0 (#250).

prerender-v0.105.0 — @harperfast/prerender

@harperfast/prerender 0.105.0

New: origin.forwardUserAgent (default off)

Sends the crawler's own User-Agent to the origin on the cache-miss proxy fetch, with suffix appended (default HarperProxy/1.0), instead of the fixed per-device browser strings in origin.userAgents. The origin's own bot metrics can then tell which crawler a Harper fetch was made for, and still recognise it as Harper's.

'@harperfast/prerender':
  origin:
    forwardUserAgent:
      enabled: true
      suffix: HarperProxy/1.0   # appended after one space; '' forwards the UA verbatim

• Only the proxy fetch changes. Renders, change probes, serve checks and sitemap fetches keep their own UAs. With no crawler UA on the request, the proxy falls back to origin.userAgents.
• Before enabling: the edge must exempt securityToken from everything it does to an unverified crawler. A crawler UA from Harper's addresses fails IP verification. That covers the rule that routes bot traffic to Harper, and bot mitigation's deny, challenge or serve-alternate actions. Then fetch a URL through the edge with the token, once with a browser UA and once with a crawler UA plus the suffix, and compare status, canonical and body. The raw cache replays a proxied document to other crawlers, so any difference means leave this off.
• Origin-side reports that count crawlers by UA substring must exclude the suffix. Otherwise each proxied miss is counted twice.

Changed: the change probe has its own User-Agents

New changeProbe.userAgents (per device): the same browser strings with a HarperProbe/1.0 token. Used by the sweep, the canary and serve checks. On upgrade, probe requests switch from …HarperProxy/1.0 to …HarperProbe/1.0 with no config change. If anything on your edge or origin keys on HarperProxy for probe traffic (allowlist, rate exemption, log filter), add HarperProbe first.

No other changes since prerender-v0.104.0 (#249).

prerender-v0.104.0 — @harperfast/prerender

@harperfast/prerender 0.104.0

New: network.publicPort

The port other cluster nodes and the render fleet use to reach a node: the port in every render job's callbackOrigin (where the fleet posts its result) and in every peer call (serve-path rescue, invalidation heal, the explainer's owner read).

• 0 (default) = Harper's own http.securePort, the same behavior as before.
• Set 443 on hosts fronted by Fabric's Symphony proxy. Those hosts publish Harper on 80, 443, 9925, 8883 and 9933 but not 9926, so a result posted to :9926 never arrives. The render fleet's queuePort must match.
• Live: applies on the next claim or peer call, no restart.

'@harperfast/prerender':
  network:
    publicPort: 443

callbackOrigin is now built by the same peerOrigin() the peer calls use, so the two can't disagree. Side effect: the callback gains peerOrigin's 127.0.0.1 / ::1 → http rule. It previously checked only localhost.

No other changes since prerender-v0.103.0 (#248).

prerender-v0.103.0 — @harperfast/prerender

Request sampling: a live, configurable log of sampled bot requests, for learning how often each crawler revisits a page by page type, bot and device (#246, #245).

• sampling (new config group, off by default). Named samplers:
  • Match declaratively: routes (path or class), bots, devices, methods, cache statuses, sources, statuses sent, exact urls, urlPattern.
  • Sample URL-stably (fnv1a32(salt + '\n' + url) < rate * 2^32, so every request for a sampled URL is recorded) or per request.
  • Record fields from a fixed catalog, plus request headers by name. Credentials are always refused, including every header the plugin authenticates on.
  • Live through override rows; no restart.
• Cost: a null check when off, two map lookups when no sampler applies, and a preallocated per-worker ring when one does. Nothing awaits on the request path.
• Storage: append-only gzipped NDJSON files on each node, <sampling.directory>/<sampler>/<YYYY-MM-DD>.w<worker>.t<thread>.ndjson.gz. The directory defaults to prerender-sampling under Harper's root and is file-only. Worker 0 deletes files older than sampling.keepDays (14). Picks not recorded (capped, dropped, lost) are written as counters lines in the same files.
• Admin: GET /prerender_admin/sampling (samplers, worker counters, node files) and GET /prerender_admin/samples?sampler&from&to (NDJSON or gzip, paged). Both are per node. The override dry run refuses sampler entries the compiler would drop, newly duplicated names, and credential-header renames onto a recorded header.
• Fix: the render-now header, which carries renderNow.token, is no longer forwarded to the origin.
• Schema: unchanged. Config: new sampling group; nothing else changes.

v1.40.0 — @harperfast/prerender-browser

Two changes for serving one entity's render at its other spellings (#239).

• Self-references are fragment-only. The serialized snapshot rewrites every CSS url(<this page's URL>#id) to url(#id), so a reference a script built from location still resolves when the snapshot is served at another URL. Measured in Chrome 148: served at another URL, an absolute self-reference paints no fill, and the browser fetches the render URL trying to resolve it. Always on; nothing changes at the URL the page was rendered at.
• declaredCanonical. A non-indexable render whose canonical names another URL (canonical-mismatch, or a strict canonical-variant) posts that canonical, resolved against the rendered URL, with or without content. Never for a noindex page. Absent from every other result, so an older plugin never sees the key. Plugin 0.102.0 records it in its entity registry.

Verified on live product pages against 1.38.0: no unexpected differences in the stored HTML, identical outcome, status, offers and page facts.

prerender-v0.99.0 — @harperfast/prerender

Serve-time checks can use the whole probe budget they are given (#238).

• New live option changeProbe.serveCheck.concurrency (default 16): checks in flight per worker, previously a fixed 4. Every check still waits for its slot in the node-wide budget; changeProbe.ratePerSecond is unchanged.
• While a pass runs, it reserves twice the checks' recent rate only while checks are waiting for budget; otherwise it leaves exactly what they used.

No schema change.

prerender-v0.102.0 — @harperfast/prerender

Every fetch from the origin resolves an entity's canonical, and the entity serve trusts the newest word (#243).

• Four observers write the entity registry: the change probe, renders (a stored page's canonical, and the canonical a canonical verdict declares, browser >= 1.40.0), serve-time checks, and documents proxied from the origin on an entityServe route (read off the head as the crawler's bytes stream by; they move rows, never create them). Any of them can adopt a canonical no target holds.
• The entity serve:
  • answers a spelling suppressed as a canonical verdict, but only with the page that verdict named (Target.suppressedCanonical, new) or one confirmed after it;
  • refuses a page the registry has since heard re-slugged (moved);
  • lets the registry break a tie between two servable spellings;
  • falls through on a query string (has-query);
  • reports each evaluation's cost (entity_serve_ms).
• served_wrong (new prerender_ops series): one sample per cached copy found to differ from the origin while it was being served, by detector and field, with an upper bound on how long it was out. No per-serve work.
• Schema: Target.suppressedCanonical (new attribute). No config changes; existing options gain readers.

prerender-v0.101.0 — @harperfast/prerender

The entity registry (#241, phase 1 of #166).

• Entity (new table, replicated, not residency-pinned): one row per entity a route declares with entityPrefix, keyed by the entity prefix, holding its current canonical URL. Written only by observations of the origin, the newer of two disagreeing ones winning, and only when the canonical moves.
• On by default (entities.enabled): a route's entityPrefix is already the opt-in, so a deployment with no such route has no entities.
• Adoption (entities.adopt, dry run by default): when an observation names a canonical that is another URL of the same entity and no target holds it in rotation, its target is filed due now. Bounded by maxPerHour per node (one budget across worker threads, with a separate lane for dry runs) and by retryAfter per entity.
• The change probe reports its rule's mapped canonical slot, and stored renders their page's canonical. explain shows the entity row.
• Metrics: prerender_ops / entity_canonical and canonical_adopt.

prerender-v0.100.0 — @harperfast/prerender

The entity serve, and canonical-specific confirmation (#240, #237).

• ingress.routes[].entityServe (needs entityPrefix; governed by ingress.entityServe, dry run by default): a true miss for a spelling of an entity's URL is answered from the cached render of the entity's canonical URL, when that page is fresh, indexable, names itself canonical and had its canonical confirmed since the last anchor. Anything else falls through to the miss path as before. Outcomes on prerender_ops / entity_serve; served as bot_serve source and status entity.
• PageCheck.canonicalAgreed (new column on a sealed table: the workers restart on deploy): written by serve-time checks and the sweep when they compared the page's canonical and agreed. The entity serve accepts only that, or a render since the anchor, as confirmation.
• Only for a site that answers every spelling of an entity with the same document; the README gives the experiment that settles it.

prerender-console-v0.26.0 — @harperfast/prerender-console

Traffic, for plugin 0.102.0: an Entity registry card (canonical moves by the observer that saw them, adoptions or the dry-run would-adopt number, and why the rest were not adopted), a Served while wrong card (wrong copies by detector, how long they were out, and which fields), and the entity serve's new outcomes and cost per evaluation.