Skip to content

perf: per-site dispatch credit, zero Kafka linger, faster Markdown, shared robots.txt and Lab lookups (SCR-90) - #39

Merged
qdequele merged 10 commits into
mainfrom
scr-90-performance-follow-ups-per-site-dispatch-ceiling
Nov 12, 2026
Merged

qdequele merged 10 commits into
mainfrom
scr-90-performance-follow-ups-per-site-dispatch-ceiling

Conversation

@qdequele

Copy link
Copy Markdown
Owner

Follow-ups to #38 (SCR-90). Stacked on #38: merge that one first, then this PR retargets to main.

What changes

  • Spacing credit for sites paced by the adaptive delay. The frontier lost the time between a request becoming due and the dispatch pass that sent it. The politeness stores (Redis Lua and in-memory) now keep a schedule per domain and give up to DOMAIN_SPACING_CREDIT_MS (default 100) of that time back. Delays set by the job, by robots.txt, or of 1 s or more stay a strict floor.
  • Kafka producer linger defaults to 0 (was 20 ms). Every publish stage awaits its delivery report, so each paid one linger. KAFKA_LINGER_MS (0 to 1000) sets it; an invalid value refuses to start.
  • Markdown conversion is 12 to 37% faster on the parser benches, byte-identical output (the previous implementation is kept as a test reference and compared on 1,750 pages).
  • robots.txt: one fetch per new origin, shared by the URLs waiting for it, through the page fetcher's DNS cache and connection pool. Body capped at 500 KiB, cache bounded at 100,000 origins.
  • Lab lookups: concurrent requests for one credential share one Lab call, and a key in use is refreshed in the background from 80% of its cache_ttl. The revocation window is unchanged. Usage events are recorded in one database round trip.
  • mimalloc as an opt-in cargo feature (--features mimalloc), off by default.
  • Fix: page text shaped like deeply nested Markdown links overflowed the stack and aborted the content worker (also on main). Nesting is now followed 32 levels deep.

Measured

End to end on a laptop (macOS arm64, Redpanda in dev mode without fsync, local test site), pages per second. Every run completed with exact counts and no errors.

Scenario main #38 this PR vs main
Single site 19.5 26.5 37.0 x1.9
20 sites in one job 22.5 31.4 about 84 (73–91) x3.7
10 concurrent jobs 21.6 216.5 282.3 x13
  • The credit is the whole single-site gain: with DOMAIN_SPACING_CREDIT_MS=0 the rate is 24 pages/s.
  • Linger 0 only matters across many sites: with KAFKA_LINGER_MS=20 the 20-site crawl drops to about 65 pages/s (two runs).
  • mimalloc gave no crawl throughput gain and doubled the content worker's peak memory (about 230 MB against 105 MB), hence opt-in.
  • The benchmarked commit predates the Markdown change and the nested-link fix.

Not verified

Tests

cargo fmt, cargo check, cargo clippy --all --all-targets -- -D warnings pass. The workspace tests and the Redis-backed frontier tests passed (0 failures) on the commit before the last one, which only changes the mimalloc default, comments and docs.

Deploy notes

  • Env: two new optional variables, KAFKA_LINGER_MS (default 0) and DOMAIN_SPACING_CREDIT_MS (default 100). Nothing to set on the box.
  • Behavior change at deploy: a site paced by the adaptive delay can receive up to 5 requests at once at the 25 ms floor (same average rate). Set DOMAIN_SPACING_CREDIT_MS=0 on the frontier to keep strict spacing.
  • Migrations: none.
  • Rollout order: after perf: remove the serial Kafka sends and other crawl, scrape and map bottlenecks (SCR-89) #38. As for any engine upgrade, deploy with no job running.

🤖 Generated with Claude Code

@qdequele
qdequele added this pull request to stack #40 November 12, 2026 15:10
Base automatically changed from scr-89-performance-review-crawl-scrape-and-map-bottlenecks to main November 12, 2026 15:11
qdequele and others added 10 commits November 12, 2026 16:11
PgOutbox::enqueue and SqliteOutbox::enqueue wrapped every insert in BEGIN / INSERT / COMMIT. Every successful /scrape, /map and /search enqueues exactly one usage event on the request path, so that was three round trips for one row.

One event is now a single auto-commit INSERT (atomic by itself). Several events keep the transaction, so a batch is still stored entirely or not at all; a test covers it on both stores.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
On a TTL miss every concurrent request for a key called the Lab: at each expiry (every 30 s for a busy API key) all its in-flight requests hit the Lab at once, bounded only by the 32 slots, and a slot wait over 1 s became a 503 'Lab client saturated'.

A cached lookup that has to ask the Lab is now a flight: one detached task per cache key, whose answer every caller of that key waits for. A caller that goes away neither cancels the call nor strands the others; a flight that panics leaves the map before its channel closes, and its waiters start one replacement, then fail Unavailable.

A fresh entry past 4/5 of its ttl is served and refreshed by a background flight: on access only, at most once per entry, through the same slots and metrics, and capped at a quarter of max_in_flight. A failed refresh changes nothing, so an entry is never served past the ttl the Lab gave for it and the stale rule is as before: a revoked credential still works for at most cache_ttl.

cache_ttl 0 stays strict: a key whose last answer was 'do not cache' is asked by every caller for itself, and a caller that joined someone else's flight does not use a cache_ttl 0 answer from it but asks again.

LabClient now wraps an Arc of its state, which the flights hold; its public API is unchanged.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ots.txt, fetch it once per origin

The robots.txt cache built its own HTTP client: no DNS cache and a
separate connection pool, so every new origin paid a second DNS lookup
and a second TCP+TLS handshake before its first page. The first URLs of a
new origin (up to CONCURRENT_PER_DOMAIN, more across jobs) also each
fetched robots.txt, and each held a worker slot for the 10 s timeout when
the host hung.

- `HttpFetcherBuilder::build_with_robots` / `HttpFetcher::new_with_robots`
  build the fetcher and its robots.txt cache together
  (`HttpFetcher::robots_cache`). robots.txt is fetched through the
  fetcher's default client, so the connection opened for robots.txt is
  reused for the origin's pages, when that client applies the robots
  client's policy exactly: same `allow_private_ips`, redirects followed
  with the default cap, certificates verified, no fetcher-level custom
  headers or cookies. Otherwise the cache keeps a client of its own, which
  resolves through the fetcher's DNS cache
  (`RobotsCache::with_dns_cache`). Both clients come from
  `safe_client_builder`, so the SSRF resolver, the raw-IP checks and the
  redirect policy are unchanged. The robots.txt request sets its own
  timeout, `User-Agent` and `Accept: */*`; on the shared client it also
  carries the fetcher's `Accept-Language`. The crawler worker uses it;
  `RobotsCache::new(config)` is unchanged for every other caller.
- Single-flight per origin: callers that miss the cache at the same time
  await one shared fetch. The fetch is not owned by the caller that
  started it: it runs as long as one caller awaits it and is dropped with
  the last one, and it leaves the in-flight map when it completes or is
  dropped. No lock is held across an await.
- The body is an `Arc<str>`: a check no longer copies it. The `robotstxt`
  crate matches while it parses and has no parsed form to keep, so each
  check still re-reads the text.
- The body is read streamed and capped at 500 KiB (truncated at the last
  complete line, never an error); it is decoded as before
  (`Response::text`: the `Content-Type` charset, UTF-8 by default).
- The cache is bounded: expired entries are dropped on insert (at most
  once a minute, and when full), and it keeps at most 100 000 origins.

What is allowed or disallowed, the TTL, how 404 / other statuses /
network errors are read (no rules), `Crawl-delay`, the user agent,
`respect_robots: false` and the SSRF refusals are unchanged.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… dispatch period

A domain was granted one slot per dispatcher pass at most, and only when
a full delay had passed since the last grant. The time between a request
becoming due and the pass that granted it was lost, so a site got
1 / (ceil(delay / period) x period) requests per second instead of
1 / delay: 30 URLs/s for a 25 ms delay with a pass every 33 ms.

The politeness stores now keep a virtual last-request time per domain
(GCRA). A request is granted when now >= last + delay, and the grant sets
last = max(last + delay, now - credit, pause end). The credit is 100 ms
(DOMAIN_SPACING_CREDIT_MS, 0 = the old strict spacing) for delays under
1 s and 0 from 1 s up, so robots Crawl-delay and slow job rates are
spaced exactly as before. Any n + 1 consecutive grants span at least
n x delay - credit: the average rate never exceeds 1 / delay and a burst
is at most 1 + credit / delay grants, within the free in-flight slots of
the domain and the job. A pause banks no credit, and a 429 or an error
drops the credit banked so far.

PolitenessStore::try_acquire_many grants up to one slot per token in one
atomic call (one Lua call in Redis) and says what one more request would
get. The dispatcher asks once per domain per pass, sends what was
granted and parks the domain's other URLs for the reported wait.

Redis keeps last_ms as its only spacing field, so instances running the
previous script can share the keys during a rolling upgrade.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… delay

The spacing credit let a job with a 250 ms delay see a 150 ms gap, and a
job with a 50 ms delay see several requests at one instant. A job's
per_domain_delay_ms and requests_per_second, a robots Crawl-delay and
default_crawl_delay_ms are floors on the gap between two requests, not
averages.

The credit now applies only when the effective delay is the domain's
adaptive delay itself. When an explicit term is above it, the call is
strict as before SCR-90: one grant, last = now, never less than the
delay after the previous request. Each call is judged on its own delay,
so jobs with different limits can share a domain.

A credited grant advances the schedule only virtually, so the stores
also keep the real time of the last grant (last_grant in memory, real_ms
in Redis; the real last grant is max(real_ms, last_ms), which is correct
for hashes written by the previous script). A strict call is measured
from it. A credited call whose job has a lower explicit term is also
never granted less than that term after the real last grant.

job_rate_limit_spaces_requests_to_a_domain is back to its original
assertions.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every Kafka producer used linger.ms=20, and `KafkaProducer::send` awaits
the delivery report, so each awaited publish stage waited at least one
linger: about 40 ms per page in the crawler (page + links, then
PageCrawled), one linger per frontier dispatch pass, and the same for the
content worker's outcome events and the API's job controls. Since SCR-89
the sends of a stage are concurrent, so they batch without a linger.

The linger is now read from KAFKA_LINGER_MS by `ProducerBuilder::build`
and `KafkaProducer::with_brokers` (every producer of every binary goes
through the builder; no Args change). It must be an integer in 0..=1000;
an invalid value fails producer creation, so the service stops at
startup. Unset or blank means the default, now 0.

Measured with the new `examples/linger_bench.rs` (real KafkaProducer,
acks=all, idempotent, lz4, 1.5 KB JSON messages, local single-node
Redpanda v25.3.4, 1-partition topic, quiet machine), linger 0/1/5/10/20:
- one awaited send, p50: 0.13 / 1.6 / 6.6 / 12.4 / 22.4 ms
- 100 sends awaited together: 0.9 / 2.2 / 7.3 / 13.1 / 23.0 ms,
  in 3.5-3.9 / 3 / 3 / 3 / 3 produce requests
- 20,000 messages, 256 in flight: 181-189k / 173-189k / 169-183k /
  158-163k / 157-160k msg/s, 436 / 435 / 435 / 435 / 435 requests
  (batches fill by size whatever the linger)
- 64 slots each awaiting its sends: 148-158k / 137-141k / 111-129k /
  84k / 75-94k msg/s
- 2,000 msg/s open loop: 0.4 / 1.2 / 3.1 / 5.8 / 10.9 ms per send,
  1,170 / 610 / 183 / 94 / 48 produce requests/s
On a 12-partition topic the closed loop falls from 127k msg/s at 0 to
8.8k at 5 ms and 2.8k at 20 ms. A linger only buys fewer produce
requests at moderate rates, and costs its full length per stage.

Not measured: a broker that fsyncs (Redpanda's container image always
runs with --unsafe-bypass-fsync). A slower ack makes librdkafka batch
more on its own at linger 0, since messages queue while requests are in
flight.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
No binary set a global allocator, and the workload allocates heavily
(html5ever DOMs, String copies, JSON). Each binary crate (scrapix,
scrapix-api, scrapix-frontier-service, scrapix-worker-crawler,
scrapix-worker-content, scrapix-cli) gets a `mimalloc` cargo feature, on
by default, that sets `#[global_allocator]` in its main.rs;
`--no-default-features` keeps the system allocator. The unified
`scrapix` binary takes the service crates with `default-features =
false`, so only its own feature decides, and
`cargo build -p scrapix --features browser` gets the allocator.
scrapix-mcp (a thin stdio proxy) is unchanged.

Measured on the scrapix-parser criterion benches, system allocator vs
mimalloc, interleaved rounds, minimum of the per-round medians
(mimalloc / system):
- Linux arm64 container (rust:1-bookworm, glibc), 6 rounds:
  html_parsing large 0.97, very large 0.97; parse_only 0.96;
  parse_and_markdown 0.92; parse_markdown_language 0.92;
  markdown_conversion large 0.90; content_extraction 0.94-0.96;
  batch_100_mixed 0.93 (15.9 ms -> 14.8 ms)
- macOS arm64, 10 rounds: html_parsing 0.98-0.99; parse_only 0.94;
  parse_and_markdown 0.92; markdown_conversion large 0.91;
  content_extraction 0.96-0.97; batch_100_mixed 0.98
The two fastest rounds of each side agree within 2%. The benches are
single-threaded; production is Linux x86-64 with many threads, which
was not measured.

Cost: peak RSS of the whole bench run rose from 38-46 MB to 142-159 MB
on Linux (36-50 MB to 113-162 MB on macOS): mimalloc keeps freed
memory longer.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ag and per link

Markdown conversion is most of a page's parse time. Around html2md, the
conversion ran nine lazy regexes over the whole page to drop script,
style, noscript, svg, head, nav, footer, header and aside, about seven
more over the Markdown, and a capture regex per line to resolve links.
The output is indexed and returned by /scrape, so it has to stay the same
bytes: this only makes the same pipeline cheaper.

- Tag stripping: the page is scanned once for the places where one of the
  nine names follows `<` or `</`, and each tag walks that short list
  instead of the page. The passes stay sequential (a later tag sees the
  text without the earlier ones' blocks), and the list is patched where a
  removed block joins two halves into a tag. A page holding `ſ` (U+017F,
  which the regexes fold to `s`) still goes through the regexes.
- Markdown cleanup: each regex is skipped when the text cannot match it
  (no `<`, no `](`, no `[` followed by whitespace), and lines are cleaned
  in place instead of through a String per line.
- Link resolution: a line without `](` is copied; on a plain line (no `"`,
  and no `![` or a single `[`) links are found by hand; a target without
  `:` is no longer parsed as an absolute URL before being joined. Other
  lines keep the regex: on them regex 1.12 does not always return the
  leftmost match (`[![](a [b](c)d](e)` gives `[b](c)`), and that is the
  output to preserve.

The previous implementation is kept under cfg(test) (markdown/legacy.rs)
and a differential suite (markdown/differential.rs) compares every public
entry point and every stage on 1750 pages: 108 handwritten around the
traps, 800 seeded random ones, 842 HTML literals harvested from the
workspace's sources, plus 6000 random link lines.

criterion, median, two runs each, same session (before -> after):
  markdown_conversion/simple_html          6.3, 6.0 µs   -> 5.1, 5.5 µs
  markdown_conversion/medium_html          36.6, 34.9 µs -> 32.1, 30.8 µs
  markdown_conversion/large_html/100p      868, 870 µs   -> 787, 759 µs
  markdown_conversion/realistic_html/60s   901, 875 µs   -> 732, 705 µs
  markdown_main_content/realistic_html_60s 957, 919 µs   -> 648, 626 µs
  markdown_for_page/medium_html            40.5, 38.5 µs -> 28.6, 27.4 µs
  markdown_for_page/large_html_100p        1303, 1249 µs -> 838, 815 µs
  markdown_for_page/realistic_html_60s     1084, 1060 µs -> 688, 667 µs
  full_parsing_pipeline/parse_and_markdown 58.6, 56.8 µs -> 48.1, 46.1 µs

html2md is now 84 to 88% of the conversion (52 to 66% before): what is
left is inside the dependency.

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…annot overflow the stack

absolutize_markdown_links recursed once per link nested in a link text.
html2md only nests once (a linked image), but page text that looks like
markdown, such as "[![a](b)" repeated 20,000 times (160 KB), made it one
stack frame per repetition: the process aborted with a stack overflow. A
content worker would crash, get the page redelivered and crash again.

The recursion now stops at 32 levels; past that the text of a link is kept
as it is. Output is unchanged up to the cap (the differential suite and a
new test against the legacy implementation).

Refs SCR-90

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The end-to-end benchmark shows no crawl throughput gain from mimalloc and
about twice the content worker's peak memory, so the feature is off by
default. Adds the CHANGELOG entries, the env var docs and the measured
numbers for the spacing credit and the linger default.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@qdequele
qdequele force-pushed the scr-90-performance-follow-ups-per-site-dispatch-ceiling branch from 1344fec to c4a7bbf Compare November 12, 2026 15:11
@qdequele
qdequele merged commit de93bb2 into main Nov 12, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant