Repository navigation
perf: per-site dispatch credit, zero Kafka linger, faster Markdown, shared robots.txt and Lab lookups (SCR-90) - #39
Merged
qdequele merged 10 commits intoNov 12, 2026
Conversation
Base automatically changed from
scr-89-performance-review-crawl-scrape-and-map-bottlenecks
to
main
November 12, 2026 15:11
PgOutbox::enqueue and SqliteOutbox::enqueue wrapped every insert in BEGIN / INSERT / COMMIT. Every successful /scrape, /map and /search enqueues exactly one usage event on the request path, so that was three round trips for one row. One event is now a single auto-commit INSERT (atomic by itself). Several events keep the transaction, so a batch is still stored entirely or not at all; a test covers it on both stores. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
On a TTL miss every concurrent request for a key called the Lab: at each expiry (every 30 s for a busy API key) all its in-flight requests hit the Lab at once, bounded only by the 32 slots, and a slot wait over 1 s became a 503 'Lab client saturated'. A cached lookup that has to ask the Lab is now a flight: one detached task per cache key, whose answer every caller of that key waits for. A caller that goes away neither cancels the call nor strands the others; a flight that panics leaves the map before its channel closes, and its waiters start one replacement, then fail Unavailable. A fresh entry past 4/5 of its ttl is served and refreshed by a background flight: on access only, at most once per entry, through the same slots and metrics, and capped at a quarter of max_in_flight. A failed refresh changes nothing, so an entry is never served past the ttl the Lab gave for it and the stale rule is as before: a revoked credential still works for at most cache_ttl. cache_ttl 0 stays strict: a key whose last answer was 'do not cache' is asked by every caller for itself, and a caller that joined someone else's flight does not use a cache_ttl 0 answer from it but asks again. LabClient now wraps an Arc of its state, which the flights hold; its public API is unchanged. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ots.txt, fetch it once per origin The robots.txt cache built its own HTTP client: no DNS cache and a separate connection pool, so every new origin paid a second DNS lookup and a second TCP+TLS handshake before its first page. The first URLs of a new origin (up to CONCURRENT_PER_DOMAIN, more across jobs) also each fetched robots.txt, and each held a worker slot for the 10 s timeout when the host hung. - `HttpFetcherBuilder::build_with_robots` / `HttpFetcher::new_with_robots` build the fetcher and its robots.txt cache together (`HttpFetcher::robots_cache`). robots.txt is fetched through the fetcher's default client, so the connection opened for robots.txt is reused for the origin's pages, when that client applies the robots client's policy exactly: same `allow_private_ips`, redirects followed with the default cap, certificates verified, no fetcher-level custom headers or cookies. Otherwise the cache keeps a client of its own, which resolves through the fetcher's DNS cache (`RobotsCache::with_dns_cache`). Both clients come from `safe_client_builder`, so the SSRF resolver, the raw-IP checks and the redirect policy are unchanged. The robots.txt request sets its own timeout, `User-Agent` and `Accept: */*`; on the shared client it also carries the fetcher's `Accept-Language`. The crawler worker uses it; `RobotsCache::new(config)` is unchanged for every other caller. - Single-flight per origin: callers that miss the cache at the same time await one shared fetch. The fetch is not owned by the caller that started it: it runs as long as one caller awaits it and is dropped with the last one, and it leaves the in-flight map when it completes or is dropped. No lock is held across an await. - The body is an `Arc<str>`: a check no longer copies it. The `robotstxt` crate matches while it parses and has no parsed form to keep, so each check still re-reads the text. - The body is read streamed and capped at 500 KiB (truncated at the last complete line, never an error); it is decoded as before (`Response::text`: the `Content-Type` charset, UTF-8 by default). - The cache is bounded: expired entries are dropped on insert (at most once a minute, and when full), and it keeps at most 100 000 origins. What is allowed or disallowed, the TTL, how 404 / other statuses / network errors are read (no rules), `Crawl-delay`, the user agent, `respect_robots: false` and the SSRF refusals are unchanged. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… dispatch period A domain was granted one slot per dispatcher pass at most, and only when a full delay had passed since the last grant. The time between a request becoming due and the pass that granted it was lost, so a site got 1 / (ceil(delay / period) x period) requests per second instead of 1 / delay: 30 URLs/s for a 25 ms delay with a pass every 33 ms. The politeness stores now keep a virtual last-request time per domain (GCRA). A request is granted when now >= last + delay, and the grant sets last = max(last + delay, now - credit, pause end). The credit is 100 ms (DOMAIN_SPACING_CREDIT_MS, 0 = the old strict spacing) for delays under 1 s and 0 from 1 s up, so robots Crawl-delay and slow job rates are spaced exactly as before. Any n + 1 consecutive grants span at least n x delay - credit: the average rate never exceeds 1 / delay and a burst is at most 1 + credit / delay grants, within the free in-flight slots of the domain and the job. A pause banks no credit, and a 429 or an error drops the credit banked so far. PolitenessStore::try_acquire_many grants up to one slot per token in one atomic call (one Lua call in Redis) and says what one more request would get. The dispatcher asks once per domain per pass, sends what was granted and parks the domain's other URLs for the reported wait. Redis keeps last_ms as its only spacing field, so instances running the previous script can share the keys during a rolling upgrade. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… delay The spacing credit let a job with a 250 ms delay see a 150 ms gap, and a job with a 50 ms delay see several requests at one instant. A job's per_domain_delay_ms and requests_per_second, a robots Crawl-delay and default_crawl_delay_ms are floors on the gap between two requests, not averages. The credit now applies only when the effective delay is the domain's adaptive delay itself. When an explicit term is above it, the call is strict as before SCR-90: one grant, last = now, never less than the delay after the previous request. Each call is judged on its own delay, so jobs with different limits can share a domain. A credited grant advances the schedule only virtually, so the stores also keep the real time of the last grant (last_grant in memory, real_ms in Redis; the real last grant is max(real_ms, last_ms), which is correct for hashes written by the previous script). A strict call is measured from it. A credited call whose job has a lower explicit term is also never granted less than that term after the real last grant. job_rate_limit_spaces_requests_to_a_domain is back to its original assertions. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every Kafka producer used linger.ms=20, and `KafkaProducer::send` awaits the delivery report, so each awaited publish stage waited at least one linger: about 40 ms per page in the crawler (page + links, then PageCrawled), one linger per frontier dispatch pass, and the same for the content worker's outcome events and the API's job controls. Since SCR-89 the sends of a stage are concurrent, so they batch without a linger. The linger is now read from KAFKA_LINGER_MS by `ProducerBuilder::build` and `KafkaProducer::with_brokers` (every producer of every binary goes through the builder; no Args change). It must be an integer in 0..=1000; an invalid value fails producer creation, so the service stops at startup. Unset or blank means the default, now 0. Measured with the new `examples/linger_bench.rs` (real KafkaProducer, acks=all, idempotent, lz4, 1.5 KB JSON messages, local single-node Redpanda v25.3.4, 1-partition topic, quiet machine), linger 0/1/5/10/20: - one awaited send, p50: 0.13 / 1.6 / 6.6 / 12.4 / 22.4 ms - 100 sends awaited together: 0.9 / 2.2 / 7.3 / 13.1 / 23.0 ms, in 3.5-3.9 / 3 / 3 / 3 / 3 produce requests - 20,000 messages, 256 in flight: 181-189k / 173-189k / 169-183k / 158-163k / 157-160k msg/s, 436 / 435 / 435 / 435 / 435 requests (batches fill by size whatever the linger) - 64 slots each awaiting its sends: 148-158k / 137-141k / 111-129k / 84k / 75-94k msg/s - 2,000 msg/s open loop: 0.4 / 1.2 / 3.1 / 5.8 / 10.9 ms per send, 1,170 / 610 / 183 / 94 / 48 produce requests/s On a 12-partition topic the closed loop falls from 127k msg/s at 0 to 8.8k at 5 ms and 2.8k at 20 ms. A linger only buys fewer produce requests at moderate rates, and costs its full length per stage. Not measured: a broker that fsyncs (Redpanda's container image always runs with --unsafe-bypass-fsync). A slower ack makes librdkafka batch more on its own at linger 0, since messages queue while requests are in flight. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
No binary set a global allocator, and the workload allocates heavily (html5ever DOMs, String copies, JSON). Each binary crate (scrapix, scrapix-api, scrapix-frontier-service, scrapix-worker-crawler, scrapix-worker-content, scrapix-cli) gets a `mimalloc` cargo feature, on by default, that sets `#[global_allocator]` in its main.rs; `--no-default-features` keeps the system allocator. The unified `scrapix` binary takes the service crates with `default-features = false`, so only its own feature decides, and `cargo build -p scrapix --features browser` gets the allocator. scrapix-mcp (a thin stdio proxy) is unchanged. Measured on the scrapix-parser criterion benches, system allocator vs mimalloc, interleaved rounds, minimum of the per-round medians (mimalloc / system): - Linux arm64 container (rust:1-bookworm, glibc), 6 rounds: html_parsing large 0.97, very large 0.97; parse_only 0.96; parse_and_markdown 0.92; parse_markdown_language 0.92; markdown_conversion large 0.90; content_extraction 0.94-0.96; batch_100_mixed 0.93 (15.9 ms -> 14.8 ms) - macOS arm64, 10 rounds: html_parsing 0.98-0.99; parse_only 0.94; parse_and_markdown 0.92; markdown_conversion large 0.91; content_extraction 0.96-0.97; batch_100_mixed 0.98 The two fastest rounds of each side agree within 2%. The benches are single-threaded; production is Linux x86-64 with many threads, which was not measured. Cost: peak RSS of the whole bench run rose from 38-46 MB to 142-159 MB on Linux (36-50 MB to 113-162 MB on macOS): mimalloc keeps freed memory longer. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ag and per link Markdown conversion is most of a page's parse time. Around html2md, the conversion ran nine lazy regexes over the whole page to drop script, style, noscript, svg, head, nav, footer, header and aside, about seven more over the Markdown, and a capture regex per line to resolve links. The output is indexed and returned by /scrape, so it has to stay the same bytes: this only makes the same pipeline cheaper. - Tag stripping: the page is scanned once for the places where one of the nine names follows `<` or `</`, and each tag walks that short list instead of the page. The passes stay sequential (a later tag sees the text without the earlier ones' blocks), and the list is patched where a removed block joins two halves into a tag. A page holding `ſ` (U+017F, which the regexes fold to `s`) still goes through the regexes. - Markdown cleanup: each regex is skipped when the text cannot match it (no `<`, no `](`, no `[` followed by whitespace), and lines are cleaned in place instead of through a String per line. - Link resolution: a line without `](` is copied; on a plain line (no `"`, and no `d](e)` gives `[b](c)`), and that is the output to preserve. The previous implementation is kept under cfg(test) (markdown/legacy.rs) and a differential suite (markdown/differential.rs) compares every public entry point and every stage on 1750 pages: 108 handwritten around the traps, 800 seeded random ones, 842 HTML literals harvested from the workspace's sources, plus 6000 random link lines. criterion, median, two runs each, same session (before -> after): markdown_conversion/simple_html 6.3, 6.0 µs -> 5.1, 5.5 µs markdown_conversion/medium_html 36.6, 34.9 µs -> 32.1, 30.8 µs markdown_conversion/large_html/100p 868, 870 µs -> 787, 759 µs markdown_conversion/realistic_html/60s 901, 875 µs -> 732, 705 µs markdown_main_content/realistic_html_60s 957, 919 µs -> 648, 626 µs markdown_for_page/medium_html 40.5, 38.5 µs -> 28.6, 27.4 µs markdown_for_page/large_html_100p 1303, 1249 µs -> 838, 815 µs markdown_for_page/realistic_html_60s 1084, 1060 µs -> 688, 667 µs full_parsing_pipeline/parse_and_markdown 58.6, 56.8 µs -> 48.1, 46.1 µs html2md is now 84 to 88% of the conversion (52 to 66% before): what is left is inside the dependency. Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…annot overflow the stack absolutize_markdown_links recursed once per link nested in a link text. html2md only nests once (a linked image), but page text that looks like markdown, such as "[" repeated 20,000 times (160 KB), made it one stack frame per repetition: the process aborted with a stack overflow. A content worker would crash, get the page redelivered and crash again. The recursion now stops at 32 levels; past that the text of a link is kept as it is. Output is unchanged up to the cap (the differential suite and a new test against the legacy implementation). Refs SCR-90 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The end-to-end benchmark shows no crawl throughput gain from mimalloc and about twice the content worker's peak memory, so the feature is off by default. Adds the CHANGELOG entries, the env var docs and the measured numbers for the spacing credit and the linger default. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
qdequele
force-pushed
the
scr-90-performance-follow-ups-per-site-dispatch-ceiling
branch
from
November 12, 2026 15:11
1344fec to
c4a7bbf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups to #38 (SCR-90). Stacked on #38: merge that one first, then this PR retargets to
main.What changes
DOMAIN_SPACING_CREDIT_MS(default 100) of that time back. Delays set by the job, by robots.txt, or of 1 s or more stay a strict floor.KAFKA_LINGER_MS(0 to 1000) sets it; an invalid value refuses to start.cache_ttl. The revocation window is unchanged. Usage events are recorded in one database round trip.--features mimalloc), off by default.main). Nesting is now followed 32 levels deep.Measured
End to end on a laptop (macOS arm64, Redpanda in dev mode without fsync, local test site), pages per second. Every run completed with exact counts and no errors.
DOMAIN_SPACING_CREDIT_MS=0the rate is 24 pages/s.KAFKA_LINGER_MS=20the 20-site crawl drops to about 65 pages/s (two runs).Not verified
0, a single site is about 9% slower than on perf: remove the serial Kafka sends and other crawl, scrape and map bottlenecks (SCR-89) #38 (24 vs 26.5 pages/s); not explained. It does not show with the default.Tests
cargo fmt,cargo check,cargo clippy --all --all-targets -- -D warningspass. The workspace tests and the Redis-backed frontier tests passed (0 failures) on the commit before the last one, which only changes the mimalloc default, comments and docs.Deploy notes
KAFKA_LINGER_MS(default0) andDOMAIN_SPACING_CREDIT_MS(default100). Nothing to set on the box.DOMAIN_SPACING_CREDIT_MS=0on the frontier to keep strict spacing.🤖 Generated with Claude Code