Repository navigation
perf: remove the serial Kafka sends and other crawl, scrape and map bottlenecks (SCR-89) - #38
Merged
qdequele merged 17 commits intoNov 12, 2026
Conversation
matches_glob built a token list from the pattern and one table row per token on every call, and it is called per link and per pattern. The literal part before the first `*` is now a plain prefix comparison (most patterns are `https://host/path/*`, most non-matching URLs differ there), tokens are read from the pattern as it is walked, and the two table rows are kept per thread. Matching is unchanged: a test compares it with the previous implementation on a grid of patterns and URLs. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… and fail oversized pages A send waits for its delivery report (the producer's 20 ms linger plus the broker ack), and the handler awaited every send in series while holding a worker slot: a page with 100 links took about 2 s to publish. - A page's link messages are sent together (at most 256 in flight) and awaited together; any failed send still leaves the message unacked. The raw page, the links, the PageRank input and the page history are published together; UrlsDiscovered and PageCrawled together after them, so PageCrawled still follows the page and its links. - FetchFeedback is sent while the page is published instead of before it. It is still queued first and awaited (2 s bound) before the ack. - Sitemap entries are sent 256 at a time instead of one after the other. - A worker publishes a link once per job: links it already sent and counted are skipped, and links_published / UrlsDiscovered count the links actually sent. A link is recorded only after the accounting event counting it was sent, so a redelivered page sends and counts its links again. The set is bounded (64-bit hashes, two generations of 50,000 per job, 64 jobs) and expires after 10 to 20 minutes, so a link the frontier refused on a full queue is sent again later. - A page whose serialized message is over the producer's message size limit (a 7.5 to 10 MB document once base64-encoded, or HTML grown by JSON escaping) is reported PageFailed and acked. It used to fail every send, stay unacked and end the job FailStalled. A broker refusal of a large page is terminal too. - Link extraction of a page over 32 KiB runs on the blocking pool. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The router had no compression layer although tower-http's
compression-gzip feature is enabled: scrape, results and analytics
payloads (markdown, HTML, JSON) went out uncompressed.
Add a CompressionLayer around every route, negotiated by
Accept-Encoding, for bodies of at least 1 KB. Event streams
(text/event-stream), WebSocket upgrades, images and gRPC are never
compressed, so /job/{id}/events still reaches the client event by
event. /health stays plain (too small); /metrics is gzipped only for a
scraper that asks for it.
Refs SCR-89
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The scrapes of /batch/scrape and /extract were futures polled by the job's own task (buffer_unordered): while that task awaited the store write of a finished result, the fetches in flight were not polled. Each URL now runs on its own task (ScrapeTasks: a JoinSet capped at the job's concurrency), so the scrapes progress while results are stored. The semantics are kept: a URL's task starts when a slot frees up and checks the job's gate first, so cancel stops the URLs not started yet, pause holds them, and the scrapes in flight finish; a batch still skips the remaining URLs once the account is out of credits; the extract job's AI usage tally is carried into the tasks. A scrape that panics now fails its URL (internal_error) instead of hanging the job. The result item is read from the stored payload in place, without the deep clone of the whole result. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
/scrape fetch: - One retry instead of the crawler's three: a page answering 429/5xx cost four fetches and 3.5 s of backoff before the error came back. - timeout_ms bounds the whole fetch for requests with headers or cookies too (it was only each attempt's timeout there). - Requests with headers (and no cookie) use the shared fetcher (connection pool, DNS cache) with the headers passed per request, instead of building a robots client and an HTTP client each time. Cookies, and a Cookie header, keep a one-off fetcher (its cookie store must live and die with the request), which now shares a DNS cache and a robots cache built once. /scrape HTML stage: - Runs in spawn_blocking instead of on the async runtime. - The page is parsed into a DOM once (twice when include/exclude selectors change it) instead of once per format, through the extractors' *_from_dom functions; the markdown is converted once when markdown and content both need it; no copy of the HTML when there is no selector. - exclude_selectors parse the page once instead of once per selector. Matched elements are still cut out of the source text, so the rest of the page is unchanged; an element whose serialization is not its source text (unquoted attribute, unclosed tag), which used to be kept silently, is now removed by detaching it from the tree. A test compares the stage with the previous per-format pipeline over every combination of formats and options. /search: one pooled client (5 s connect, 30 s request timeout) instead of a new client without a timeout per request; a timeout is the timeout error (504). /map: with depth > 0 the start page is fetched while the sitemaps are read, and pages of 128 KiB and more are parsed for links off the runtime. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…stores
A politeness slot is granted to a domain at most once per effective delay,
whatever CONCURRENT_PER_DOMAIN is (it only caps the in-flight count). The
dispatcher did not know that delay, so after a grant it asked the store
again for the domain's next URL only to be told to wait: half of all
acquire calls in a single-site crawl.
PolitenessStore::try_acquire_spaced returns, with a grant, the effective
delay the grant was checked against (zero when the domain can be granted
again right away). The in-memory scheduler and the Redis script (which now
returns {status, ms}) report the same value; try_acquire is unchanged.
Refs SCR-89
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… every tick The dispatcher handled the active jobs one after another, and each job awaits its batch's bus sends (about 25 ms with the producer's linger and acks=all): ten single-site jobs shared about 40 URLs/s, and one broad job's batch stalled every other job. - A dispatcher pass now runs up to 32 jobs at once. Each job is dispatched at most once per pass (and a guard refuses a second concurrent dispatch of a job), the pass ends when every job has returned, and passes never overlap. Shutdown is unchanged per job: nothing more is popped, granted URLs are sent, the rest of every batch is requeued, hard stop included. - A batch asks the politeness store for one slot per domain, 32 domains at once (the store checks the domain and job caps atomically). After a grant with a non-zero delay the domain's other URLs are parked for that delay without asking; with a zero delay they keep asking, as before. - Jobs with nothing to dispatch are not asked about on every tick: a paused job or one leased by another instance every 200 ms (a Resume ends the wait at once), a running job whose pops come back empty every 100 ms (an admission on this instance ends the wait at once, and URLs the dispatcher parked itself for up to 1 s are still popped when due). Held leases are still renewed far inside their 5 s TTL. - The tracked-domain gauge is refreshed once per second instead of on every tick (it scans the tracked domains under the politeness lock). - The progress publisher reads each job's counters and queue depth once, the jobs concurrently, and sends the changed jobs' events at once instead of one awaited send per job. Requeue order (a bounced URL gets a fresh sequence number) is left as it is: keeping the original position would let one throttled domain at the head of a job's queue starve the job's other domains. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ce-review-crawl-scrape-and-map-bottlenecks
…ce-review-crawl-scrape-and-map-bottlenecks
… batch page setup Chrome renders downloaded every image, font and media file and waited for them before `load`, because `CdpConfig::block_images`, `block_stylesheets` and `block_fonts` were never applied. - The `block_*` settings (plus a new `block_media`) are now applied through the same `Fetch` interception as the request guard. Without the guard only the blocked resource types are paused, so other requests cost no CDP round trip; with the guard every request is still paused and checked, and a request failed for its type is never sent. The decision is per render: a render with a screenshot, with actions (clicks and scrolls depend on layout) or with `PageOptions::load_all_resources` loads everything. - Crawl renders (crawler worker) and the API's `/scrape` and `/map` renders block images, fonts and media. Stylesheets still load. - The API browser's page pool is configurable (`API_BROWSER_CONCURRENCY`, default 5, at least 1). `/map` with `render_js` runs at most the pool minus a reserve for `/scrape` (a fifth, at least one page) instead of 50 tasks on 5 pages, and its 30 s render timeout starts once the render holds its page (`PageOptions::timeout_after_slot`), so pages are no longer lost to time spent queueing. A timed-out map render now also closes its page. - The per-render setup calls (init scripts, link-local block, request interception, user agent, device metrics, cookies, extra headers) are sent concurrently instead of one round trip after the other. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…language The content worker built its parser with markdown always on and dropped the result afterwards when the job did not ask for it. The conversion re-parses the HTML (html2md) and is about 70% of a page's parse time in the parser benchmark (`full_parsing_pipeline`: 17 us without markdown, 56 us with it, on the medium page). - `HtmlParser::parse_with_options` takes per-call `HtmlParseOptions` (`convert_to_markdown`, `extract_schema`) that can turn off a step the parser's config enables. `parse` is unchanged (it uses the default options). - `HtmlParser::parse_with_dom` also hands the parsed DOM and the document to a closure, so schema options, custom selectors and block splitting reuse the one DOM instead of parsing the HTML again (`scraper::Html` is not `Send`, so the DOM never leaves the call). - `strip_non_content_tags` / `strip_boilerplate_tags` ran 9 regexes, each copying the whole page even without a match. They now keep a `Cow` and copy only when a pattern matches; the passes stay sequential, so the output is the same (a merged alternation would differ on interleaved tags). - Language detection grows with the text (about 15 us plus 6 us per KB in the benchmark). The parser now detects on the first 8 KiB of the content (`detect_language_sampled`, cut on a char boundary), in `HtmlParser` and `parse_markdown_page`. `detect_language` itself is unchanged. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Batches were cut by document count only (`BATCH_SIZE`, 2000), so a batch of large documents could approach Meilisearch's payload limit or the 30 s request timeout, and a 413 rejected every document of the batch as a permanent failure. - A batch is now cut at `batch_size` documents or `max_batch_bytes` serialized bytes (default 20 MiB, `DEFAULT_MAX_BATCH_BYTES`, `MeilisearchStorageBuilder::max_batch_bytes`), whichever comes first. A single larger document is still sent, alone. The size of a document is counted once, when it is buffered. - Reaching `max_batch_bytes` buffered bytes flushes right away, and the backpressure bound is four batches' worth of documents or of bytes. - A request refused for its size (413 from Meilisearch or from a proxy in front of it) is split in two and each half retried, in buffer order; only a single document refused for its size is rejected. A retryable failure after a split keeps the unsent documents, in order. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The content worker published `DocumentIndexed` events one awaited send at a time. A send waits for its delivery report (about one producer linger, 20-25 ms), so events and Kafka acks were capped near 45 pages/s per worker and a 2000-document batch took about 45 s to be reported. Publishing: - The publisher keeps up to 256 sends in flight and acks each page when its own delivery resolves. A page whose event was not published is still not acked (redelivered); `pending_notices` and the metrics stay exact. Order across pages is not kept: each outcome carries its `url_message_id`. - Skipped and failed pages hand their event to the same publisher instead of holding a `CONCURRENCY` slot during the send (at most 1024 queued, then handlers wait). `PUBLISH_TO_KAFKA` sends a page's documents together. Parsing: - A page's features are resolved before parsing. Markdown is converted only when the page keeps it: not when `features.markdown` is off, and for a block-split page only when no block comes out. - The job's schema options, custom selectors and the block split reuse the parser's DOM (one HTML parse instead of up to four, html2md's own aside). - The parse stage runs in `spawn_blocking`, bounded to the number of cores; `CONCURRENCY` stays the bound on pages in flight. A panic in it fails the page. Flushing: - The periodic flush starts one flush per storage and does not wait for them, so a slow or unreachable Meilisearch only delays its own tenant; after a failed flush a storage is left alone for 5 s. The shutdown flush runs the storages concurrently. - The interval is `FLUSH_INTERVAL_MS` (default 1000, was a fixed 5000), read from the environment: `Args` is built field by field by `scrapix all`, outside this change. - A storage unused for 10 minutes, with nothing buffered or in flight, is dropped from the cache. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ce-review-crawl-scrape-and-map-bottlenecks # Conflicts: # bins/scrapix-api/src/lib.rs
…ce-review-crawl-scrape-and-map-bottlenecks
Adds the Unreleased entries for the crawler, frontier, content worker, renderer and API changes, documents API_BROWSER_CONCURRENCY and FLUSH_INTERVAL_MS, and updates the dispatcher description in CLAUDE.md. BATCH_TIMEOUT_SECS is dropped from the env docs: no code reads it. Refs SCR-89 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Lab added POST /internal/deploys and the DeployReport schema (216 added lines, nothing removed). The engine does not call it; the vendored copy only has to match the Lab's for the drift check. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up of #36 (SCR-88). A code review of the crawl pipeline and the sync endpoints found that most of the remaining ceiling had one cause: Kafka sends awaited one at a time (about 20 to 25 ms each with
linger_ms: 20,acks: all). This PR removes them and fixes the second-order items found on the way. Tracking issue: SCR-89.What changes
Crawler worker
PageCrawledstill follows the page and its links. Sitemap entries go 256 at a time.links_publishedandUrlsDiscovered.countare the links actually sent.page too large to publish) instead of an unacked message that ends the jobFailStalled.Frontier
ACQUIRE_LUAnow returns the effective delay with a grant.Content worker, parser, Meilisearch storage
FLUSH_INTERVAL_MS(default 1000, was a fixed 5 s), per storage. Requests are cut atBATCH_SIZEdocuments or 20 MiB. A 413 splits the batch. A storage idle for 10 minutes is dropped.Browser renderer
block_images,block_fonts,block_stylesheetswere never applied. They are now, with a newblock_media. The crawler worker and the API block images, fonts and media, except for renders with a screenshot or actions.API_BROWSER_CONCURRENCY(default 5)./mapwithrender_jsuses at most the pool minus a reserve for/scrape, and its 30 s timeout starts when the render gets its page.API
/scrape,/batch/scrape,/extract: one retry instead of three.timeout_msbounds the whole fetch for requests withheadersorcookies./scraperuns its HTML stage once, on blocking threads. Requests withheadersand no cookies use the shared fetcher./searchuses one pooled client with timeouts./batch/scrapeand/extractrun each URL on its own task. Responses of 1 KB and more are gzipped on request.exclude_selectorsparses once and removes every match./mapwithdepthfetches the start page while reading sitemaps.Behaviour changes
links_publishedandUrlsDiscovered.countare lower on sites with repeated navigation./scrapewithheadersorcookies: a slow fetch is a 504timeout, where it was a 502fetch_error./scrapewithheadersand no cookies no longer keeps a cookie store across a redirect chain./metricsis gzipped when the scraper asks for it./mapwithrender_jsruns 4 renders at once by default.languagecomes from the first 8 KiB of content.Verification
cargo fmt --check,cargo check,cargo clippy --all --all-targets -- -D warnings: pass.cargo test --workspace: 1,523 passed, 0 failed, 8 ignored.cdp_browsertests passed against a real Chrome in the renderer's worktree.Not verified:
SCRAPIX_TEST_REDIS_URL=... cargo test -p scrapix-frontier -p scrapix-frontier-service --all-featurespasses against a real Redis 7:politeness_redis5 passed (including the new parity test),store_redis1 passed. Production runs Redis; DragonflyDB was not tried.SCRAPIX_TEST_CHROME) did not run on the combined branch.Not in this PR: the
PAGES_RAWpartition key, the per-site dispatch ceiling, the Markdown converter, slimmer link messages.Deploy notes
API_BROWSER_CONCURRENCY(API, default5),FLUSH_INTERVAL_MS(content worker, default1000).BATCH_TIMEOUT_SECSis removed from the env docs: no code reads it. It is still set in.env.example,docker-compose.ymland the Kubernetes configs.🤖 Generated with Claude Code