Skip to content

perf: remove the serial Kafka sends and other crawl, scrape and map bottlenecks (SCR-89) - #38

Merged
qdequele merged 17 commits into
mainfrom
scr-89-performance-review-crawl-scrape-and-map-bottlenecks
Nov 12, 2026
Merged

qdequele merged 17 commits into
mainfrom
scr-89-performance-review-crawl-scrape-and-map-bottlenecks

Conversation

@qdequele

@qdequele qdequele commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Follow-up of #36 (SCR-88). A code review of the crawl pipeline and the sync endpoints found that most of the remaining ceiling had one cause: Kafka sends awaited one at a time (about 20 to 25 ms each with linger_ms: 20, acks: all). This PR removes them and fixes the second-order items found on the way. Tracking issue: SCR-89.

What changes

Crawler worker

  • A page, its links (256 at a time) and its fetch feedback are sent together. A page with 100 links used to spend about 2 s publishing while holding a worker slot. PageCrawled still follows the page and its links. Sitemap entries go 256 at a time.
  • A worker publishes a link once per job (bounded per-job set, entries expire after 10 to 20 minutes). links_published and UrlsDiscovered.count are the links actually sent.
  • A page whose encoded message is over the 10 MiB limit is a failed page (page too large to publish) instead of an unacked message that ends the job FailStalled.
  • Link extraction of pages over 32 KiB runs on blocking threads. URL glob matching no longer allocates per call.

Frontier

  • A dispatch pass runs up to 32 jobs concurrently (never the same job twice, passes never overlap). Ten single-site jobs used to share about 40 URLs/s.
  • One politeness call per domain per round, up to 32 domains at once. ACQUIRE_LUA now returns the effective delay with a grant.
  • Paused jobs and jobs leased by another instance are polled every 200 ms, idle running jobs every 100 ms.

Content worker, parser, Meilisearch storage

  • Page outcomes are published with up to 256 sends in flight, each page acked when its own event is delivered (was one at a time, about 45 pages/s per worker).
  • A page is parsed once, on blocking threads. Markdown is converted only when the job keeps it. Language detection reads the first 8 KiB.
  • Flush every FLUSH_INTERVAL_MS (default 1000, was a fixed 5 s), per storage. Requests are cut at BATCH_SIZE documents or 20 MiB. A 413 splits the batch. A storage idle for 10 minutes is dropped.

Browser renderer

  • block_images, block_fonts, block_stylesheets were never applied. They are now, with a new block_media. The crawler worker and the API block images, fonts and media, except for renders with a screenshot or actions.
  • API_BROWSER_CONCURRENCY (default 5). /map with render_js uses at most the pool minus a reserve for /scrape, and its 30 s timeout starts when the render gets its page.
  • Page setup CDP calls are issued together.

API

  • /scrape, /batch/scrape, /extract: one retry instead of three. timeout_ms bounds the whole fetch for requests with headers or cookies.
  • /scrape runs its HTML stage once, on blocking threads. Requests with headers and no cookies use the shared fetcher.
  • /search uses one pooled client with timeouts. /batch/scrape and /extract run each URL on its own task. Responses of 1 KB and more are gzipped on request. exclude_selectors parses once and removes every match. /map with depth fetches the start page while reading sitemaps.

Behaviour changes

  • Meilisearch receives more, smaller tasks on slow crawls (1 s flush).
  • links_published and UrlsDiscovered.count are lower on sites with repeated navigation.
  • /scrape with headers or cookies: a slow fetch is a 504 timeout, where it was a 502 fetch_error.
  • /scrape with headers and no cookies no longer keeps a cookie store across a redirect chain.
  • /metrics is gzipped when the scraper asks for it.
  • /map with render_js runs 4 renders at once by default.
  • language comes from the first 8 KiB of content.
  • Outcome events of different pages can arrive out of order.

Verification

  • cargo fmt --check, cargo check, cargo clippy --all --all-targets -- -D warnings: pass.
  • cargo test --workspace: 1,523 passed, 0 failed, 8 ignored.
  • Renderer: 16 cdp_browser tests passed against a real Chrome in the renderer's worktree.

Not verified:

  • Redis politeness script. SCRAPIX_TEST_REDIS_URL=... cargo test -p scrapix-frontier -p scrapix-frontier-service --all-features passes against a real Redis 7: politeness_redis 5 passed (including the new parity test), store_redis 1 passed. Production runs Redis; DragonflyDB was not tried.
  • Throughput. Nothing was timed against a real Redpanda or Meilisearch. The gains are argued from the code.
  • The API browser tests (SCRAPIX_TEST_CHROME) did not run on the combined branch.
  • Resource blocking was not checked on live sites against the bot-challenge and "almost no text" fallback heuristics.

Not in this PR: the PAGES_RAW partition key, the per-site dispatch ceiling, the Markdown converter, slimmer link messages.

Deploy notes

  • Env: two new optional variables with defaults, nothing to set on the box first: API_BROWSER_CONCURRENCY (API, default 5), FLUSH_INTERVAL_MS (content worker, default 1000).
  • Migrations: none. Contracts: unchanged.
  • Rollout: upgrade with no job running, as usual. During a rolling deploy, old and new frontier instances run different acquire scripts on the same Redis keys; the key layout and semantics are the same.
  • BATCH_TIMEOUT_SECS is removed from the env docs: no code reads it. It is still set in .env.example, docker-compose.yml and the Kubernetes configs.

🤖 Generated with Claude Code

Quentin de Quelen and others added 17 commits November 12, 2026 12:07
matches_glob built a token list from the pattern and one table row per
token on every call, and it is called per link and per pattern. The
literal part before the first `*` is now a plain prefix comparison (most
patterns are `https://host/path/*`, most non-matching URLs differ there),
tokens are read from the pattern as it is walked, and the two table rows
are kept per thread. Matching is unchanged: a test compares it with the
previous implementation on a grid of patterns and URLs.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… and fail oversized pages

A send waits for its delivery report (the producer's 20 ms linger plus the
broker ack), and the handler awaited every send in series while holding a
worker slot: a page with 100 links took about 2 s to publish.

- A page's link messages are sent together (at most 256 in flight) and
  awaited together; any failed send still leaves the message unacked. The
  raw page, the links, the PageRank input and the page history are
  published together; UrlsDiscovered and PageCrawled together after them,
  so PageCrawled still follows the page and its links.
- FetchFeedback is sent while the page is published instead of before it.
  It is still queued first and awaited (2 s bound) before the ack.
- Sitemap entries are sent 256 at a time instead of one after the other.
- A worker publishes a link once per job: links it already sent and
  counted are skipped, and links_published / UrlsDiscovered count the
  links actually sent. A link is recorded only after the accounting event
  counting it was sent, so a redelivered page sends and counts its links
  again. The set is bounded (64-bit hashes, two generations of 50,000 per
  job, 64 jobs) and expires after 10 to 20 minutes, so a link the frontier
  refused on a full queue is sent again later.
- A page whose serialized message is over the producer's message size
  limit (a 7.5 to 10 MB document once base64-encoded, or HTML grown by
  JSON escaping) is reported PageFailed and acked. It used to fail every
  send, stay unacked and end the job FailStalled. A broker refusal of a
  large page is terminal too.
- Link extraction of a page over 32 KiB runs on the blocking pool.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The router had no compression layer although tower-http's
compression-gzip feature is enabled: scrape, results and analytics
payloads (markdown, HTML, JSON) went out uncompressed.

Add a CompressionLayer around every route, negotiated by
Accept-Encoding, for bodies of at least 1 KB. Event streams
(text/event-stream), WebSocket upgrades, images and gRPC are never
compressed, so /job/{id}/events still reaches the client event by
event. /health stays plain (too small); /metrics is gzipped only for a
scraper that asks for it.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The scrapes of /batch/scrape and /extract were futures polled by the
job's own task (buffer_unordered): while that task awaited the store
write of a finished result, the fetches in flight were not polled.

Each URL now runs on its own task (ScrapeTasks: a JoinSet capped at the
job's concurrency), so the scrapes progress while results are stored.
The semantics are kept: a URL's task starts when a slot frees up and
checks the job's gate first, so cancel stops the URLs not started yet,
pause holds them, and the scrapes in flight finish; a batch still skips
the remaining URLs once the account is out of credits; the extract
job's AI usage tally is carried into the tasks. A scrape that panics
now fails its URL (internal_error) instead of hanging the job.

The result item is read from the stored payload in place, without the
deep clone of the whole result.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
/scrape fetch:
- One retry instead of the crawler's three: a page answering 429/5xx
  cost four fetches and 3.5 s of backoff before the error came back.
- timeout_ms bounds the whole fetch for requests with headers or
  cookies too (it was only each attempt's timeout there).
- Requests with headers (and no cookie) use the shared fetcher
  (connection pool, DNS cache) with the headers passed per request,
  instead of building a robots client and an HTTP client each time.
  Cookies, and a Cookie header, keep a one-off fetcher (its cookie
  store must live and die with the request), which now shares a DNS
  cache and a robots cache built once.

/scrape HTML stage:
- Runs in spawn_blocking instead of on the async runtime.
- The page is parsed into a DOM once (twice when include/exclude
  selectors change it) instead of once per format, through the
  extractors' *_from_dom functions; the markdown is converted once
  when markdown and content both need it; no copy of the HTML when
  there is no selector.
- exclude_selectors parse the page once instead of once per selector.
  Matched elements are still cut out of the source text, so the rest
  of the page is unchanged; an element whose serialization is not its
  source text (unquoted attribute, unclosed tag), which used to be
  kept silently, is now removed by detaching it from the tree.
  A test compares the stage with the previous per-format pipeline over
  every combination of formats and options.

/search: one pooled client (5 s connect, 30 s request timeout) instead
of a new client without a timeout per request; a timeout is the
timeout error (504).

/map: with depth > 0 the start page is fetched while the sitemaps are
read, and pages of 128 KiB and more are parsed for links off the
runtime.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…stores

A politeness slot is granted to a domain at most once per effective delay,
whatever CONCURRENT_PER_DOMAIN is (it only caps the in-flight count). The
dispatcher did not know that delay, so after a grant it asked the store
again for the domain's next URL only to be told to wait: half of all
acquire calls in a single-site crawl.

PolitenessStore::try_acquire_spaced returns, with a grant, the effective
delay the grant was checked against (zero when the domain can be granted
again right away). The in-memory scheduler and the Redis script (which now
returns {status, ms}) report the same value; try_acquire is unchanged.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… every tick

The dispatcher handled the active jobs one after another, and each job
awaits its batch's bus sends (about 25 ms with the producer's linger and
acks=all): ten single-site jobs shared about 40 URLs/s, and one broad
job's batch stalled every other job.

- A dispatcher pass now runs up to 32 jobs at once. Each job is dispatched
  at most once per pass (and a guard refuses a second concurrent dispatch
  of a job), the pass ends when every job has returned, and passes never
  overlap. Shutdown is unchanged per job: nothing more is popped, granted
  URLs are sent, the rest of every batch is requeued, hard stop included.
- A batch asks the politeness store for one slot per domain, 32 domains at
  once (the store checks the domain and job caps atomically). After a
  grant with a non-zero delay the domain's other URLs are parked for that
  delay without asking; with a zero delay they keep asking, as before.
- Jobs with nothing to dispatch are not asked about on every tick: a
  paused job or one leased by another instance every 200 ms (a Resume
  ends the wait at once), a running job whose pops come back empty every
  100 ms (an admission on this instance ends the wait at once, and URLs
  the dispatcher parked itself for up to 1 s are still popped when due).
  Held leases are still renewed far inside their 5 s TTL.
- The tracked-domain gauge is refreshed once per second instead of on
  every tick (it scans the tracked domains under the politeness lock).
- The progress publisher reads each job's counters and queue depth once,
  the jobs concurrently, and sends the changed jobs' events at once
  instead of one awaited send per job.

Requeue order (a bounced URL gets a fresh sequence number) is left as it
is: keeping the original position would let one throttled domain at the
head of a job's queue starve the job's other domains.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ce-review-crawl-scrape-and-map-bottlenecks
…ce-review-crawl-scrape-and-map-bottlenecks
… batch page setup

Chrome renders downloaded every image, font and media file and waited
for them before `load`, because `CdpConfig::block_images`,
`block_stylesheets` and `block_fonts` were never applied.

- The `block_*` settings (plus a new `block_media`) are now applied
  through the same `Fetch` interception as the request guard. Without
  the guard only the blocked resource types are paused, so other
  requests cost no CDP round trip; with the guard every request is still
  paused and checked, and a request failed for its type is never sent.
  The decision is per render: a render with a screenshot, with actions
  (clicks and scrolls depend on layout) or with
  `PageOptions::load_all_resources` loads everything.
- Crawl renders (crawler worker) and the API's `/scrape` and `/map`
  renders block images, fonts and media. Stylesheets still load.
- The API browser's page pool is configurable (`API_BROWSER_CONCURRENCY`,
  default 5, at least 1). `/map` with `render_js` runs at most the pool
  minus a reserve for `/scrape` (a fifth, at least one page) instead of
  50 tasks on 5 pages, and its 30 s render timeout starts once the
  render holds its page (`PageOptions::timeout_after_slot`), so pages
  are no longer lost to time spent queueing. A timed-out map render now
  also closes its page.
- The per-render setup calls (init scripts, link-local block, request
  interception, user agent, device metrics, cookies, extra headers) are
  sent concurrently instead of one round trip after the other.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…language

The content worker built its parser with markdown always on and dropped
the result afterwards when the job did not ask for it. The conversion
re-parses the HTML (html2md) and is about 70% of a page's parse time in
the parser benchmark (`full_parsing_pipeline`: 17 us without markdown,
56 us with it, on the medium page).

- `HtmlParser::parse_with_options` takes per-call `HtmlParseOptions`
  (`convert_to_markdown`, `extract_schema`) that can turn off a step the
  parser's config enables. `parse` is unchanged (it uses the default
  options).
- `HtmlParser::parse_with_dom` also hands the parsed DOM and the document
  to a closure, so schema options, custom selectors and block splitting
  reuse the one DOM instead of parsing the HTML again (`scraper::Html` is
  not `Send`, so the DOM never leaves the call).
- `strip_non_content_tags` / `strip_boilerplate_tags` ran 9 regexes, each
  copying the whole page even without a match. They now keep a `Cow` and
  copy only when a pattern matches; the passes stay sequential, so the
  output is the same (a merged alternation would differ on interleaved
  tags).
- Language detection grows with the text (about 15 us plus 6 us per KB in
  the benchmark). The parser now detects on the first 8 KiB of the content
  (`detect_language_sampled`, cut on a char boundary), in `HtmlParser` and
  `parse_markdown_page`. `detect_language` itself is unchanged.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Batches were cut by document count only (`BATCH_SIZE`, 2000), so a batch
of large documents could approach Meilisearch's payload limit or the 30 s
request timeout, and a 413 rejected every document of the batch as a
permanent failure.

- A batch is now cut at `batch_size` documents or `max_batch_bytes`
  serialized bytes (default 20 MiB, `DEFAULT_MAX_BATCH_BYTES`,
  `MeilisearchStorageBuilder::max_batch_bytes`), whichever comes first. A
  single larger document is still sent, alone. The size of a document is
  counted once, when it is buffered.
- Reaching `max_batch_bytes` buffered bytes flushes right away, and the
  backpressure bound is four batches' worth of documents or of bytes.
- A request refused for its size (413 from Meilisearch or from a proxy in
  front of it) is split in two and each half retried, in buffer order; only
  a single document refused for its size is rejected. A retryable failure
  after a split keeps the unsent documents, in order.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The content worker published `DocumentIndexed` events one awaited send at
a time. A send waits for its delivery report (about one producer linger,
20-25 ms), so events and Kafka acks were capped near 45 pages/s per worker
and a 2000-document batch took about 45 s to be reported.

Publishing:
- The publisher keeps up to 256 sends in flight and acks each page when its
  own delivery resolves. A page whose event was not published is still not
  acked (redelivered); `pending_notices` and the metrics stay exact. Order
  across pages is not kept: each outcome carries its `url_message_id`.
- Skipped and failed pages hand their event to the same publisher instead
  of holding a `CONCURRENCY` slot during the send (at most 1024 queued,
  then handlers wait). `PUBLISH_TO_KAFKA` sends a page's documents
  together.

Parsing:
- A page's features are resolved before parsing. Markdown is converted
  only when the page keeps it: not when `features.markdown` is off, and
  for a block-split page only when no block comes out.
- The job's schema options, custom selectors and the block split reuse
  the parser's DOM (one HTML parse instead of up to four, html2md's own
  aside).
- The parse stage runs in `spawn_blocking`, bounded to the number of
  cores; `CONCURRENCY` stays the bound on pages in flight. A panic in it
  fails the page.

Flushing:
- The periodic flush starts one flush per storage and does not wait for
  them, so a slow or unreachable Meilisearch only delays its own tenant;
  after a failed flush a storage is left alone for 5 s. The shutdown flush
  runs the storages concurrently.
- The interval is `FLUSH_INTERVAL_MS` (default 1000, was a fixed 5000),
  read from the environment: `Args` is built field by field by
  `scrapix all`, outside this change.
- A storage unused for 10 minutes, with nothing buffered or in flight, is
  dropped from the cache.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ce-review-crawl-scrape-and-map-bottlenecks

# Conflicts:
#	bins/scrapix-api/src/lib.rs
…ce-review-crawl-scrape-and-map-bottlenecks
Adds the Unreleased entries for the crawler, frontier, content worker,
renderer and API changes, documents API_BROWSER_CONCURRENCY and
FLUSH_INTERVAL_MS, and updates the dispatcher description in CLAUDE.md.
BATCH_TIMEOUT_SECS is dropped from the env docs: no code reads it.

Refs SCR-89

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Lab added POST /internal/deploys and the DeployReport schema (216
added lines, nothing removed). The engine does not call it; the vendored
copy only has to match the Lab's for the drift check.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@qdequele
qdequele added this pull request to stack #40 November 12, 2026 15:10
@qdequele
qdequele merged commit 2c29e0c into main Nov 12, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant