Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,71 @@ SDKs in `sdks/` are versioned and tagged on their own (`sdk-*-vX.Y.Z`).

### Changed

- **A site paced by the adaptive delay is crawled at the rate that delay
allows, whatever the dispatch period.** The frontier granted a domain one
URL per dispatcher pass, and only once a full delay had passed since the
last one, so the time between a request becoming due and the pass that
sent it was lost: a 25 ms delay with a pass every 33 ms gave 30 pages/s
instead of 40. The politeness stores now keep a schedule per domain and
give that time back: a late pass sends the URLs it is owed in one call.
The credit is 100 ms (`DOMAIN_SPACING_CREDIT_MS`, `0` restores the old
spacing), so such a site gets at most `1 + 100 / delay` requests at once
(5 at the 25 ms floor), never more than its free in-flight slots, and
never more than `1 / delay` on average. Explicit delays are unchanged: a
job's `per_domain_delay_ms` or `requests_per_second`, a robots
`Crawl-delay`, `default_crawl_delay_ms` and any delay of 1 s or more stay
a strict floor on the gap between two requests. A `Retry-After` pause
banks no credit, and a 429 or an error drops it (SCR-90). Measured end to
end on a local stack: a single site goes from 26 to 37 pages/s (24 with
the credit at `0`), against a ceiling of 40 at the 25 ms floor.

- **Kafka producers no longer wait 20 ms before sending.** Every publish
stage awaits its delivery report, so the old `linger.ms=20` added about
20 ms to each one (about 40 ms per page in the crawler, one linger per
frontier dispatch pass). The default is now `0`: on a local Redpanda an
awaited send takes 0.12 ms instead of 22 ms, and sustained throughput is
unchanged because batches fill by size. End to end, a crawl of 20 sites
in one job runs at about 84 pages/s with `0` and 65 with `20`; a single
site is paced by politeness and does not change. The new `KAFKA_LINGER_MS`
(integer, 0 to 1000) sets it for every producer; an invalid value refuses
to start.

- **The binaries can be built with mimalloc.** A new cargo feature,
`mimalloc`, on each binary crate, off by default (`--features mimalloc`).
On the parser benchmarks in a Linux container it is 3 to 4% faster on HTML
parsing and 8 to 10% faster on Markdown conversion and the full parse
pipeline. It is not the default because a full crawl is not faster with
it (84 pages/s with and without, 20 sites in one job, macOS arm64) and
the content worker's peak memory doubles (about 230 MB against 105 MB).

- **Markdown conversion is up to a third faster, with the same output.**
The tags dropped from a page are found in one scan instead of one regex
pass per tag, the Markdown cleanup skips the patterns that cannot match,
and links on ordinary lines are resolved without the regex engine. On the
parser benches: 29 to 37% less time for the crawl's conversion, 12 to 20%
for `html_to_markdown`. The Markdown is byte-identical: the previous
implementation is kept as a test reference and compared on 1,750 pages.

- **robots.txt costs one connection and one request per new origin.** The
crawler worker fetches robots.txt through the page fetcher's DNS cache
and connection pool, and the first URLs of an origin share a single
robots.txt fetch instead of each fetching it. The body is read up to
500 KiB (the rest is ignored, cut at a line end), and the cache drops
expired entries and keeps at most 100,000 origins. What robots.txt
allows, its TTL, `Crawl-delay` and the SSRF checks are unchanged. The
request now also carries the fetcher's `Accept-Language` header.

- **The engine asks the Lab once per credential, not once per request.**
When a cached Lab answer expired (every 30 s for a busy API key), every
request in flight for that key called the Lab at the same time.
Concurrent requests for the same key now share one Lab call, and a key in
use is refreshed in the background from 80% of its `cache_ttl`. The
guarantees are unchanged: a revoked key works for at most `cache_ttl`,
answers are served stale for at most 5 minutes while the Lab is down, and
`cache_ttl` 0 answers are fetched by every request. `/scrape`, `/map` and
`/search` also record their usage event in one database round trip
instead of three.

- **The crawler publishes a page's messages together.** Each message
waited about 20 ms for its delivery report while the page held a worker
slot, so a page with 100 links took about 2 s to publish. A page, its
Expand Down Expand Up @@ -192,6 +257,13 @@ SDKs in `sdks/` are versioned and tagged on their own (`sdk-*-vX.Y.Z`).

### Fixed

- **Page text that looks like deeply nested Markdown links no longer
crashes the process.** Resolving links recursed once per link nested in a
link text, so a 160 KB page repeating `[![a](b)` overflowed the stack and
aborted the content worker (or the API on `/scrape`), which then got the
page again. The nesting is followed 32 levels deep; past that the text of
a link is kept as it is.

- **A page too large to publish fails instead of stalling its job.** A
document between about 7.5 and 10 MB is over the 10 MiB message limit
once base64-encoded, and so is HTML that grows past it when escaped. Its
Expand Down
30 changes: 22 additions & 8 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,9 @@ that names a Lab-owned table, reads included).
(`{LAB_URL}/internal/*`, `X-Lab-Instance-Id: LAB_INSTANCE_ID` + Bearer
`LAB_INSTANCE_SECRET`): credentials are sent to
`POST /internal/auth/introspect` (cached for the Lab's `cache_ttl`, default
30 s, stale up to 5 min if the Lab is down, except answers with
30 s, one Lab call per key however many requests wait for it, and a key in
use is refreshed in the background from 80% of its `cache_ttl`, never
served past it because of a refresh; stale up to 5 min if the Lab is down, except answers with
`cache_ttl` 0 (expired or expiring), which are never served stale; an unknown credential while the
Lab is down is a 503 with `Retry-After: 5`; so a revoked key keeps working
on the engine for up to `cache_ttl`, documented in
Expand Down Expand Up @@ -548,20 +550,30 @@ toward the crawler-reported `FetchFeedback.fetch_ms / CONCURRENT_PER_DOMAIN`
(dropping at most ×0.9 per success), clamped to
`[min(MIN_DOMAIN_DELAY_MS, DOMAIN_DELAY_MS), 30s]`; 429s and error streaks
raise it ×1.5 (a job can only raise the delay, never lower it). A domain
gets at most one URL per dispatch batch, so `DISPATCH_INTERVAL_MS` (20 ms)
also caps one site at about 40 pages/s. The frontier holds a per-domain
paced by its adaptive delay gets `1 / delay` whatever the dispatch period:
the stores keep a virtual last-request time per domain (a grant needs
`now >= last + delay` and sets `last = max(last + delay, now - credit, pause
end)`), so a pass that comes late grants the URLs it is owed in one call. The
credit is `DOMAIN_SPACING_CREDIT_MS` (default `100`, `0` = strict spacing
everywhere): at most `1 + credit / delay` requests at once (5 at the 25 ms
floor), within the free in-flight slots. Only the adaptive delay is
credited. When a job's `per_domain_delay_ms` or `requests_per_second`, a
robots `Crawl-delay` or `default_crawl_delay_ms` sets the delay, and for any
delay of 1 s or more, spacing is strict: two requests are never closer than
the delay. An explicit delay below the adaptive one is still a floor on the
gap. A pause banks no credit, and a 429 or an error drops it. The frontier holds a per-domain
slot from dispatch until the crawler's `FetchFeedback` — topic
`scrapix.fetch.feedback` — reports the fetch back (not at dispatch time); a
slot without feedback expires after `2 × REQUEST_TIMEOUT` (default `30`s), so
a lagging `scrapix.urls.processing` consumer effectively widens that safety
window. The effective per-domain delay is
`max(adaptive delay, job.rate_limit.per_domain_delay_ms (default 0), robots
Crawl-delay if respected, 1000 / requests_per_second)`. The dispatcher runs
up to 32 jobs concurrently per pass (never the same job twice), asks for one
politeness slot per domain per round (up to 32 domains at once; after a grant
the domain's other URLs in the batch are parked without asking) and sends a
batch's granted URLs to the bus concurrently (one serial send waits ~20 ms of
producer linger plus the broker ack, about 45 sends/s). Paused jobs and jobs
up to 32 jobs concurrently per pass (never the same job twice), asks the
politeness store once per domain per pass (one atomic call grants up to one
slot per queued URL of that domain, up to 32 domains at once) and sends a
batch's granted URLs to the bus concurrently (a send waits for its delivery
report: `KAFKA_LINGER_MS`, default `0`, plus the broker ack). Paused jobs and jobs
leased by another instance are polled every 200 ms, idle running jobs every
100 ms. `rate_limit.default_crawl_delay_ms`
(config default `0`) is a fallback `Crawl-delay` used only when robots.txt was
Expand Down Expand Up @@ -750,6 +762,8 @@ GROUP BY date ORDER BY date;
| `OCR_BACKEND` | API + content workers: `auto` (vision if an AI provider is set, else tesseract), `vision`, `tesseract`, `off` |
| `PDFIUM_LIB_PATH` | API + content workers: PDFium library used to rasterize pages for OCR (`/opt/pdfium/lib` in the images) |
| `API_BROWSER_CONCURRENCY` | API: pages its browser renders at once (default `5`, at least 1); `/map` with `render_js` uses at most the pool minus a fifth, kept for `/scrape` |
| `KAFKA_LINGER_MS` | Every service with a Kafka producer: producer `linger.ms`, `0` to `1000` (default `0`: send immediately). A send waits for its delivery report, so each publish stage costs at least one linger. An invalid value refuses to start |
| `DOMAIN_SPACING_CREDIT_MS` | Frontier: how much of the time lost between a request becoming due and the dispatch pass is given back to a domain paced by its adaptive delay (default `100`; `0` = strict spacing). Never applies to a delay set by the job, by robots.txt, or of 1 s or more |
| `FLUSH_INTERVAL_MS` | Content worker: milliseconds between periodic Meilisearch flushes (default `1000`, minimum `50`); a request is also cut at `BATCH_SIZE` documents or 20 MiB |
| `OCR_MAX_PAGES_PER_DOCUMENT` / `OCR_DAILY_PAGE_BUDGET` | OCR cost controls (defaults `50` / `1000` per account per UTC day, `0` = unlimited) |

Expand Down
25 changes: 25 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,9 @@ rdkafka = { version = "0.38", features = ["cmake-build", "ssl"] }
# Object storage (S3-compatible: RustFS, MinIO, AWS S3)
# Using rust-s3 crate directly in scrapix-storage

# Global allocator of the binaries (feature `mimalloc`, off by default)
mimalloc = "0.1"

# Bloom filter
bloomfilter = "1.0"
probabilistic-collections = "0.7"
Expand Down
9 changes: 9 additions & 0 deletions bins/scrapix-api/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,16 @@ path = "src/lib.rs"
name = "scrapix-api"
path = "src/main.rs"

[features]
default = []
# `mimalloc`: use mimalloc as the binary's global allocator (set in
# `src/main.rs`). Off by default: build with `--features mimalloc`. It is
# faster on parsing (8 to 10% on Linux) but doubles the content worker's
# peak memory, with no gain in crawl throughput end to end.
mimalloc = ["dep:mimalloc"]

[dependencies]
mimalloc = { workspace = true, optional = true }
scrapix-core = { path = "../../crates/scrapix-core" }
scrapix-auth = { path = "../../crates/scrapix-auth" }
scrapix-queue = { path = "../../crates/scrapix-queue" }
Expand Down
Loading
Loading