Skip to content

feat(api): suggest a crawl config from a website URL (SCR-95) - #43

Merged
qdequele merged 3 commits into
mainfrom
scr-95-suggest-a-crawl-config-from-a-website-url-post
Nov 16, 2026
Merged

qdequele merged 3 commits into
mainfrom
scr-95-suggest-a-crawl-config-from-a-website-url-post

Conversation

@qdequele

Copy link
Copy Markdown
Owner

Resolves SCR-95.

What

POST /crawl/suggest + GET /crawl/suggest/{id}: give a website URL, get a ready-to-run crawl config with a reason for every choice.

The job scrapes the start page (following redirects), maps the site, samples 8 pages (configurable, 1–20) and:

  • groups pages into sections by first path segment (language prefixes handled; a lone two-letter code like /uk/ only counts as a language next to others);
  • classifies each section (product, listing, article, docs, support, marketing; archive, legal and account pages are left out, including older versions of versioned docs) and the site (ecommerce, blog, docs, marketing, support, mixed), from Product markup, the platform of the section's own pages (Shopify, WooCommerce, Docusaurus, Mintlify, Zendesk, Ghost, …), article/docs/Q&A markup, dated URLs and section names;
  • renders up to 2 pages that look like JavaScript app shells to decide on crawler_type: browser (when the server has a browser and the plan includes JS rendering; otherwise a warning);
  • asks the AI provider (hosted: the team's Lumen gateway) to confirm or correct the classification, when there is one; without it the deterministic result stands;
  • proposes a compact CrawlConfig: each feature (schema.org, block split, AI summary, AI extraction, PDF, office documents) scoped to its sections with include_pages, URL excludes, a page cap only when the page count is known, plus reasons, an estimate and warnings.

Billed as one or two /maps and one /scrape per sampled page (/map is split into its handler and run_map, which the job reuses). New job type crawl_suggest.

Also in this PR:

  • fix(crawler): in sitemaps whose <url> carries an <image:image> / <video:video> (every Shopify store), the image's <image:loc> replaced the page's <loc>, so /map and crawls got CDN image URLs instead of product pages. Only the direct children of <url> / <sitemap> are read now.
  • SDKs: suggestConfig / getSuggestion / suggestConfigAndWait (TypeScript), suggest_config / get_suggestion / suggest_config_and_wait (Python, sync and async).
  • Contracts (openapi.json, openapi.engine.json), docs (docs/api-reference/crawl-suggest.mdx, crawling guide, SDK pages), CHANGELOG.

Testing

  • 37 unit/end-to-end tests in bins/scrapix-api/src/suggest/ (mock Shopify-style shop with a mock LLM, the no-AI path, a JS app shell), sitemap parser regression tests, SDK tests (TS 37, Python 53).
  • Run against 13 live sites with the deterministic classification (Allbirds, Gymshark, IKEA, Meilisearch home/docs/blog, Docusaurus, Python docs, Ghost blog, Webflow, Zendesk help center, BBC News, Excalidraw); the misclassifications found there are fixed and covered by tests.
  • Not verified live: the AI pass (no provider locally) and the browser check (no Chrome locally). Locally the scrapix-api binary does not link on macOS (same on main), so the integration tests (openapi_snapshot, lab_boundary) were not run here; the engine snapshot was regenerated from the same ScrapixApi::openapi() code.

Deploy notes

  • No new env var or secret, no migration.
  • /crawl/* already routes to the engine (not in Caddy's @saas / SAAS_PREFIXES).
  • Contract change: after this merges and is released, the Lab re-vendors contracts/openapi.json; the Lab console PR meilisearch/lab#142 merges after that.

🤖 Generated with Claude Code

qdequele and others added 3 commits November 16, 2026 10:27
In a sitemap whose <url> entries carry an <image:image> or <video:video>
(every Shopify store, among others), the image's <image:loc> replaced
the page's <loc>: /map and crawls got the images' CDN URLs instead of
the product pages. Only the direct children of <url> (and of <sitemap>
in an index) are read now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (SCR-95)

TypeScript: suggestConfig, getSuggestion, suggestConfigAndWait.
Python (sync and async): suggest_config, get_suggestion,
suggest_config_and_wait, returning a SuggestJobResult whose config goes
back to crawl() as is. Generated models regenerated from the spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /crawl/suggest starts a job that scrapes the start page, maps the
site, samples a few pages and proposes a crawl config, polled with
GET /crawl/suggest/{id}. Sections are classified (product, listing,
article, docs, support, marketing; archive, legal and account pages
left out) from schema.org types, the platform of their pages, dated
URLs and section names, then the site (ecommerce, blog, docs,
marketing, support, mixed). Pages that look like JavaScript app shells
are rendered to decide on browser crawling. The AI provider (hosted:
the team's gateway) confirms or corrects the classification when there
is one. Each feature is scoped to its sections with include_pages, and
every choice comes with a reason, plus an estimate and warnings.

/map is split into its handler and run_map, which the job reuses (same
billing). New job type crawl_suggest. Contracts, docs and CHANGELOG
updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qdequele
qdequele merged commit 5273d5f into main Nov 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant