Repository navigation
feat(api): suggest a crawl config from a website URL (SCR-95) - #43
Merged
qdequele merged 3 commits intoNov 16, 2026
Merged
Conversation
In a sitemap whose <url> entries carry an <image:image> or <video:video> (every Shopify store, among others), the image's <image:loc> replaced the page's <loc>: /map and crawls got the images' CDN URLs instead of the product pages. Only the direct children of <url> (and of <sitemap> in an index) are read now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (SCR-95) TypeScript: suggestConfig, getSuggestion, suggestConfigAndWait. Python (sync and async): suggest_config, get_suggestion, suggest_config_and_wait, returning a SuggestJobResult whose config goes back to crawl() as is. Generated models regenerated from the spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /crawl/suggest starts a job that scrapes the start page, maps the
site, samples a few pages and proposes a crawl config, polled with
GET /crawl/suggest/{id}. Sections are classified (product, listing,
article, docs, support, marketing; archive, legal and account pages
left out) from schema.org types, the platform of their pages, dated
URLs and section names, then the site (ecommerce, blog, docs,
marketing, support, mixed). Pages that look like JavaScript app shells
are rendered to decide on browser crawling. The AI provider (hosted:
the team's gateway) confirms or corrects the classification when there
is one. Each feature is scoped to its sections with include_pages, and
every choice comes with a reason, plus an estimate and warnings.
/map is split into its handler and run_map, which the job reuses (same
billing). New job type crawl_suggest. Contracts, docs and CHANGELOG
updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves SCR-95.
What
POST /crawl/suggest+GET /crawl/suggest/{id}: give a website URL, get a ready-to-run crawl config with a reason for every choice.The job scrapes the start page (following redirects), maps the site, samples 8 pages (configurable, 1–20) and:
/uk/only counts as a language next to others);Productmarkup, the platform of the section's own pages (Shopify, WooCommerce, Docusaurus, Mintlify, Zendesk, Ghost, …), article/docs/Q&A markup, dated URLs and section names;crawler_type: browser(when the server has a browser and the plan includes JS rendering; otherwise a warning);CrawlConfig: each feature (schema.org, block split, AI summary, AI extraction, PDF, office documents) scoped to its sections withinclude_pages, URL excludes, a page cap only when the page count is known, plusreasons, anestimateandwarnings.Billed as one or two
/maps and one/scrapeper sampled page (/mapis split into its handler andrun_map, which the job reuses). New job typecrawl_suggest.Also in this PR:
<url>carries an<image:image>/<video:video>(every Shopify store), the image's<image:loc>replaced the page's<loc>, so/mapand crawls got CDN image URLs instead of product pages. Only the direct children of<url>/<sitemap>are read now.suggestConfig/getSuggestion/suggestConfigAndWait(TypeScript),suggest_config/get_suggestion/suggest_config_and_wait(Python, sync and async).openapi.json,openapi.engine.json), docs (docs/api-reference/crawl-suggest.mdx, crawling guide, SDK pages), CHANGELOG.Testing
bins/scrapix-api/src/suggest/(mock Shopify-style shop with a mock LLM, the no-AI path, a JS app shell), sitemap parser regression tests, SDK tests (TS 37, Python 53).scrapix-apibinary does not link on macOS (same onmain), so the integration tests (openapi_snapshot,lab_boundary) were not run here; the engine snapshot was regenerated from the sameScrapixApi::openapi()code.Deploy notes
/crawl/*already routes to the engine (not in Caddy's@saas/SAAS_PREFIXES).contracts/openapi.json; the Lab console PR meilisearch/lab#142 merges after that.🤖 Generated with Claude Code