Skip to content

feat(sitemap): find sitemaps on the site's real host, at more locations and in more formats (SCR-91) - #41

Merged
qdequele merged 1 commit into
mainfrom
scr-91-sitemap-discovery-canonical-host-more-locations-more
Nov 18, 2026
Merged

qdequele merged 1 commit into
mainfrom
scr-91-sitemap-discovery-canonical-host-more-locations-more

Conversation

@qdequele

Copy link
Copy Markdown
Owner

Sitemap discovery read robots.txt and otherwise only /sitemap.xml. It failed when robots.txt could not be fetched, and /map dropped every sitemap URL when the site answers on www and the request named the bare host (or the reverse). Shared by /map, /extract globs and crawls. Linear: SCR-91.

What changes

  • Host. robots.txt tells where the site lives: when it redirects to the www (or bare) host, or to https, the sitemaps are looked up there. When the host does not answer at all, the other one is tried once. /map keeps URLs and links of both hosts.
  • Locations. Without a declared sitemap, or when the declared ones give no URL: sitemap.xml, sitemap_index.xml, sitemap-index.xml, sitemap.xml.gz, wp-sitemap.xml at the root, fetched in parallel. A start URL with a path also reads sitemap.xml in its folder and in its first folder, before the site-wide ones.
  • Formats. RSS and Atom feeds, and text sitemaps (one URL per line). An HTML page served with a 200 in place of a sitemap is skipped.
  • Robustness. An unreachable robots.txt no longer stops discovery (it gets 10 s). URLs are deduplicated across sitemaps, and a file reachable under two names is read once. A gzipped sitemap is capped at 50 MB once decompressed.
  • /map. URLs under the folder of the start URL come first, so limit keeps them; nothing is filtered out. The response has a new field, sitemaps: the sitemaps the links were read from.

Contract

MapResponse gains a required sitemaps: string[] in contracts/openapi.json and contracts/openapi.engine.json. The Python and TypeScript SDKs are regenerated. The Lab's drift check fails until it re-vendors openapi.json.

Tests

  • cargo fmt, cargo clippy --all --all-targets -- -D warnings pass.
  • cargo test -p scrapix-crawler -p scrapix-api -p scrapix-worker-crawler: 643 passed, 0 failed, 16 of them new. The rest of the workspace was not rerun locally.
  • Live check: meilisearch.com resolves to www.meilisearch.com (902 URLs in 0.7 s); /docs/learn also reads docs/sitemap.xml; the text sitemap of doc.rust-lang.org is read; sites without a sitemap return empty in about 1 s.

Not covered

  • /extract globs get the new discovery but not the www equivalence: the pattern itself names the host.
  • The crawl's own domain scoping is untouched and was not checked for the www case.
  • A site with no declared sitemap now gets up to 5 requests in parallel instead of 1, also during a crawl.

Deploy notes

  • Env and config: none.
  • Migrations: none.
  • Rollout: merge and release here, then the Lab's Contracts sync PR re-vendors openapi.json.

🤖 Generated with Claude Code

…ns and in more formats (SCR-91)

Discovery read robots.txt and otherwise only /sitemap.xml, failed when
robots.txt could not be fetched, and /map dropped every sitemap URL when
the site answers on www and the request named the bare host.

- robots.txt gives the host the site answers on (www or not, https); the
  other host is tried once when the first does not answer
- usual file names at the root when nothing is declared or the declared
  sitemaps give nothing; sitemap.xml in the folder of the start URL
- RSS/Atom feeds and text sitemaps; an HTML page is not a sitemap
- /map keeps URLs of the www and bare host, lists the start URL's section
  first and returns the sitemaps it read

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@qdequele
qdequele force-pushed the scr-91-sitemap-discovery-canonical-host-more-locations-more branch from 40d7e02 to af5961f Compare November 18, 2026 07:35
@qdequele
qdequele merged commit 1ba88c0 into main Nov 18, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant