Repository navigation
feat(sitemap): find sitemaps on the site's real host, at more locations and in more formats (SCR-91) - #41
Merged
qdequele merged 1 commit intoNov 18, 2026
Conversation
…ns and in more formats (SCR-91) Discovery read robots.txt and otherwise only /sitemap.xml, failed when robots.txt could not be fetched, and /map dropped every sitemap URL when the site answers on www and the request named the bare host. - robots.txt gives the host the site answers on (www or not, https); the other host is tried once when the first does not answer - usual file names at the root when nothing is declared or the declared sitemaps give nothing; sitemap.xml in the folder of the start URL - RSS/Atom feeds and text sitemaps; an HTML page is not a sitemap - /map keeps URLs of the www and bare host, lists the start URL's section first and returns the sitemaps it read Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
qdequele
force-pushed
the
scr-91-sitemap-discovery-canonical-host-more-locations-more
branch
from
November 18, 2026 07:35
40d7e02 to
af5961f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sitemap discovery read
robots.txtand otherwise only/sitemap.xml. It failed whenrobots.txtcould not be fetched, and/mapdropped every sitemap URL when the site answers onwwwand the request named the bare host (or the reverse). Shared by/map,/extractglobs and crawls. Linear: SCR-91.What changes
robots.txttells where the site lives: when it redirects to thewww(or bare) host, or to https, the sitemaps are looked up there. When the host does not answer at all, the other one is tried once./mapkeeps URLs and links of both hosts.sitemap.xml,sitemap_index.xml,sitemap-index.xml,sitemap.xml.gz,wp-sitemap.xmlat the root, fetched in parallel. A start URL with a path also readssitemap.xmlin its folder and in its first folder, before the site-wide ones.robots.txtno longer stops discovery (it gets 10 s). URLs are deduplicated across sitemaps, and a file reachable under two names is read once. A gzipped sitemap is capped at 50 MB once decompressed./map. URLs under the folder of the start URL come first, solimitkeeps them; nothing is filtered out. The response has a new field,sitemaps: the sitemaps the links were read from.Contract
MapResponsegains a requiredsitemaps: string[]incontracts/openapi.jsonandcontracts/openapi.engine.json. The Python and TypeScript SDKs are regenerated. The Lab's drift check fails until it re-vendorsopenapi.json.Tests
cargo fmt,cargo clippy --all --all-targets -- -D warningspass.cargo test -p scrapix-crawler -p scrapix-api -p scrapix-worker-crawler: 643 passed, 0 failed, 16 of them new. The rest of the workspace was not rerun locally.meilisearch.comresolves towww.meilisearch.com(902 URLs in 0.7 s);/docs/learnalso readsdocs/sitemap.xml; the text sitemap ofdoc.rust-lang.orgis read; sites without a sitemap return empty in about 1 s.Not covered
/extractglobs get the new discovery but not thewwwequivalence: the pattern itself names the host.wwwcase.Deploy notes
openapi.json.🤖 Generated with Claude Code