Repository navigation
feat(schema): index Microdata in crawled documents (SCR-93) - #42
Merged
qdequele merged 1 commit intoNov 12, 2026
Merged
Conversation
… extractor (SCR-93) The `schema` field of a crawled document now carries the page's schema.org Microdata items after its JSON-LD. A Microdata item is left out when the JSON-LD already has a top-level item of the same type. `only_types` and `convert_dates` apply to both formats. The Microdata extractor reads an item nested in another without `itemprop` as its own item, follows `itemref`, splits multi-name `itemprop`, reads `content` on any element, returns `itemid` as `@id`, and bounds its work on one page. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear: SCR-93
What changes
Crawled documents. The
schemafield now carries the page's schema.org Microdata items (itemscope/itemprop) after its JSON-LD. Until now only/scrapereturned Microdata; a site that describes its products or articles in Microdata got noschemafield in Meilisearch.@graphmembers count).schema.only_typesandschema.convert_datesapply to both formats.schemawhere it had an object. Pages with several JSON-LD scripts already had an array.WebPage,BreadcrumbListorSiteNavigationElementin Microdata. These now reachschema;only_typesfilters them.Microdata extractor (also
schema.microdataon/scrape):itempropis its own item. AnArticlein a<body>marked asWebPagewas lost.itemrefis followed.itemprop="name headline"sets both properties.contentis read on any element,itemidis returned as@id.itemrefloops.Contract
None.
/scrapekeepsschema.itemsandschema.microdataas separate fields, without deduplication.Tests
cargo fmt,cargo check,cargo clippy --all --all-targets -- -D warnings: clean.cargo test -p scrapix-extractor -p scrapix-parser -p scrapix-worker-content: pass. New tests cover nested top-level items, values and multi-name properties,itemref, loops and deep nesting, the options, the merge rule, and both crawl paths (with and without schema options).Not covered
itemtype: they name no vocabulary, so they stay ignored.href,src) are kept as written, not resolved to absolute URLs.Deploy notes
schema; existing documents change on their next crawl.🤖 Generated with Claude Code