Skip to content

feat(schema): index Microdata in crawled documents (SCR-93) - #42

Merged
qdequele merged 1 commit into
mainfrom
scr-93-schemaorg-index-microdata-in-crawled-documents-fix
Nov 12, 2026
Merged

qdequele merged 1 commit into
mainfrom
scr-93-schemaorg-index-microdata-in-crawled-documents-fix

Conversation

@qdequele

Copy link
Copy Markdown
Owner

Linear: SCR-93

What changes

Crawled documents. The schema field now carries the page's schema.org Microdata items (itemscope / itemprop) after its JSON-LD. Until now only /scrape returned Microdata; a site that describes its products or articles in Microdata got no schema field in Meilisearch.

  • A Microdata item is left out when the page's JSON-LD already has a top-level item of the same type (@graph members count).
  • schema.only_types and schema.convert_dates apply to both formats.
  • A page with one JSON-LD script and Microdata of another type now has an array in schema where it had an object. Pages with several JSON-LD scripts already had an array.
  • Themes often mark WebPage, BreadcrumbList or SiteNavigationElement in Microdata. These now reach schema; only_types filters them.

Microdata extractor (also schema.microdata on /scrape):

  • An item inside another item without itemprop is its own item. An Article in a <body> marked as WebPage was lost.
  • itemref is followed.
  • itemprop="name headline" sets both properties.
  • content is read on any element, itemid is returned as @id.
  • Text values have their whitespace collapsed.
  • The work on one page is bounded: item nesting depth (16), a node budget, itemref loops.

Contract

None. /scrape keeps schema.items and schema.microdata as separate fields, without deduplication.

Tests

  • cargo fmt, cargo check, cargo clippy --all --all-targets -- -D warnings: clean.
  • cargo test -p scrapix-extractor -p scrapix-parser -p scrapix-worker-content: pass. New tests cover nested top-level items, values and multi-name properties, itemref, loops and deep nesting, the options, the merge rule, and both crawl paths (with and without schema options).
  • The rest of the workspace was not rerun locally, and nothing was checked against a live site.

Not covered

  • RDFa.
  • Top-level items without itemtype: they name no vocabulary, so they stay ignored.
  • URL values (href, src) are kept as written, not resolved to absolute URLs.

Deploy notes

  • Env and config: none.
  • Migrations: none.
  • Rollout: merge. Documents crawled after the deploy can gain Microdata items in schema; existing documents change on their next crawl.

🤖 Generated with Claude Code

… extractor (SCR-93)

The `schema` field of a crawled document now carries the page's
schema.org Microdata items after its JSON-LD. A Microdata item is left
out when the JSON-LD already has a top-level item of the same type.
`only_types` and `convert_dates` apply to both formats.

The Microdata extractor reads an item nested in another without
`itemprop` as its own item, follows `itemref`, splits multi-name
`itemprop`, reads `content` on any element, returns `itemid` as `@id`,
and bounds its work on one page.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@qdequele
qdequele merged commit 569b8ac into main Nov 12, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant