Skip to content

fix: index Tailwind pages' content, crawl every language from a language root - #44

Merged
qdequele merged 3 commits into
mainfrom
claude/crawler-auto-detect-nous-larochelle-ffaa53
Nov 17, 2026
Merged

qdequele merged 3 commits into
mainfrom
claude/crawler-auto-detect-nous-larochelle-ffaa53

Conversation

@qdequele

@qdequele qdequele commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Why

A Site Search onboarding of nous-larochelle.fr (Next.js + Tailwind, in fr/en/es/de) crawled 700 pages and indexed 36, all English.

  • Content dropped. Readability matched its non-content class names as substrings: <main class="group-data-[menu=opened]/body:hidden"> contains menu, so the whole <main> was discarded and most pages' content became "Skip to main content / Skip to footer". With near-duplicate detection on, every such page was skipped as a copy of the first.
  • One language. Suggested from /en, the config only crawled /en/**, though the site has /fr, /es and /de with translated section slugs.

What

  • scrapix-parser (readability): class/id names matched word by word (camelCase split, plurals and compounds like navbar/megamenu kept; address, shadow-md, downloads no longer match); Tailwind variant tokens (md:hidden) ignored; the chosen content element (<main>, <article>) is never vetoed by its own classes; a short <main> beats the body fallback; hidden elements (hidden, aria-hidden, display:none, hidden/d-none/sr-only unless a responsive variant shows them) left out; block elements set apart in text.
  • POST /crawl/suggest: a language root next to other languages widens to its parent, with a warning. SiteProfile.base_path (new, additive) gives the path the crawl stays under. Both OpenAPI specs updated, SDKs regenerated.

Lab side (adopts base_path, no page cap when the map was truncated): https://github.com/meilisearch/lab/pull/146.

Tests

  • New readability tests (this site's layout, hidden elements, word matching, short <main>); the old "known false positive" test now asserts.
  • New end-to-end suggest test on a mock fr/en/de site; language_root_parent unit test.
  • cargo test --lib for parser, extractor, worker-content and api pass; clippy clean. The openapi_snapshot integration test couldn't link locally (rdkafka/OpenSSL on macOS); the snapshot was checked through an equivalent lib test.

Deploy notes

None: no env, config or migrations. Additive contract change: release here, then the Lab's Contracts sync re-vendors openapi.json. The Lab change works with or without the new field.

🤖 Generated with Claude Code

qdequele and others added 3 commits November 17, 2026 10:00
…ents

Extraction matched its non-content class names (menu, nav, ad, ...) as
substrings, so a Tailwind `<main class="group-data-[menu=opened]/body:hidden">`
was dropped as a menu and `shadow-md` or `address` as ads. On
nous-larochelle.fr the content fell back to the skip links, and with
near-duplicate detection on, 664 of 700 crawled pages were skipped as
copies of the first.

Class names are now matched word by word (camelCase split, Tailwind
variants ignored), the element chosen as the content is never vetoed by
its own classes, a short <main> beats the body, hidden elements are left
out and blocks are set apart in the text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /crawl/suggest from https://site/en on a site also in /fr and /de
proposed a crawl of /en/** only. A language root next to other languages
now widens to its parent, with a warning, and the profile's new
`base_path` says where the crawl stays (additive contract change; SDKs
regenerated).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…/claim)

meilisearch/lab main added `POST /internal/deploys/claim` (the deploy
agent's claim of a requested deploy), which the engine does not call.
`contracts/check-drift.sh --fix`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qdequele
qdequele merged commit 02e7469 into main Nov 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant