Repository navigation
fix: index Tailwind pages' content, crawl every language from a language root - #44
Merged
qdequele merged 3 commits intoNov 17, 2026
Merged
Conversation
…ents Extraction matched its non-content class names (menu, nav, ad, ...) as substrings, so a Tailwind `<main class="group-data-[menu=opened]/body:hidden">` was dropped as a menu and `shadow-md` or `address` as ads. On nous-larochelle.fr the content fell back to the skip links, and with near-duplicate detection on, 664 of 700 crawled pages were skipped as copies of the first. Class names are now matched word by word (camelCase split, Tailwind variants ignored), the element chosen as the content is never vetoed by its own classes, a short <main> beats the body, hidden elements are left out and blocks are set apart in the text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /crawl/suggest from https://site/en on a site also in /fr and /de proposed a crawl of /en/** only. A language root next to other languages now widens to its parent, with a warning, and the profile's new `base_path` says where the crawl stays (additive contract change; SDKs regenerated). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…/claim) meilisearch/lab main added `POST /internal/deploys/claim` (the deploy agent's claim of a requested deploy), which the engine does not call. `contracts/check-drift.sh --fix`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Site Search onboarding of nous-larochelle.fr (Next.js + Tailwind, in fr/en/es/de) crawled 700 pages and indexed 36, all English.
<main class="group-data-[menu=opened]/body:hidden">containsmenu, so the whole<main>was discarded and most pages'contentbecame "Skip to main content / Skip to footer". With near-duplicate detection on, every such page was skipped as a copy of the first./en, the config only crawled/en/**, though the site has/fr,/esand/dewith translated section slugs.What
scrapix-parser(readability): class/id names matched word by word (camelCase split, plurals and compounds likenavbar/megamenukept;address,shadow-md,downloadsno longer match); Tailwind variant tokens (md:hidden) ignored; the chosen content element (<main>,<article>) is never vetoed by its own classes; a short<main>beats the body fallback; hidden elements (hidden,aria-hidden,display:none,hidden/d-none/sr-onlyunless a responsive variant shows them) left out; block elements set apart in text.POST /crawl/suggest: a language root next to other languages widens to its parent, with a warning.SiteProfile.base_path(new, additive) gives the path the crawl stays under. Both OpenAPI specs updated, SDKs regenerated.Lab side (adopts
base_path, no page cap when the map was truncated): https://github.com/meilisearch/lab/pull/146.Tests
<main>); the old "known false positive" test now asserts.language_root_parentunit test.cargo test --libfor parser, extractor, worker-content and api pass; clippy clean. Theopenapi_snapshotintegration test couldn't link locally (rdkafka/OpenSSL on macOS); the snapshot was checked through an equivalent lib test.Deploy notes
None: no env, config or migrations. Additive contract change: release here, then the Lab's Contracts sync re-vendors
openapi.json. The Lab change works with or without the new field.🤖 Generated with Claude Code