Search engines:
- add robots.txt pointing at the sitemap
- give the home page its own <title> instead of the site name twice
- describe the software in JSON-LD on the home page, and each other page
as a TechArticle with a breadcrumb
- rewrite the page descriptions that only repeated the page title
- fix the heading level of --initialPageParam in the CLI options
LLMs and coding agents:
- build llms.txt from the pages with starlight-llms-txt, which adds
llms-full.txt, llms-small.txt and one file per topic. The handwritten
llms.txt had fallen behind (it did not list the benchmarks).
- serve every page as Markdown at its own address plus .md, and link it
from the page with rel="alternate"
- keep the Markdown copies out of search results with X-Robots-Tag
hast-util-to-mdast resolves to 10.1.2: 10.1.3 was published less than
seven days ago.
Why
The documentation site was missing the basics that search engines and LLM crawlers look for: no
robots.txt, a home page<title>that was the site name twice, no structured data, several descriptions that only repeated the page title, and a handwrittenllms.txtthat had fallen behind the pages (it did not list the benchmarks).What changes
Search engines
robots.txtthat points at the sitemap.OpenAPI React Query Codegen (orqc): TanStack Query hooks from OpenAPI.WebSiteandSoftwareApplicationon the home page,TechArticleand a breadcrumb on every other page (src/routeData.ts).--initialPageParamin the CLI options was anh2amongh3s.LLMs and coding agents
llms.txtis built from the pages bystarlight-llms-txt, with the same "Key facts for agents" as before. It adds/llms-full.txt,/llms-small.txtand one file per topic under/_llms-txt/..md(/guides/usage.md), linked from the page withrel="alternate" type="text/markdown".docs/vercel.jsonsendsX-Robots-Tag: noindexfor the Markdown copies so they do not compete with the pages in search results.Notes for review
src/pages/[...slug].md.tsimportsentryToSimpleMarkdownfrom insidestarlight-llms-txt, which does not export it. It is the conversion that buildsllms-full.txt, and the plugin version is pinned, so a release that moves the file fails the build.hast-util-to-mdastresolves to 10.1.2 because 10.1.3 was published less than seven days ago; no override was added. The lockfile review script passes locally.vitepeer range of vitest goes back from8.3.6to the range the package declares.docs/vercel.jsononly takes effect if the Vercel project's root directory isdocs. Worth checking the header on the preview:curl -I <preview>/guides/usage.md.llms.txtnow links to the documentation sets instead of listing each page.Checked
pnpm --filter docs build(astro check: 0 errors), then read the built<head>,robots.txt,llms*.txtand the.mdfiles.pnpm lint,pnpm test(753 passed),pnpm install --frozen-lockfile.Summary by CodeRabbit