Skip to content

docs: improve how search engines and LLMs find and read the site - #274

Merged
7nohe merged 2 commits into
mainfrom
docs/seo-llmo
Nov 15, 2026
Merged

7nohe merged 2 commits into
mainfrom
docs/seo-llmo

Conversation

@7nohe

@7nohe 7nohe commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Why

The documentation site was missing the basics that search engines and LLM crawlers look for: no robots.txt, a home page <title> that was the site name twice, no structured data, several descriptions that only repeated the page title, and a handwritten llms.txt that had fallen behind the pages (it did not list the benchmarks).

What changes

Search engines

  • robots.txt that points at the sitemap.
  • The home page title is now OpenAPI React Query Codegen (orqc): TanStack Query hooks from OpenAPI.
  • JSON-LD: WebSite and SoftwareApplication on the home page, TechArticle and a breadcrumb on every other page (src/routeData.ts).
  • Nine page descriptions rewritten to say what the page covers.
  • --initialPageParam in the CLI options was an h2 among h3s.

LLMs and coding agents

  • llms.txt is built from the pages by starlight-llms-txt, with the same "Key facts for agents" as before. It adds /llms-full.txt, /llms-small.txt and one file per topic under /_llms-txt/.
  • Every page is served as Markdown at its own address plus .md (/guides/usage.md), linked from the page with rel="alternate" type="text/markdown".
  • docs/vercel.json sends X-Robots-Tag: noindex for the Markdown copies so they do not compete with the pages in search results.

Notes for review

  • src/pages/[...slug].md.ts imports entryToSimpleMarkdown from inside starlight-llms-txt, which does not export it. It is the conversion that builds llms-full.txt, and the plugin version is pinned, so a release that moves the file fails the build.
  • The lockfile gains 36 package versions. hast-util-to-mdast resolves to 10.1.2 because 10.1.3 was published less than seven days ago; no override was added. The lockfile review script passes locally.
  • One unrelated lockfile line changes: the vite peer range of vitest goes back from 8.3.6 to the range the package declares.
  • docs/vercel.json only takes effect if the Vercel project's root directory is docs. Worth checking the header on the preview: curl -I <preview>/guides/usage.md.
  • llms.txt now links to the documentation sets instead of listing each page.

Checked

  • pnpm --filter docs build (astro check: 0 errors), then read the built <head>, robots.txt, llms*.txt and the .md files.
  • pnpm lint, pnpm test (753 passed), pnpm install --frozen-lockfile.

Summary by CodeRabbit

  • New Features
    • Documentation is now available in full and condensed LLM-friendly formats, with individual pages also served as Markdown.
    • Added structured page and breadcrumb data, social preview image dimensions, and richer metadata for the documentation home page.
  • Documentation
    • Updated page descriptions to better summarize examples, CLI options, usage, and migration guides.
    • Clarified how to access Markdown documentation.
  • Chores
    • Updated crawler access rules and configured no-index headers for Markdown and generated documentation paths.

Search engines:
- add robots.txt pointing at the sitemap
- give the home page its own <title> instead of the site name twice
- describe the software in JSON-LD on the home page, and each other page
  as a TechArticle with a breadcrumb
- rewrite the page descriptions that only repeated the page title
- fix the heading level of --initialPageParam in the CLI options

LLMs and coding agents:
- build llms.txt from the pages with starlight-llms-txt, which adds
  llms-full.txt, llms-small.txt and one file per topic. The handwritten
  llms.txt had fallen behind (it did not list the benchmarks).
- serve every page as Markdown at its own address plus .md, and link it
  from the page with rel="alternate"
- keep the Markdown copies out of search results with X-Robots-Tag

hast-util-to-mdast resolves to 10.1.2: 10.1.3 was published less than
seven days ago.
@vercel

vercel Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
openapi-react-query-codegen Ready Ready Preview Nov 15, 2026 1:31am UTC

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The documentation site now configures generated LLM text files, serves non-draft documentation pages as Markdown, and adds page metadata and crawler directives. Documentation descriptions and agent-facing guidance were also updated.

Changes: Documentation site outputs

Layer / File(s) Summary
Configure generated LLM documentation
docs/package.json, docs/astro.config.mjs, docs/public/llms.txt, docs/src/content/docs/examples/*, docs/src/content/docs/guides/*
The site adds the starlight-llms-txt plugin and its project details, removes the static llms.txt guide, and updates descriptions for example and guide pages.
Serve Markdown pages and add page metadata
docs/src/pages/[...slug].md.ts, docs/src/routeData.ts, docs/astro.config.mjs, docs/src/content/docs/index.mdx, docs/src/content/docs/guides/ci-and-agents.mdx, docs/public/robots.txt, docs/vercel.json
The site adds prerendered Markdown routes, Markdown alternate links, and JSON-LD page metadata. The home page metadata, sitemap declaration, and noindex response headers are also updated.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Merge Risk: Minimal · up to ff6ba

This change improves how search engines and LLMs read the documentation site. No merge-blocking issues were found. It is worth checking the preview to confirm the noindex headers are applied.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "docs: improve how search engines and LLMs find and read the site" matches the main change. The PR adds robots.txt and a sitemap reference, replaces the handwritten llms.txt with generated l…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "docs: improve how search engines and LLMs find and read the site" matches the main change. The PR adds robots.txt and a sitemap reference, replaces the handwritten llms.txt with generated llms-full.txt, llms-small.txt and topic files, serves pages as Markdown, and adds JSON-LD and page metadata for search engines. The title is specific and uses the conventional docs: prefix. It does not cover every detail, such as the heading-level fix in the CLI options page, which is expected.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d7acc8cf-0dbc-4a6c-9ed4-e51a53779c47
Commits

Reviewing files that changed from the base of the PR and between 6060520 and ff6bab1.

Files ignored due to path filters: 1

  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
Files selected for processing: 18

  • docs/astro.config.mjs
  • docs/package.json
  • docs/public/llms.txt
  • docs/public/robots.txt
  • docs/src/content/docs/examples/nextjs.md
  • docs/src/content/docs/examples/react-router.md
  • docs/src/content/docs/examples/tanstack-router.md
  • docs/src/content/docs/examples/vite-react.md
  • docs/src/content/docs/guides/ci-and-agents.mdx
  • docs/src/content/docs/guides/cli-options.mdx
  • docs/src/content/docs/guides/introduction.mdx
  • docs/src/content/docs/guides/migrating-to-v3.mdx
  • docs/src/content/docs/guides/migrating-to-v4.mdx
  • docs/src/content/docs/guides/usage.mdx
  • docs/src/content/docs/index.mdx
  • docs/src/pages/[...slug].md.ts
  • docs/src/routeData.ts
  • docs/vercel.json
Files with no reviewable changes: 1

  • docs/public/llms.txt

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

main edited docs/public/llms.txt to list the new Mocking with MSW guide.
That file is generated now, so the guide joins the "Getting started and
usage" set and the pages sorted first in llms-full.txt instead, and the
CLI options description mentions --mock.

pnpm-lock.yaml is main's with starlight-llms-txt added again;
hast-util-to-mdast still resolves to 10.1.2.
@github-actions

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🟢 Lines 99.04% (🎯 95%) 2797 / 2824
🟢 Statements 98.65% (🎯 95%) 3149 / 3192
🟢 Functions 98.48% (🎯 95%) 587 / 596
🟢 Branches 96.95% (🎯 90%) 2770 / 2857
File CoverageNo changed files found.
Generated in workflow #635 for commit ff6bab1 by the Vitest Coverage Report Action

@7nohe
7nohe merged commit e464947 into main Nov 15, 2026
9 checks passed
@7nohe
7nohe deleted the docs/seo-llmo branch November 15, 2026 04:19

This branch was successfully deployed

1 active deployment
Preview — ff6bab13 Deployed Nov 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant