Make product docs landing pages stronger for SEO - #3738
Conversation
Move Observability, Prompt Management, Evaluation, and Metrics from /overview to the section root. Rewrite those pages as keyword-aligned pillar content, add FAQ and breadcrumb structured data, raise sitemap priority, pin the pages in llms.txt, and point spoke docs at the hubs. Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Treat each product index.mdx as the folder page so breadcrumbs read Docs > Observability (not Overview). Point homepage tool cards and a few stale hub links at the new section roots, and hop old /scores, /tracing-features, and /experimentation URLs straight to their destinations. Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
|
Too many files changed for review (193 files, 100 file limit). Bypass the limit by tagging |
|
@claude review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.
Tip: disable this comment in your organization's Code Review settings.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1fb26800e0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Sitemap priority and llms.txt pinning now read the same CJS list so a rename cannot update one without the other. Also hop /docs/{product}/overview.md to the new section-root markdown URLs so agent bookmarks do not 404.
Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
Structured data now includes Home, known section/product landings, and the current page. Intermediate paths such as /docs/observability/features have no index page, so they are skipped instead of pointing crawlers at 404s. Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
components/DocsTocFooter.tsx— getGithubEditUrl still builds<contentDir>/<slugPath>.mdxfor every non-root path, so on the four new product landing pages it now links tocontent/docs/observability.mdx(etc.) — a file this PR deleted (moved toobservability/index.mdx). The 'Edit this page on GitHub' action on /docs/observability, /docs/prompt-management, /docs/evaluation, and /docs/metrics now opens GitHub's create-new-file view instead of editing the real file. Fix: special-case a slug that resolves to a folder with an index.mdx (or generally try<slugPath>.mdxthen fall back to<slugPath>/index.mdx), mirroring the fallbackgetContributorsalready has a few lines below.Extended reasoning...
getGithubEditUrl(path) splits
pathinto section+slugParts and computesfilePath = contentDir/slugPath.mdxunless slugParts is empty (root case, mapped toindex.mdx). It has no equivalent handling for a slug that maps to a folder whose content lives inindex.mdxone level deeper. Before this PR,/docs/observabilitydidn't exist as a page at all (it 302'd), and/docs/observability/overviewcorrectly producedcontent/docs/observability/overview.mdx, which existed. After this PR, content/docs/observability/overview.mdx was deleted and content/docs/observability/index.mdx added, and the route/docs/observabilityis now a real, highly-trafficked landing page — but getGithubEditUrl (untouched by the diff) still computescontent/docs/observability.mdx, which does not exist on disk, so every visitor who clicks 'Edit this page on GitHub' on the four new landing pages lands on a nonexistent-file edit screen instead of the real source file.Verification: normal (low-impact but a real newly-introduced regression).
getGithubEditUrlin components/DocsTocFooter.tsx:33-36 computesfilePath = ${contentDir}/${slugPath === "" ? "index" : slugPath}.mdx. It only maps the empty-slug (root) case toindex; a slug that resolves to a folder whose content lives inindex.mdxis not handled. For pathname/docs/observability: section="docs" (in…
| if (node.type === "page") { | ||
| const prev = i > 0 ? path[i - 1] : undefined; | ||
| const isFolderIndex = | ||
| prev?.type === "folder" && | ||
| (prev.index === node || prev.index?.url === node.url); | ||
| // Folder index pages use the folder name (Observability), not the | ||
| // page sidebar title (Overview). The section root is the leading crumb. | ||
| if (isFolderIndex || node.url === resolveSectionUrl(root)) continue; | ||
| result.push({ name: node.name, url: node.url }); | ||
| } else if (node.type === "folder") { | ||
| // The active root folder is rendered as the leading crumb below. | ||
| if (node.root) continue; | ||
| // Fumadocs collapses a folder and its index page into a single crumb. | ||
| if (i === path.length - 1 || node.index !== path[i + 1]) { | ||
| result.push({ | ||
| name: node.name, | ||
| url: node.index?.url ?? resolveFirstUrl(node), | ||
| }); | ||
| } | ||
| result.push({ | ||
| name: node.name, | ||
| url: node.index?.url ?? resolveFirstUrl(node), | ||
| }); |
There was a problem hiding this comment.
🟡 (optional) The folder-index collapsing rule was flipped to always show the folder's meta.json title and skip the index page's own name for EVERY folder with an index page sitewide, not only the four new product pages this PR manually checked. Existing folders that already used index.mdx with a distinct sidebarTitle (e.g. self-hosting/configuration, self-hosting/upgrade) now show a less specific breadcrumb label than before. Fix: only apply the folder-name-wins rule for the new product-section landing pages, or preserve using the page's own name/sidebarTitle when it differs from the folder title, so previously informative breadcrumbs on unrelated pre-existing pages aren't silently degraded.
Extended reasoning...
Before this diff, for a folder whose index page is the currently viewed page, the code pushed the folder crumb only when node.index !== path[i+1] (line removed); when the index page IS the next node, the folder crumb was skipped and the page node was pushed instead using node.name, which fumadocs derives from the page's own sidebarTitle frontmatter. Several already-published folders use index.mdx with a sidebarTitle distinct from the folder's meta.json title: content/self-hosting/configuration/index.mdx has sidebarTitle: "Environment Variables" while content/self-hosting/configuration/meta.json has "title": "Configuration"; content/self-hosting/upgrade/index.mdx has sidebarTitle: "How to Upgrade" vs meta.json "title": "Upgrade". After this diff, the folder crumb is now pushed unconditionally (new line 94-97) and the page crumb is skipped whenever isFolderIndex is true (new lines 84-89), so visiting /self-hosting/configuration now renders the breadcrumb ending in "Configuration" instead of the previous "Environment Variables", and…
Verification: nit. The reworked folder-index rule at components/DocsBreadcrumb.tsx:84-97 applies to EVERY folder whose index is the current page, not only the four product pages. For a folder-index page, fumadocs' searchPath returns the trail [folder, folder.index] (corroborated by the removed guard node.index !== path[i + 1]). Old code skipped the folder crumb and pushed the index page with its own .name…
There was a problem hiding this comment.
Leaving this as-is for this PR. Showing the folder title (Observability, Configuration) instead of the index page’s sidebarTitle (Overview, Environment Variables) is the intended breadcrumb for a folder landing. The product pages were the ones that looked wrong with a trailing Overview crumb. Happy to revisit self-hosting labels in a follow-up if you want those index pages to keep a distinct crumb name.
| function humanizeSegment(segment: string): string { | ||
| return ( | ||
| SLUG_LABELS[segment] ?? | ||
| segment | ||
| .split("-") | ||
| .map((word) => word.charAt(0).toUpperCase() + word.slice(1)) | ||
| .join(" ") | ||
| ); | ||
| } |
There was a problem hiding this comment.
🟡 (optional) breadcrumbListJsonLd derives ListItem names by title-casing raw URL slugs (falling back when not in the small SLUG_LABELS map), so the BreadcrumbList JSON-LD emitted on every docs page (wired in DocsChromePage.tsx) mismatches the visible breadcrumb rendered by DocsBreadcrumb.tsx, which uses the real page-tree titles. E.g. folder "sdk" (title "SDKs") renders as "Sdk", and page "url" (title "Trace URLs") renders as "Url" in the structured data — a Rich Results / Search Console breadcrumb-mismatch warning, undermining this PR's own SEO goal. Fix: build breadcrumb names from the actual page-tree node names used by DocsBreadcrumb (or pass them through), not from humanized URL segments.
Extended reasoning...
DocsChromePage.tsx calls breadcrumbListJsonLd(pageUrl) for every docs page and injects it via . The function (lib/json-ld.ts:37-59) only knows the URL, so for each segment it calls humanizeSegment (line 25-33), which looks up a tiny hardcoded SLUG_LABELS dict and otherwise just capitalizes each hyphen-split word. For content/docs/observability/sdk/meta.json the folder title is "SDKs" (shown in the visible breadcrumb via DocsBreadcrumb.tsx's node.name), but the JSON-LD for /docs/observability/sdk/... emits name: "Sdk". Same for content/docs/observability/features/url.mdx, titled "Trace URLs", where JSON-LD says "Url". Google's structured-data guidelines require breadcrumb JSON-LD to match the on-page breadcrumb; this mismatch is systemic across most nested docs pages, not a one-off, and is entirely new code introduced by this diff.
Verification: nit. breadcrumbListJsonLd (lib/json-ld.ts:37-59) names each crumb from the URL slug via humanizeSegment (lines 25-33): SLUG_LABELS lookup, else capitalize hyphen-split tokens. DocsChromePage.tsx:61,76 injects it on every sidebar docs page. The visible breadcrumb (DocsBreadcrumb.tsx:90,94,101) instead uses real page-tree titles (node.name/root.name) from meta.json/frontmatter. Verified…
There was a problem hiding this comment.
Agreed this is a real polish item, but it needs the page-tree titles on the server, not URL slug humanizing. Out of scope for this PR — the JSON-LD trail now at least points at resolvable URLs. Fine to take as a follow-up so structured names match the visible crumbs (SDKs, Trace URLs, …).
Product landings live at content/docs/<product>/index.mdx. The TOC footer used to build content/docs/<product>.mdx, which does not exist. Resolve the source file on the server by trying the slug file first and falling back to index.mdx. Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
|
Fixed the Edit-on-GitHub regression for the four product landings. |
Use Agent in product titles, nav, and hub links for observability, evals, and metrics. Drop the LLM prefix from prompt management. Move the standalone LLM observability FAQ into the pillar FAQ accordion and 301 the old URL so one page ranks for the topic. Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
The Vercel preview started failing on the commit that added fs.existsSync to every docs-chrome page. Map folder landings to index.mdx from a static path set instead so the server module no longer pulls in a Node builtin. Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b0ebc20. Configure here.
Keep the static path map so this module does not import Node fs. Cookbook, integration, and engineering pages that live as .md now get that extension instead of a missing .mdx file. Co-authored-by: marc.klingen <marc.klingen@clickhouse.com>
2 participants
Move the product overview pages linked from the top-nav Product menu to shorter, keyword-rich URLs and treat them as SEO pillar pages.
What changed
URL swap
/docs/observability/overview→/docs/observability/docs/prompt-management/overview→/docs/prompt-management/docs/evaluation/overview→/docs/evaluation/docs/metrics/overview→/docs/metricsOld
/overviewand/overview.mdURLs redirect to the section roots. Redirect chains such as/observability→/docs/tracing→ overview,/docs/scores→/docs/scores/overview→ evaluation, and/docs/tracing-features→/docs/tracingare flattened./docs/promptsnow goes to prompt management (not get-started)./docs/api-and-data-platformis unchanged.On-page
Each landing page has a keyword-aligned
seoTitleand H1, a short definition and audience, an eager-loaded hero, concrete jobs-to-be-done, and an FAQ./docsstays the platform hub.Current titles:
Nav and footer use Agent Observability, Prompt Management, Agent Evals, and Agent Analytics.
Cannibalization
/faq/all/llm-observability(“What is LLM Observability & Monitoring?”). That page was already canonicalized to the pillar, so it added no independent ranking value. Its unique explainer (why agents need observability; observability vs tracing vs monitoring; what an agent observability platform is; how to monitor outputs) now lives in the observability FAQ accordion./faq/all/llm-observability301s to/docs/observability#faqTechnical
Docs > Observability(folder name, not an extra Overview crumb)BreadcrumbListJSON-LD on docs-chrome pagesFAQPageJSON-LD on the four product pagesSoftwareApplicationJSON-LD on/docs0.9for the five product URLsllms.txtsectionlib/product-overview-paths.jsso sitemap + llms.txt stay in syncindex.mdxwithout importing Nodefs(that import failed Vercel preview tracing)Verification
Titles, H1s, FAQ JSON-LD, and the FAQ 301 were checked against the local dev server. Browser walkthrough confirmed the Product menu labels, landing H1s, expanded observability FAQs, and the old FAQ URL landing on
#faq. GitHub Actions format, H1, build, and sitemap checks passed. Vercel preview failed starting at thefs.existsSyncEdit-on-GitHub commit; that import is removed on this head.