LOU-D12: llms.txt, llms-full.txt and docs shipped in the npm package

.github/workflows/ci.yml

@@ -37,6 +37,8 @@ jobs:
       # Every ```ts block in README.md and docs/*.md must type-check against
       # src/ (quick-start also runs against the packed tarball). Needs dist/.
       - run: npm run docs:verify-snippets -- --skip-build
+      # llms.txt / llms-full.txt are generated from README + docs; fail if stale.
+      - run: npm run docs:llms:check
       - run: npm run test:coverage
       # Agent Forge (apps/agent-forge) imports the SDK as
       # `@loushy/build-ai-agent` via `file:../..`, i.e. the root dist/ built

README.md

@@ -216,6 +216,19 @@ console.log(result.usage.totalTokens, result.finishReason, result.steps);
 - [Agent Forge](docs/agent-forge.md) - the visual dashboard (`loushy studio`): quickstart, first-agent walkthrough, hook authoring
 - Full guides site: [linuxdevil.github.io/agent-sdk-docs](https://linuxdevil.github.io/agent-sdk-docs/)
 
+## For AI coding agents
+
+This package ships its documentation in machine-readable form, so a coding
+agent can read it straight from `node_modules` without browsing the web:
+
+- `node_modules/@loushy/build-ai-agent/llms-full.txt` - the README and every
+  user-facing docs page in one file, with absolute links.
+- `llms.txt` (repo root, also in the package) - a short [llmstxt.org](https://llmstxt.org)
+  index of the docs and examples.
+- `llms-full.txt` (repo root) - the same full text as above, for use from a clone.
+
+Both files are generated (`npm run docs:llms`) and checked in CI.
+
 ## Core Concepts
 
 ### Agents

llms.txt

@@ -0,0 +1,28 @@
+# @loushy/build-ai-agent
+
+> A composable, framework-agnostic SDK for building AI agents that run safely in production. Zero-config to full control, human-in-the-loop approval gates, durable checkpoints, multi-agent delegation, guardrails, and a visual dashboard — any provider, any deploy target, no lock-in.
+
+## Docs
+
+- [@loushy/build-ai-agent](https://github.com/LinuxDevil/agent-sdk/blob/main/README.md): A composable, framework-agnostic SDK for building AI agents that run safely in production.
+- [Installation](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/installation.md): `ai` (the Vercel AI SDK, `^4.3.19`) and `zod` (`^3.25.76`) are required peer dependencies.
+- [Quick Start](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/quick-start.md): Every TypeScript snippet on this page is a complete, standalone ES module (`.mts`).
+- [API Overview](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/api-overview.md): The package root (`@loushy/build-ai-agent`) exports everything below.
+- [Configuration](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/configuration.md): A declarative agent is a YAML (`.yaml`/`.yml`) or JSON (`.json`) file, validated with zod by `loadSpec()` (`src/spec/schema.ts`).
+- [Skills](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/skills.md): A skill is a named chunk of instructions (a markdown document) that the model loads only when it needs it.
+- [Testing agents](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/testing.md): `@loushy/build-ai-agent/testing` ships `mockModel`: a scripted, deterministic `LLMProvider` for unit tests.
+
+## Examples
+
+- [doc-qa](https://github.com/LinuxDevil/agent-sdk/blob/main/examples/doc-qa/README.md): A question-answering agent scoped to a single fixed document (a short refund policy, inlined in the prompt), built with `createAgent()`.
+- [openrouter](https://github.com/LinuxDevil/agent-sdk/blob/main/examples/openrouter/README.md): Runnable snippets showing `OpenRouterProvider` usage (LOU-B6): basic and streaming generation, multi-model comparison, tool calling, `AgentBuilder` configuration, model listing, cost-aware model choice, error handling, multi-turn conversation, and capability checks.
+- [ops-pipeline (LOU-J8)](https://github.com/LinuxDevil/agent-sdk/blob/main/examples/ops-pipeline/README.md): An end-to-end "flagship pipeline" demo composing every LOU-J primitive
+- [research-assistant](https://github.com/LinuxDevil/agent-sdk/blob/main/examples/research-assistant/README.md): An agent with the built-in `http` tool wired up so it can fetch sources while researching a question, built with `createAgent()`.
+- [slack-notifier](https://github.com/LinuxDevil/agent-sdk/blob/main/examples/slack-notifier/README.md): Turns a raw event description into a short, Slack-ready notification message, built with `createAgent()`.
+- [support-bot](https://github.com/LinuxDevil/agent-sdk/blob/main/examples/support-bot/README.md): A minimal customer-support agent built with `createAgent()`.
+- [workflow-router](https://github.com/LinuxDevil/agent-sdk/blob/main/examples/workflow-router/README.md): An agent that classifies an incoming request into one of a fixed set of categories (`billing`/`technical`/`sales`), built with `createAgent()`.
+
+## Optional
+
+- [Agent Forge](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/agent-forge.md): Agent Forge (`apps/agent-forge`) is this SDK's companion visual dashboard: build an agent's graph on a canvas, run it, watch it execute in a live debug console, chat with it, and author sandboxed pre/post hooks - all reading and writing the exact same `AgentSpec` YAML that `loushy dev` and `loushy build` use.
+- [Deployment](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/deployment.md): `loushy build` turns an agent spec file (see Configuration) into a deployable artifact for one target platform

package.json

@@ -75,6 +75,10 @@
     "apps/agent-forge/dist",
     "apps/agent-forge/dist-server",
     "README.md",
+    "llms.txt",
+    "llms-full.txt",
+    "docs/*.md",
+    "!docs/eslint-baseline-followup.md",
     "LICENSE",
     "CHANGELOG.md",
     "CONTRIBUTING.md"
@@ -108,6 +112,8 @@
     "prepublishOnly": "npm run build && npm run build:studio",
     "docs:build": "typedoc",
     "docs:verify-snippets": "tsx scripts/verify-docs-snippets.ts",
+    "docs:llms": "tsx scripts/generate-llms-txt.ts",
+    "docs:llms:check": "tsx scripts/generate-llms-txt.ts --check",
     "fallow": "fallow",
     "fallow:score": "fallow health --score",
     "example:openrouter": "tsx examples/openrouter/index.ts"

scripts/generate-llms-txt.ts

@@ -0,0 +1,79 @@
+/**
+ * generate-llms-txt (LOU-D12)
+ *
+ * Generates `llms.txt` and `llms-full.txt` at the repo root from README.md,
+ * the user-facing docs/*.md pages and examples/<name>/README.md. Both files
+ * are generated, never hand-edited, and are shipped in the npm package so
+ * coding agents can read them from node_modules.
+ *
+ * Usage:
+ *   npx tsx scripts/generate-llms-txt.ts           write the files
+ *   npx tsx scripts/generate-llms-txt.ts --check   exit 1 if they are stale
+ */
+import * as fs from 'node:fs';
+import * as path from 'node:path';
+import { isUserDoc, normalizeNewlines, renderLlmsFull, renderLlmsTxt } from './llmsTxt';
+import type { DocPage, ExampleEntry, LlmsInput } from './llmsTxt';
+
+const REPO_ROOT = path.resolve(__dirname, '..');
+
+function readText(file: string): string {
+  return normalizeNewlines(fs.readFileSync(file, 'utf8'));
+}
+
+function readDocs(): DocPage[] {
+  return fs
+    .readdirSync(path.join(REPO_ROOT, 'docs'))
+    .map((name) => `docs/${name}`)
+    .filter(isUserDoc)
+    .sort()
+    .map((p) => ({ path: p, markdown: readText(path.join(REPO_ROOT, p)) }));
+}
+
+function readExamples(): ExampleEntry[] {
+  return fs
+    .readdirSync(path.join(REPO_ROOT, 'examples'), { withFileTypes: true })
+    .filter((d) => d.isDirectory() && fs.existsSync(path.join(REPO_ROOT, 'examples', d.name, 'README.md')))
+    .map((d) => ({
+      dir: `examples/${d.name}`,
+      readme: readText(path.join(REPO_ROOT, 'examples', d.name, 'README.md')),
+    }));
+}
+
+function loadInput(): LlmsInput {
+  const pkg = JSON.parse(readText(path.join(REPO_ROOT, 'package.json'))) as {
+    name: string;
+    repository: { url: string };
+  };
+  return {
+    name: pkg.name,
+    repositoryUrl: pkg.repository.url,
+    readme: { path: 'README.md', markdown: readText(path.join(REPO_ROOT, 'README.md')) },
+    docs: readDocs(),
+    examples: readExamples(),
+  };
+}
+
+function main(): void {
+  const input = loadInput();
+  const outputs: Record<string, string> = {
+    'llms.txt': renderLlmsTxt(input),
+    'llms-full.txt': renderLlmsFull(input),
+  };
+  if (process.argv.includes('--check')) {
+    const stale = Object.entries(outputs).filter(([file, content]) => {
+      const target = path.join(REPO_ROOT, file);
+      return !fs.existsSync(target) || readText(target) !== content;
+    });
+    if (stale.length > 0) {
+      console.error(`llms.txt is stale: run npm run docs:llms (out of date: ${stale.map(([f]) => f).join(', ')})`);
+      process.exit(1);
+    }
+    return;
+  }
+  for (const [file, content] of Object.entries(outputs)) {
+    fs.writeFileSync(path.join(REPO_ROOT, file), content);
+  }
+}
+
+main();

scripts/llmsTxt.test.ts

@@ -0,0 +1,191 @@
+import { describe, it, expect } from 'vitest';
+import {
+  absoluteHref,
+  extractParagraph,
+  extractTitle,
+  firstSentence,
+  isUserDoc,
+  normalizeNewlines,
+  orderDocs,
+  removeTitle,
+  renderLlmsFull,
+  renderLlmsTxt,
+  repoBaseUrl,
+  rewriteLinks,
+  stripNoise,
+} from './llmsTxt';
+import type { LlmsInput } from './llmsTxt';
+
+const BASE = 'https://github.com/o/r';
+
+describe('repoBaseUrl', () => {
+  it('turns git remotes into a browsable https URL', () => {
+    expect(repoBaseUrl('https://github.com/o/r.git')).toBe(BASE);
+    expect(repoBaseUrl('git+https://github.com/o/r.git')).toBe(BASE);
+    expect(repoBaseUrl('git://github.com/o/r.git')).toBe(BASE);
+  });
+});
+
+describe('isUserDoc', () => {
+  it('keeps user pages and drops research, plan, api and maintainer notes', () => {
+    expect(isUserDoc('docs/installation.md')).toBe(true);
+    expect(isUserDoc('docs/research/x.md')).toBe(false);
+    expect(isUserDoc('docs/plan/tickets.md')).toBe(false);
+    expect(isUserDoc('docs/api/index.html')).toBe(false);
+    expect(isUserDoc('docs/eslint-baseline-followup.md')).toBe(false);
+    expect(isUserDoc('docs/docs-links.test.ts')).toBe(false);
+  });
+});
+
+describe('extractTitle / removeTitle', () => {
+  it('uses the first H1 and ignores # lines inside code fences', () => {
+    const md = '```sh\n# comment\n```\n\n# Real Title\n\ntext\n';
+    expect(extractTitle(md)).toBe('Real Title');
+    expect(removeTitle(md)).toBe('```sh\n# comment\n```\n\n\ntext\n');
+  });
+
+  it('throws when there is no H1', () => {
+    expect(() => extractTitle('## only h2')).toThrow(/no top-level/);
+  });
+});
+
+describe('extractParagraph / firstSentence', () => {
+  it('skips badges, comments, headings and lists to find the first prose paragraph', () => {
+    const md = [
+      '# T',
+      '',
+      '![CI](x.svg)',
+      '[![npm](a.svg)](b)',
+      '',
+      '<!-- hidden -->',
+      '',
+      '- a list',
+      '',
+      '**Bold lead.** See [the docs](./d.md)',
+      'for more. Second sentence.',
+      '',
+      'Later paragraph.',
+    ].join('\n');
+    expect(extractParagraph(md)).toBe('Bold lead. See the docs for more. Second sentence.');
+  });
+
+  it('returns an empty string when there is no prose', () => {
+    expect(extractParagraph('# T\n\n- a\n- b\n')).toBe('');
+  });
+
+  it('trims to one sentence, or strips a trailing colon when there is no full stop', () => {
+    expect(firstSentence('One. Two.')).toBe('One.');
+    expect(firstSentence('Uses `a.b` here. Next.')).toBe('Uses `a.b` here.');
+    expect(firstSentence('Run it like this:')).toBe('Run it like this');
+  });
+});
+
+describe('stripNoise', () => {
+  it('removes comments and badge lines but leaves code fences alone', () => {
+    const md = 'a\n\n![b](x)\n\n<!-- c -->\n\n```html\n<!-- keep -->\n```\n';
+    const out = stripNoise(md);
+    expect(out).not.toContain('![b]');
+    expect(out).toContain('<!-- keep -->');
+    expect(out).not.toContain('<!-- c -->');
+  });
+});
+
+describe('absoluteHref / rewriteLinks', () => {
+  it('resolves relative paths against the source file', () => {
+    expect(absoluteHref('./configuration.md', 'docs/x.md', BASE)).toBe(`${BASE}/blob/main/docs/configuration.md`);
+    expect(absoluteHref('docs/quick-start.md', 'README.md', BASE)).toBe(`${BASE}/blob/main/docs/quick-start.md`);
+    expect(absoluteHref('../examples/doc-qa/', 'docs/x.md', BASE)).toBe(`${BASE}/tree/main/examples/doc-qa`);
+    expect(absoluteHref('./a.md#sec', 'docs/x.md', BASE)).toBe(`${BASE}/blob/main/docs/a.md#sec`);
+  });
+
+  it('leaves absolute URLs and pure anchors alone', () => {
+    expect(absoluteHref('https://x.dev/a', 'README.md', BASE)).toBe('https://x.dev/a');
+    expect(absoluteHref('mailto:a@b.c', 'README.md', BASE)).toBe('mailto:a@b.c');
+    expect(absoluteHref('#top', 'README.md', BASE)).toBe('#top');
+  });
+
+  it('rewrites links in prose but not inside code fences', () => {
+    const md = 'See [cfg](./c.md).\n\n```ts\nconst s = "[x](./c.md)";\n```\n';
+    expect(rewriteLinks(md, 'docs/x.md', BASE)).toBe(
+      `See [cfg](${BASE}/blob/main/docs/c.md).\n\n\`\`\`ts\nconst s = "[x](./c.md)";\n\`\`\`\n`,
+    );
+  });
+});
+
+describe('orderDocs', () => {
+  it('puts installation, quick-start, api-overview first, then the rest alphabetically', () => {
+    const paths = ['docs/testing.md', 'docs/api-overview.md', 'docs/agent-forge.md', 'docs/quick-start.md', 'docs/installation.md'];
+    expect(orderDocs(paths.map((path) => ({ path }))).map((p) => p.path)).toEqual([
+      'docs/installation.md',
+      'docs/quick-start.md',
+      'docs/api-overview.md',
+      'docs/agent-forge.md',
+      'docs/testing.md',
+    ]);
+  });
+});
+
+describe('normalizeNewlines', () => {
+  it('converts CRLF and CR to LF', () => {
+    expect(normalizeNewlines('a\r\nb\rc')).toBe('a\nb\nc');
+  });
+});
+
+const INPUT: LlmsInput = {
+  name: '@o/pkg',
+  repositoryUrl: 'git+https://github.com/o/r.git',
+  readme: { path: 'README.md', markdown: '# Pkg\n\n![b](x)\n\nA tool. More.\n\n[Go](docs/b.md)\n' },
+  docs: [
+    { path: 'docs/deployment.md', markdown: '# Deployment\n\nShip it. Done.\n' },
+    { path: 'docs/b.md', markdown: '# B Page\n\nAbout B. See [A](./a.md).\n' },
+    { path: 'docs/installation.md', markdown: '# Installation\n\nInstall it.\n' },
+  ],
+  examples: [{ dir: 'examples/z', readme: '# z\n\nAn example. Yes.\n' }],
+};
+
+describe('renderLlmsTxt', () => {
+  it('follows the llmstxt.org layout with absolute URLs and a single trailing newline', () => {
+    expect(renderLlmsTxt(INPUT)).toBe(
+      [
+        '# @o/pkg',
+        '',
+        '> A tool. More.',
+        '',
+        '## Docs',
+        '',
+        `- [Pkg](${BASE}/blob/main/README.md): A tool.`,
+        `- [Installation](${BASE}/blob/main/docs/installation.md): Install it.`,
+        `- [B Page](${BASE}/blob/main/docs/b.md): About B.`,
+        '',
+        '## Examples',
+        '',
+        `- [z](${BASE}/blob/main/examples/z/README.md): An example.`,
+        '',
+        '## Optional',
+        '',
+        `- [Deployment](${BASE}/blob/main/docs/deployment.md): Ship it.`,
+        '',
+      ].join('\n'),
+    );
+  });
+});
+
+describe('renderLlmsFull', () => {
+  it('concatenates README first, then docs in reading order, with headers and rewritten links', () => {
+    const out = renderLlmsFull(INPUT);
+    expect(out.startsWith('# Pkg\n\nSource: README.md\n\nA tool. More.')).toBe(true);
+    expect(out).not.toContain('![b]');
+    expect(out.indexOf('Source: docs/installation.md')).toBeLessThan(out.indexOf('Source: docs/b.md'));
+    expect(out.indexOf('Source: docs/b.md')).toBeLessThan(out.indexOf('Source: docs/deployment.md'));
+    expect(out).toContain(`[A](${BASE}/blob/main/docs/a.md)`);
+    expect(out).toContain(`[Go](${BASE}/blob/main/docs/b.md)`);
+    expect(out.endsWith('\n') && !out.endsWith('\n\n')).toBe(true);
+    expect(out).not.toContain('\r');
+  });
+});
+
+describe('extractParagraph on real-world shapes', () => {
+  it('ignores table rows and blockquotes', () => {
+    expect(extractParagraph('# T\n\n| a | b |\n|---|---|\n\n> quote\n\nReal text.\n')).toBe('Real text.');
+  });
+});