DocsForge

Universal software documentation → clean Markdown for LLMs.

An open-source MCP server that finds a library's official documentation from its name alone, verifies it, harvests it, and serves it to AI agents and coding assistants — Claude Code, Cursor, Codex and any other MCP client.

Website: docsforge.vercel.app · Tools · Connect your client · Installation · Contributing

🚀 In production — feedback and contributors can support

Found a bug, a bad extraction, or something that could work better? Open an issue with as much detail as you can. Every report makes it more reliable.

Want to contribute? See CONTRIBUTING.md, or email rajpatel9408019@gmail.com with the subject DocsForge Contribution Proposal, and include:

Your GitHub — this is what I read to get a picture of your work
Your experience with crawlers, MCP tools or AI agents or any project related to it
What you'd like to work on, and any ideas you already have

Useful ground: crawling and scraping, documentation extraction, MCP tools, AI agents, backend and developer tooling. Bug fixes and documentation count as much as new crawling strategies.

— Raj Patel, creator and maintainer · Portfolio · LinkedIn · GitHub

DocsForge points at any documentation source, figures out what kind of source it is, and extracts it into tidy, LLM-ready Markdown. Feed it a docs site, an OpenAPI spec, a GitHub repo, a sitemap, or a raw Markdown file — it detects the format and handles each one appropriately.

It ships in three forms, all sharing one extraction engine:

Surface	File	What it's for
CLI	`python -m docsforge` (`docsforge/core/engine.py`)	One-shot scraping into `.md` files.
MCP server	`main.py` (`docsforge/server/mcp_server.py`)	Give any MCP client (Claude Code, Claude Desktop, your agent) live docs-fetching tools.
Web chat	`python -m docsforge.server.app` + `docsforge/server/static/`	A chat UI over any of six providers that fetches docs and answers in rendered Markdown, plus DocsStore for browsing what has been harvested.

                   ┌─────────────────┐
   CLI ───────────▶│                 │
                   │  core/engine.py │  detect → extract → Markdown
   MCP client ────▶│ tools/forge_tools│
                   │                 │
   Web chat ──────▶│                 │
                   └─────────────────┘

docsforge/tools/forge_tools.py defines each tool exactly once. docsforge/server/mcp_server.py generates the MCP surface from those definitions; docsforge/server/app.py hands the same schemas to whichever provider is selected. An MCP client and the web chat therefore run identical code, and neither can drift from the other.

main.py                 the only script in the root: starts the MCP server
docsforge/              the package
  core/                 engine (fetch/extract/crawl), resolver, languages, topics,
                        llmsfinder, versions, manifests, federation, selection,
                        passages, reasoning, observation, instrument
  store/                kb_store — Markdown files or Postgres
  tools/                forge_tools, harvest_jobs, tracing, applog
  server/               mcp_server, app (+ static/)
  lab/                  the test lab at /lab in the web chat: accounts, tests, verdicts
  providers/            one model backend per file
scripts/                measurement harnesses and live smoke drivers;
                        scripts/benchmark/ is the live suite -- every tool measured
                        over MCP against a running DocsForge, hosted or --offline
tests/                  the offline suite
benchmarks/             published benchmark runs, bench-1 onward, each ending with
                        the issues that run faced
System Files/           Architecture, PRD, Design, Workflow, Issues, Audit

Features

Automatic source detection — probes the URL (and content, when needed) to pick the right extraction strategy.
Supported sources:
- llms.txt / llms-full.txt — the LLM-native docs standard (passthrough).
- OpenAPI / Swagger (JSON or YAML) → readable API reference with endpoint tables, params, request bodies, and response codes. Local $refs are resolved and path-level parameters are applied to every operation.
- sitemap.xml → structured crawl of every listed page, including sitemap indexes.
- GitHub repos → README + all Markdown under /docs via the GitHub API.
- Generic HTML docs sites → readability-style extraction (strips nav/footer/ads, keeps main content).
- Raw Markdown / plaintext → passthrough with cleanup.
Bare-domain probing — auto-checks for llms.txt at the root before falling back to HTML.
Optional site crawling (--crawl) with same-host link following, page limits, and asset filtering.
JS rendering (--js) via Playwright, reusing a single browser across the whole run.
Single-file output (--single-file) to concatenate everything into one .md.
Provenance headers — every output file records its source URL, type, and scrape time.
Whole, checked — a site's llms.txt, generator manifest, sitemaps and the links on its own pages are read together, so a curated or stale list is never taken for the whole; dead entries and tables of contents are reported, not counted as gaps.
Any kind of site — the section is read from the site's own sidebar; pages that render client-side are taken from their Markdown copy or rendered; docsify sites are read from their Markdown files; tabs, code lines and links survive extraction intact.
Scoped asks — language for one language's edition ("langgraph for node"), topic for the part about one subject ("go for web dev"), each harvested whole and filed apart.
Durable as it goes — a page is stored before the next is fetched, so an interrupted crawl keeps what it had, and a page the store refuses costs that page rather than the whole harvest.
Concurrent fetching — pages overlap within a per-host politeness cap, so a crawl waits once instead of once per page, and returns the same pages in the same order a sequential run would.

Installation

pipx install docsforge          # the `docsforge` CLI and `docsforge-mcp` server

Or from a clone, which is what you want if you intend to change anything:

git clone https://github.com/R204570/DocsForge.git
cd DocsForge
pip install -e .                # or: pip install -r requirements.txt

Optional extras, each pulling only what it needs:

pip install -e ".[web]"         # the chat panel (FastAPI, sanitiser, renderer)
pip install -e ".[postgres]"    # store harvests in Postgres with a text index
pip install -e ".[js]"          # then: playwright install chromium
pip install -e ".[providers]"   # every model SDK; or pick one: [claude], [groq]
pip install -e ".[dev]"         # everything, plus pytest

requirements.txt is the exact, reproducible set used by CI. pyproject.toml states the version range each dependency is known to work across. Four packages are pinned outright — beautifulsoup4, markdownify, mcp and nh3 — because a float in any of them changes extraction, output or sanitising with no code change of ours.

1. CLI

python -m docsforge <URL> [options]

# A docs site (auto-detected)
python -m docsforge https://docs.stripe.com

# An OpenAPI / Swagger spec → API reference tables
python -m docsforge https://petstore3.swagger.io/api/v3/openapi.json

# A GitHub repo → README + /docs
python -m docsforge https://github.com/tiangolo/fastapi

# Crawl a docs site, up to 50 pages
python -m docsforge https://docs.example.com --crawl --max-pages 50

# JS-rendered site
python -m docsforge https://site.com --js

# Combine everything into one Markdown file
python -m docsforge https://site.com --single-file

Options

Flag	Default	Description
`-o`, `--out`	`./docs_md`	Output directory.
`--crawl`	off	Follow same-host links from the start URL.
`--max-pages`	`25`	Max pages to fetch (crawl / sitemap / repo docs).
`--js`	off	Render JavaScript with Playwright.
`--delay`	`0.4`	Seconds to wait between requests when crawling.
`--single-file`	off	Write one combined `.md` instead of per-page files.
`--force`	—	Skip detection and force a strategy: `llms_txt`, `openapi`, `sitemap`, `github`, `raw_text`, `html`.
`--allow-private`	off	Permit private/loopback hosts (see Security).
`-q`, `--quiet`	off	Suppress progress output.

As a library

from docsforge.core.engine import forge, Options

docs = forge("https://docs.example.com", Options(crawl=True, max_pages=10))
for d in docs:
    print(d.title, len(d.markdown))

2. MCP server

python main.py                       # at a terminal: the site + /mcp on http://127.0.0.1:8765
DOCSFORGE_MCP_TOKEN=… python main.py --http --host 0.0.0.0   # shared: token required

One command, two callers. Typed at a terminal, main.py serves HTTP: the public site — / (what DocsForge is), /tools (the reference), /connect (client setup) — plus /health and the MCP endpoint at /mcp. Launched by an MCP client, which always talks over pipes, the same command speaks MCP on stdin/stdout instead; --http and --stdio force either (a service with no terminal, like the Containerfile, passes --http). Only /mcp is gated: clients send Authorization: Bearer <DOCSFORGE_MCP_TOKEN>, and without it the server answers 401. Binding a non-loopback address with no token is refused outright (DOCSFORGE_MCP_INSECURE=1 overrides, for a network you own).

OAuth, for clients that cannot send a token. ChatGPT's connectors authenticate with OAuth or not at all, so with a token configured the server is also an OAuth 2.1 authorization server (docsforge/server/oauth.py) — discovery at /.well-known/…, dynamic client registration, PKCE — whose login is the DocsForge token: the client opens a DocsForge sign-in page, the person pastes the token once, and the client walks away with an access token that /mcp accepts beside the bearer header. Everything the flow needs to remember is carried in signed, expiring blobs rather than stored, which is what lets it run on a stateless host and off the database; rotating the token revokes every client and token at once. Discovery names absolute URLs, so the server has to know where it lives: DOCSFORGE_PUBLIC_URL, or on Vercel the production domain the platform provides. Without an HTTPS (or loopback) address OAuth is not offered and the bearer token still works.

The web chat in docsforge/server/app.py is not part of this process and nothing routes to it. See Deploying.

Register with Claude Code:

claude mcp add docsforge -- python /absolute/path/to/DocsForge/main.py

Or in an MCP client config file:

{
  "mcpServers": {
    "docsforge": {
      "command": "python",
      "args": ["E:/DocsForge/main.py"]
    }
  }
}

Tools exposed

Tool	Arguments	Returns
`detect_source_type`	`url`	Which strategy the URL would use — a cheap probe.
`fetch_docs`	`url`, `crawl`, `max_pages`, `js`, `force`	The extracted Markdown.
`save_docs`	`url`, `out_dir`, `crawl`, `max_pages`, `js`, `force`, `single_file`	Paths written to disk.
`harvest_docs`	`url`, `name`, `max_pages`, `js`, `scope`, `version`, `topic`	Learns a whole technology from one URL and stores it. Unlimited by default. Returns a summary.
`learn_technology`	`name`, `language`, `topic`, `version`, `ecosystem`, `max_pages`, `js`, `intent`, `corpora`, `strict`	Learns a library from its name alone — no URL. Resolves, verifies, harvests, stores. `language` takes one language's edition ("langgraph for node"), `topic` the part about one subject ("go for web dev").
`harvest_status`	`harvest`, `wait`	How a harvest is getting on — phase, pages of how many, elapsed — and how it ended. Omit `harvest` for every harvest in flight; `wait` lets it run up to the deadline before answering.
`find_docs`	`name`, `ecosystem`, `language`	Where a name resolves to, with evidence. Harvests nothing.
`scan_project`	`path`, `unknown_only`	A project's dependencies, versions, and which are documented.
`search_knowledge_base`	`query`, `technology`, `version`, `limit`	Ranked search across every stored page at once.
`list_knowledge_base`	—	What has already been harvested, and which versions of each.
`read_knowledge_base`	`name`, `section`, `version`	Reads stored docs back, optionally only matching pages. Defaults to the current release (the one `list_knowledge_base` lists first).
`forget_resolution`	`name`	Forgets where a name previously resolved to, so the next lookup starts over. Deletes no documentation. Omit `name` to clear all.
`forget_selection`	`name`	Forgets which corpora were chosen for a technology, so it asks again. Deletes no documentation.

learn_technology also takes intent, corpora and strict; search_knowledge_base takes kind. See Serving a purpose below.

forget_documentation also exists but is opt-in behind DOCSFORGE_ALLOW_DELETE=1: deleting a harvest is the one irreversible thing here, and a model that has just mis-resolved a name is not the caller you want holding that lever.

Results handed to a model are capped at DOCSFORGE_MAX_CHARS (60k default) with an explicit truncation marker.

3. Web chat

cp .env.example .env      # add a key for ONE provider — or none at all
python -m docsforge.server.app   # http://127.0.0.1:8000

Three surfaces, one window: the chat at /, DocsStore at /library, and everything that would otherwise need explaining at /docs. Charcoal on black with a single accent; nothing on the home screen but the input.

The sidebar opens with the panel button or Ctrl+B, and holds your past conversations. They live in your browser — forty most recent, Markdown only, nothing uploaded — so New chat no longer throws work away. The server keeps its own record of each turn (the question, the tool calls and what they returned, the answer) for the test lab, in lab_data/ on this machine; DOCSFORGE_LAB_RECORD=0 turns that off.

Every tool call the model makes is listed above the answer it produced — what was fetched, what kind of source it was, and how much came back — so an answer built from a page that was actually read does not look like one that was not. Click any of those rows to open its execution trace: the exact input the tool was given, the stages it went through, and what each returned, nested and arriving live rather than assembled at the end. A three-minute harvest is watchable for three minutes, a failure is attached to the operation that caused it, and where a page had to be clipped the row says how much it left out. There are no invented percentages — a progress figure appears only where there is a real denominator to divide by, such as a manifest's own page count. A harvest too big to finish inside the request keeps going in the background, and a card in the corner shows what it is doing — the phase, the page count, and how long it has been going — on the chat and in DocsStore alike. Where the source states how many pages it has, the card fills a bar; where it does not — a crawl discovers its frontier as it walks it — you get elapsed time and a page count and no invented percentage. DocsStore refreshes itself the moment one lands. The same transitions go to logs/docsforge.log and to the console, so a harvest is watchable without a browser open.

Each answer carries Copy / Download .md / Edit, because the Markdown is something you keep, not just something you read. The send button becomes a stop button while an answer streams, and stopping keeps what already arrived.

Providers

Pick one from the model control at the top right; unconfigured ones are disabled and say why. The choice rides on each request, so when one provider hits its daily cap you switch and keep going mid-conversation.

Provider	Key	Default model	Notes
Claude Code	none	your CLI default	Runs the local `claude` CLI against your existing login. No API key and no per-token bill.
Ollama	none	best installed	Models running on your own machine. No key, no quota, works offline.
Claude	`ANTHROPIC_API_KEY`	`claude-opus-5`	Strongest on long documents and tool use.
Groq	`GROQ_API_KEY`	`gpt-oss-120b`	Fast and cheap; free tier caps at 100k tokens/day.
ChatGPT	`OPENAI_API_KEY`	`gpt-4.1`	Billed per token, no free tier.
Gemini	`GEMINI_API_KEY`	`gemini-2.5-flash`	Large free tier.

Each provider is one file in docsforge/providers/, and they all speak the same small event stream (text, tool_start, tool_end, notice) so app.py never learns which one is running. What differs is the tool-calling shape, which is why each owns its own loop:

groq.py and chatgpt.py share _openai_shape.py — tool_calls deltas stitched by index, answered with role: "tool" messages.
claude.py — tool_use blocks answered by tool_result blocks in one user turn. Sends no temperature/top_p/top_k: they were removed on Opus 5 and return a 400. Refusal fallbacks are on by default (ANTHROPIC_FALLBACKS=off).
gemini.py — functionCall / functionResponse parts, automatic function calling disabled so the JSON-Schema tool definitions stay shared.
ollama.py — reuses the same OpenAI-shaped loop (Ollama serves an OpenAI-compatible endpoint), but probes the daemon instead of checking for a key, and auto-picks the best tool-capable model you have pulled. Models that cannot call tools — embeddings, phi3, vision-only — are filtered out, since one that silently answers from memory looks like DocsForge being broken.
claudecode.py — the odd one out: it shells out to the claude CLI with DocsForge's own MCP server attached, so the tools run out of process over real MCP. --strict-mcp-config keeps your other MCP servers out of the session.

Adding a provider means one file and one line in docsforge/providers/__init__.py.

Configuration

Everything is optional; see .env.example for the full list.

Variable	Default	Purpose
`DOCSFORGE_PROVIDER`	first configured	Which provider to start on.
`<NAME>_MODEL`	per provider	Override a provider's model, e.g. `CLAUDE_MODEL`.
`ANTHROPIC_FALLBACKS`	`default`	`off` disables Claude's server-side refusal fallbacks.
`OPENAI_BASE_URL`	—	Point ChatGPT at Azure or an OpenAI-compatible host.
`OLLAMA_HOST`	`http://127.0.0.1:11434`	Where the Ollama daemon listens.
`OLLAMA_MODEL`	auto	Pin a local model; otherwise the best tool-capable one installed.
`GITHUB_TOKEN`	—	Raises the GitHub API rate limit.
`DOCSFORGE_MAX_CHARS`	`200000`	Largest tool result returned to a model. Not a storage limit — files and the knowledge base keep every page whole. Raise it if your provider's context window allows; the cap exists because a result that overflows the window fails the turn rather than truncating.
`DOCSFORGE_OUT_ROOT`	`./docs_md`	Directory `save_docs` may write into.
`DOCSFORGE_ALLOW_PRIVATE`	unset	Allow fetching private/loopback addresses.
`DOCSFORGE_ALLOW_DELETE`	unset	Let the model delete stored documentation. Off by default — see below.
`DOCSFORGE_REASONING`	`off`	Allow a bounded number of model calls at four decision points — see below.
`DOCSFORGE_LAB`	on	`0` removes the test lab and its routes.
`DOCSFORGE_LAB_RECORD`	on	`0` keeps the lab but stops recording chat turns and tool calls.
`DOCSFORGE_LAB_DIR`	`./lab_data`	The lab's database and harvest-test runs (beside the code, like `knowledge_base/`).
`DOCSFORGE_LAB_REMOTE`	unset	Let non-loopback clients reach `/lab`. Only on a network you own.
`DOCSFORGE_LAB_WORKERS` / `_HARVESTS`	`2` / `1`	Resolution and content tests at once; harvest tests at once.

The test lab

python -m docsforge.server.app      # then open http://127.0.0.1:8000/lab

A signed-in panel for testing DocsForge by hand while the machine does the running. It is linked from nowhere, answers this machine only (a request from anywhere else gets a 404), and is never part of main.py's public server. The first visit makes two accounts:

tester — lands on the test bench. Three tests, each run the way DocsForge runs them for a model:
- Resolve a name — find_docs with the cache off, then the page it resolved to fetched with fetch_docs, so you check the address and what is there. Give where it should land (zod.dev) and an automatic check runs too.
- Check a page — detect_source_type and fetch_docs on one URL, the Markdown beside a link to the live page, measured for collapsed code, UI chrome, permalink marks and relative links.
- Harvest — learn_technology (a name) or harvest_docs (a URL) in a process of its own, with the store, caches and harvest records under lab_data/runs/ whatever .env says, so a test harvest never lands in DocsStore. Its trace streams live; every stored page is listed and can be read in place.
Or a batch: a list, one test per line, run by itself — the held-out rounds and field-test sites are ready-made lists. Every test shows each tool it ran, the arguments, what it returned and the stages underneath. When a test finishes it asks for a verdict: a few questions for that kind of test, issue tags, the right URL if it was wrong, notes. The review queue holds every finished test nobody has judged. A finished resolution can be carried on as a content check or a harvest in one click.
admin — lands on the overview: technologies and pages stored, tests run, accuracy as testers judged it (with the automatic check's figure under it, never in place of it), tool calls per day and by tool, harvests in flight, who has been testing, the most reported problems. Plus the verdicts inbox (open → triaged → fixed, with a note the tester sees), exports as Markdown, JSON or scripts/heldout.py cases, setup (providers, store, paths, every DOCSFORGE_* setting — secrets shown as set, never echoed), accounts, and the logs.

Both see AI activity: every chat turn — the question, each tool call with its arguments and full output, the answer — and every tool call on its own, filterable by tool, result and where it came from, each open to a verdict.

Passwords are scrypt-hashed, sessions are HttpOnly SameSite=Strict cookies stored only as hashes, five wrong passwords lock a name for ten minutes, and nothing changes state without the panel's own request header. A lost admin password is reset from a terminal: python -m docsforge.lab reset NAME.

Removing a harvest

A harvest can be wrong: the wrong project, a partial copy, a table of contents stored as though it were the documentation. Three ways to take one back out.

In DocsStore. Open a technology at /library and each version has a delete control, with a second one below for the whole technology. It asks once before acting.

From the command line, which is the one to reach for when clearing several:

python -m docsforge --forget pydantic@1.10     # one version
python -m docsforge --forget astro             # every version of it
python -m docsforge --forget a --forget b --yes  # several, no prompt

It prints what it is about to remove — pages and characters — and does nothing unless you type yes or pass --yes.

Over HTTP, which is what the UI uses:

curl -X DELETE http://127.0.0.1:8000/api/library/astro
curl -X DELETE http://127.0.0.1:8000/api/library/pydantic/1.10

Can the model delete? Not unless you say so. Deleting is the one irreversible thing DocsForge does, and a model that has just mis-resolved a name is the last caller who should hold that lever — 703 pages of Effect are one confident hallucination away. Set DOCSFORGE_ALLOW_DELETE=1 to add a forget_documentation tool to the surface. Note that you do not need it to re-harvest: harvesting the same name again replaces that version on its own.

Endpoints

Method	Path	Purpose
`GET`	`/`	The chat UI.
`GET`	`/docs`	How it all works. (Swagger UI is disabled so this URL is the product's; the schema is still at `/openapi.json`.)
`GET`	`/library`	DocsStore: everything harvested so far.
`GET`	`/api/config`	Provider catalog, current default, tool list.
`POST`	`/api/chat`	SSE stream: `token`, `tool`, `notice`, `done`, `error`. Takes an optional `provider`.
`POST`	`/api/render`	Markdown → sanitized HTML.
`GET`	`/api/library`	One page of technologies. `?page=N&q=filter`.
`GET`	`/api/library/{tech}`	Every stored version of one technology.
`GET`	`/api/library/{tech}/{version}`	That version's page index.
`GET`	`/api/library/{tech}/{version}/page/{n}`	One page, as Markdown and sanitized HTML.
`GET`	`/api/library-search`	Ranked search. `?q=&tech=&version=&limit=`.
`GET`	`/lab`	The test lab. Loopback only; its API is under `/api/lab/`, signed in.

The server is stateless — the browser holds the conversation and posts it back each turn.

DocsStore

/library is the box: every documentation set that has ever been harvested, three levels deep.

effect                  a divider in the box
  v3, v2                every crawled version of it
    703 pages           the pages of the version you opened

The technology list is paged (12 at a time) rather than loaded whole, because the box grows every time anyone harvests and there is no natural ceiling. Opening a divider lists every version with its page count, size, harvest date and strategy; opening a version puts its page index beside the page you are reading. The search box inside a version runs the ranked full-text search over that version only, and marks what matched.

Every view is addressable, so a link goes exactly where you meant: /library#/effect/v3/41.

Right-click a technology to open or to delete it outright — every version of it, in one action. Removing a language's documentation one version at a time is not a route once it has eight of them. The delete arms before it fires: it asks once, and needs a second click.

The menu bar's right-hand corner names the backend that answered. That is not decoration: a Postgres box ranks search and a file box cannot, and reading unranked results while believing they are ranked is worse than knowing.

Learning something by name

The caller usually does not have a documentation URL — it is a model that has just met a library it does not know, so where the docs live is exactly the knowledge it is missing. It does not need one:

learn_technology(name="effect")
learn_technology(name="pydantic", version="1.10")
  -> resolved to https://docs.pydantic.dev (names 'pydantic' 214 times)
  -> harvested 85 pages, stored as pydantic 1.10

It asks the package registries — npm, PyPI, crates.io, all keyless — where the library declares its documentation, probes the site for an llms.txt or a docs root, and then confirms the page actually documents that package before harvesting anything.

How a name is resolved

A bounded ladder, cheapest lap first. Every lap ends at the same identity gate, so reaching further never means believing more:

Lap	What it tries
memory	A resolution seen in the last 30 days, per language asked for. Costs zero requests. Refusals are re-tried after 7 days, because a site can add the evidence later.
language	When the name is a language or runtime — `go`, `python`, `rust`, `node` — its own manual (`go.dev/doc/`), verified like anything else. A language is not a package: asked for `go`, a registry offered a Rust crate.
domain	`{name}.dev` / `.io` / `.org` / `.com`, and the docs root beneath whichever answers.
registries	Exact-name lookup in npm, PyPI and crates.io. A winning GitHub repository gives way to the documentation site it declares as its homepage — or, failing that, to `docs.rs` / `pkg.go.dev`, or to the docs its README names. A package that is a language binding (`docker-py`, `redis-py`) never unseats the project's own site, and a package index's page (`pypi.org/project/…`) is never documentation.
name shapes	For multi-word names: `opentelemetry.io`, `airflow.apache.org`, `tanstack.com/query`, `docs.spring.io/spring-boot`.
search	npm and crates.io fuzzy search — only the registry you named, if you named one — then `DOCSFORGE_SEARCH` if you configure one. Never a search engine's HTML.
what the name usually means	The most-starred GitHub repository named exactly that — when it is the search's top result — and the site it declares, verified like anything else. Above 2,000 stars it overrules a same-named package the ladder reached: `redis` is redis.io, not the Python client; `helm` is helm.sh, not a Rust crate. Skipped when you name a registry.
front page → docs	An answer that is a site's front page gives way to the documentation it links (`mypy-lang.org` → `mypy.readthedocs.io`, `symfony.com` → `/doc`), and a hub gives way to the manual beside it (`www.sympy.org/en/docs.html` → `docs.sympy.org`, `rubyonrails.org/docs` → `guides.rubyonrails.org`) — but never to a mirror of the same site, a preview (`alpha.`), a stub, or an API reference generated from the code.

One language's edition. Many projects document each language separately — LangGraph and LangChain at /oss/python/… and /oss/javascript/…, Playwright at playwright.dev/docs/ (Node) and /python/docs/. With language=, the name is resolved as usual and then the asked language's edition is found beside it: a language segment swapped, a prefix added, a host label swapped, or the section a many-language front page links to — read from HTML or from an llms.txt, one hop away on a hub page (grpc.io/docs/languages/), on a sibling host (docs.temporal.io), or inside a compound name (/client/python-api/). A site that answers any path with a page (a login screen) cannot vouch for an edition by its address alone. It is taken only when the page's own code shows that language, or the site files it under that language's name, and it is stored as <name>-<language>:

learn_technology(name="langgraph", language="node")
  -> Resolved langgraph (javascript) to docs.langchain.com/oss/javascript/langgraph/overview
     docs.langchain.com/oss/python/langgraph/overview documents langgraph for another
     language; the site publishes the javascript edition at …, and that is what was asked for.
  -> Harvested langgraph-javascript — 43 pages via llms-full.txt

The whole ladder is capped at 40 requests and 20 seconds, so a name that does not exist refuses promptly instead of wandering.

forget_resolution(name="polars")   # resolved to the wrong site? start over
forget_resolution()                # clear every remembered lookup

Verification is the point

A resolver that is merely usually right is a slower guess. Names collide: the npm package fastapi is unrelated to the Python framework, and the effect crate is not the TypeScript library. Harvesting an unverified page stores the wrong project and then answers from it confidently.

So a page must name the package repeatedly before it is trusted — one passing mention is noise. If nothing verifies, DocsForge reports that and asks for a URL instead of picking something plausible. find_docs shows the candidates and the evidence without fetching the documentation.

Two rules were added after being caught live, and both are benchmarked on every run:

A candidate is judged against its own registry. The same word exists on npm, PyPI and crates.io, on different projects. click on PyPI used to be tested against npm's ecosystem and stored under npm's version, 0.1.0; it is filed under PyPI's 8.5.0 now, and the ecosystem and release in the answer are the winner's own.
A candidate that could not be read blocks anything weaker. With click.palletsprojects.com answering HTTP 429, click once resolved to a Kubernetes CLI's wiki and requests to a Rust crate — both verified, because both document a project of that name. A registry-nominated page that could not be fetched for a transient reason is now unexamined, and nothing weaker wins in its place; the answer names the page and says to try later, and that refusal is never cached.

Reading the project

scan_project reads package.json, pyproject.toml, requirements.txt, Cargo.toml and go.mod, and reports each dependency, its version, and whether it is documented here yet.

The manifest is the only place the correct version can be read from. A bare name cannot tell you this project is pinned to Pydantic 1.10, and 1.10 and 2.11 contradict each other — which is what the versioned store exists to prevent, and what a name-only lookup would defeat.

Names are matched, not memorised

Effect.ts, effect-ts and effect all reach the same stored copy, and @scope/pkg files under pkg. Everything is stored under one canonical name, so a library cannot end up saved twice under two spellings. Genuinely ambiguous names are refused rather than guessed.

Learning a whole technology

fetch_docs answers "extract this URL". harvest_docs answers "extract this technology" — the question you actually have when a model does not know a stack:

harvest_docs(url="https://www.effect.website/docs/v3/getting-started/introduction/")
  -> 200 pages, 2,668,643 characters, stored as knowledge_base/effect.md

Give it any page of a docs site and it finds the rest, best strategy first:

llms.txt / llms-full.txt — the site already published itself for machines. The files nearest the docs section are asked for before the origin's (an origin's is often about the company: resend.com/llms-full.txt is 8 KB of product overview, /docs/llms-full.txt is 2.2 MB of documentation). A dump is cut into the pages it says it holds, each under its own URL. A file is read for what it is, not what it is called: a short one that only points at others (threejs.org/llms.txt) is followed, and an llms-full.txt that is really a list of pages in URL: / Markdown: fields (Docker's) is fetched page by page.
The site's own lists — its generator's manifest (Sphinx objects.inv, MkDocs search index) and its sitemaps (XML, plain text or gzipped, every one robots.txt declares, the section's own first), read together.
A crawl, seeded with everything those lists name and following every link inside the section — so a stale sitemap is not a ceiling — or from the start page alone when nothing is listed.

First is not only. A published llms.txt is then checked against the site's other lists, and against the links on its own pages: what they name in the section and the file left out is fetched too. Measured, llms.txt files are often curated subsets — Angular's lists 85 pages, AWS's two of the Lambda guide's hundreds, GitHub's seven of Actions — and each used to be stored as complete.

The section is the site's. The boundary is read from the start page's own sidebar, after following any redirect, not guessed from the URL's shape: angular.dev/overview is a page, not a folder called overview; docs.github.com/en/actions is Actions, not all of GitHub's docs. And a section that turns out to be one page is not the documentation: htmx.org/docs/ is one long page and htmx's docs are the site, so the harvest widens one level — once, articles and marketing held out — and says so in the answer. A section you name is never widened.

JavaScript sites. A page with no readable HTML is taken from its Markdown copy when it declares one (<link rel="alternate" type="text/markdown"> — Apple, Cloudflare, Mintlify and Fern do), and otherwise rendered once in a browser if it ships the scripts that draw it; a site that is mostly rendered switches the rest of the crawl to rendering. Rendering waits for the content, not for the network to go quiet, skips images and fonts, and opens collapsed sidebar sections so the links behind them are found. docsify sites are read from their _sidebar.md and each route's Markdown file directly.

Clean pages. Every tab of a tab set is kept under its label, every code line on its own line with the block's language on the fence, every link made absolute; heading permalinks, copy buttons and "Edit this page" are dropped.

Everything lands as one Markdown file with a contents index, and read_knowledge_base(name, section=...) reads it back so a stored technology is never re-scraped to answer a question. section matches page titles first and falls back to searching the text, so a specific question pulls the handful of relevant pages instead of a whole manual.

How much it crawls

harvest_docs has no page limit by default — it keeps going until the documentation section is exhausted. A page count is a guess at how big someone else's manual is; the scope prefix above is the boundary that actually means something. The Effect v3 docs come to 703 pages, and nothing in DocsForge knew that in advance.

Pass max_pages=N only to deliberately cut a harvest short. fetch_docs keeps its own 200-page ceiling, so a single page fetch can never turn into an open-ended crawl by accident.

A long harvest does not block

The Effect v3 docs take around twelve minutes to harvest. Every MCP client gives up long before that, so learn_technology used to look broken on exactly the technologies it was most needed for.

It now waits up to 25 seconds (DOCSFORGE_HARVEST_DEADLINE) and then hands back a harvest id while the crawl carries on in the background:

**Learning effect from https://effect.website/docs/ — still running.** Harvest id `effect-1`.

Currently harvesting 118/703 pages, 25s elapsed.

harvest_status(harvest="effect-1") reports how it is getting on — phase, pages of how many, elapsed time, and once it ends whether it stored, failed or stopped — and wait=20 lets one call watch it finish. Every running status also says, in so many words, that the pages are landing in DocsForge's own store rather than in the model's context: a model told "still running" tends to decide the harvest is now its job — to fetch the pages itself, or to cut the harvest short — and neither is wanted. list_knowledge_base reports every harvest in flight too, and any that failed — a background failure leaves no other trace, so it is surfaced there rather than lost. Anything that finishes inside the deadline returns exactly what it always returned; short harvests are unaffected.

Jobs live in the server process and are not persisted. The durable record of a harvest is the knowledge-base entry it writes, so a harvest interrupted by a restart simply did not happen.

A short harvest says so

When you do set a limit and hit it, that is reported — in the harvest result, in list_knowledge_base, and again every time the content is read:

**INCOMPLETE — stopped at the 200-page limit, 400+ pages still queued.**

That matters more than the limit itself: a third of a manual that looks whole produces confident, wrong answers.

Versions are kept apart

A library's v2 and v3 docs contradict each other, and a model handed both will quote the wrong one. So a harvest is filed under a version, not just a name, and re-harvesting one version leaves the others alone:

pydantic
  2.11   85 pages   1.2 MB   harvested 2026-08-14
  1.10   24 pages   198 KB   harvested 2026-08-14

The label comes from the URL — /docs/v3/, /docs/validation/2.11/ — and is then checked against what came back. A site publishes one llms.txt for its current release, so if the harvest served that instead of crawling the version you asked for, the label falls back to the harvest date rather than claiming a precision the content does not have. Sites that publish one version at a time are always dated: the date says which snapshot this is.

read_knowledge_base(name, version=...) picks one; without it you get the most recently harvested.

For the same reason the crawl scope follows the version down the path. Pydantic keeps versions at /docs/validation/2.11/, and stopping at /docs/ there crawls every version of the manual at once and calls the result one harvest.

Where harvested docs live

Two backends, chosen automatically:

	When	What you get
Markdown files	the default	`knowledge_base/<tech>/<version>.md`, one file per version, plus `index.json`. Zero setup, and the file is a deliverable you can hand to anyone.
PostgreSQL	`DOCSFORGE_DB` set and reachable	A row per page with a GIN-indexed `tsvector`. Section lookups are ranked and stay fast as the store grows, and several DocsForge instances can share one store.

DOCSFORGE_DB=postgresql://postgres:password@127.0.0.1:5432/DocsForge

The schema is created on first use. If the database is unreachable DocsForge falls back to files rather than losing a harvest — losing ten minutes of crawling to a database outage would be the worse failure.

The file path is anchored to the package directory rather than the working directory: keying it off cwd meant launching the app from elsewhere silently produced a second, empty knowledge base. Override with DOCSFORGE_KB_ROOT. Both knowledge_base/ and .env are gitignored.

Postgres is what makes section= worth using. The file backend answers it by regex over one large string, which cannot rank; Postgres ranks, with page titles weighted above body text. Over 703 harvested Effect pages (6.3 MB):

'error handling'                  0.20s   1 page   (by title)
'retry with exponential backoff'  0.11s   6 pages  (by content, ranked)

Already have a file store? scripts/migrate_kb.py reads the combined Markdown back into Postgres, so a site that took ten minutes to crawl is not crawled again.

To read what is stored, open DocsStore at /library.

Why the crawl is scoped

Documentation shares a domain with marketing, blogs and changelogs. Crawling by host from an Effect docs page reached /podcast and /community-hub within four pages, and those pages then dominated the truncated result the model saw — so it summarised a podcast feed instead of the library.

harvest_docs therefore stays inside the documentation root of the start URL (/docs/v3/getting-started/introduction/ -> /docs/v3/), which took that same crawl from 7/12 relevant pages to 200/200. Pass scope="host" for the old behaviour, or a literal prefix like scope="/docs/v3/" to pin it exactly.

Measuring the crawler

docsforge/core/instrument.py measures what extraction actually does — which CONTENT selector won on each page, how often none did and <body> was stored whole, how many distinct templates a site really has, how much of the "documentation" is link text, and how many pages are JS shells. It decides nothing and no shipping module imports it; a test enforces both.

scripts/measure.py drives it across real technologies and writes the numbers out:

python scripts/measure.py             # the built-in 20, ~40 pages each
python scripts/measure.py fastapi --pages 6   # one technology, quickly

Results land in measurements/ as JSON plus a readable table, and the run is resumable. This exists because every adaptive rule the crawler runs is a threshold over a number, and thresholds picked before the numbers are guesses.

The field test: real sites, every generator

scripts/fieldtest.py harvests 53 real documentation sites chosen for the generator that built them — Docusaurus, MkDocs, Sphinx, VitePress, Starlight, Nextra, Mintlify, GitBook, ReadMe, mdBook, Hugo, Antora, docsify, rustdoc, godoc, and hand-built React, Next and SvelteKit sites, Apple's and Microsoft's among them — one subprocess per site, and measures every page it stored: strategy, coverage claim, code blocks with collapsed lines, relative links, UI chrome, permalink marks, thin and untitled pages. Every page is written out so a number can be checked by reading.

python scripts/fieldtest.py                         # all 53, 15 pages each, ~5 minutes
python scripts/fieldtest.py --pages 0 docusaurus flask go-dev   # uncapped, chosen sites
python scripts/fieldtest.py --list

The first run (2026-09-24) found a dozen ways a real site defeated the harvester — one-page harvests of Next.js, Laravel, Angular and the Rust book, a company overview stored as Resend's docs, a Rust crate resolved for langgraph on npm, seven-minute timeouts reading AWS's sitemap index — each now a test in tests/test_realworld.py and an entry in System Files/Issues.md.

On the final build (2026-09-25) all 53 harvest without an error, 17 of them proven complete, and the residue the field test counts — collapsed code, chrome lines, permalink marks, dead relative links — fell from 18,441 lines to 460 against the first full run, with the same 52.7 million characters stored.

Held-out accuracy: names nobody tuned for

The field test is where failures were found and fixed, so its numbers are the ones the code was shaped by. scripts/heldout.py measures the other thing: technologies chosen, and their right documentation written down, before the first run. Resolution from the name alone with the cache off; then a 40-page harvest of the answer, every stored page measured. Three rounds of 53–57 names each — JavaScript, Python, Rust, Go, and tools like Docker, Kafka and Helm — plus language editions ("gRPC for Go", "Sentry for Python", "Temporal for TypeScript").

python scripts/heldout.py --set round3              # 56 names, 40 pages each
python scripts/heldout.py --set round1 --resolve-only

A round stops being unseen the moment its failures are worked on, so each is reported twice: its first pass, on the build that had never met it, and the final build.

round	first pass, resolution	final: resolution	final: harvest	final: pages clean
1 (53 names)	33/53 (62%)	52/53 (98%)	51/52 (98%)	99%
2 (57, written after round 1's fixes)	47/57 (82%)	57/57 (100%)	52/57 (91%)	99%
3 (56, written after round 2's fixes)	51/56 (91%)	53/56 (95%)	50/53 (94%)	98%

Round 3's first pass — 91% of fresh names resolved to their official documentation — is the honest figure for a name nobody has looked at. The harvest column is strict: a harvest "passes" at ten pages or everything listed, so five of the nine misses are sites that are complete in fewer pages (esbuild is nine pages; Koa's and Ramda's manuals are one page each — 36,870 characters and 56 code blocks for Koa). What is left, named: socket.io is unreachable from the network this was measured on; tonic is not found by GitHub's search under its own name; sqlx is two well-known projects (Go and Rust — ask with language=); Selenium documents every language as tabs on one set of pages, so there is no separate Python edition to switch to; Ember's guides and egui's site render only in a browser.

Measuring whether any of it helps

scripts/measure_answers.py asks the only question that matters: does a harvested corpus change what a model can answer? The same questions are put three ways — the model alone, the model with the documentation retrieved for it, and the model given DocsForge's tools and left to use them.

python scripts/measure_answers.py --provider ollama --model llama3.2:latest
python scripts/measure_answers.py --phases closed,passages --out answers.json

Every expected answer is checked against the stored corpus before a question is asked, so a fixture whose answer is not in the documentation fails the run rather than quietly measuring nothing. Grading is a string match on an exact identifier, not one model judging another.

Measured 2026-09-10, twelve questions across six technologies:

phase             all   unknown tech   known tech
closed           0/12            0/7          0/5     llama3.2 (3B), alone
passages         8/12            6/7          2/5     with the corpus retrieved
tools            1/12            1/7          0/5     llama3.2 driving the tools
tools             3/4            3/4            -     qwen3.5 (9B), 4 questions

Closed-book the model answered none of the twelve and said so only once — useEffect for Effect, agentkit for google-adk, hydrate for model_validate. Confident inventions, which is the failure this exists to prevent. With the documentation in front of it: 6 of 7 on technologies it did not know. A model too small to call tools cannot reach the corpus by itself; one that can, does.

Measuring it live

The offline tests ask whether the code does what its fixtures expect; the fixtures tend to agree with whatever assumption wrote them. scripts/benchmark/ asks a running DocsForge what it actually answers: every tool, over MCP, every reply read and every call timed — 76 cases across transport, detection, extraction, the SSRF guard, resolution, the store, concurrency, and a technology harvested whole.

python -m scripts.benchmark.run --offline                    # the whole thing on this machine, ~5 minutes
python -m scripts.benchmark.run --offline --reset --publish  # a fresh database and caches, published as benchmarks/bench-<next>/
python -m scripts.benchmark.run                              # the read-only suites against the hosted server

--offline creates a docsforge database on the local Postgres, starts main.py --http on loopback with every DocsForge variable pointed at an offline value so .env never reaches it, refuses any database that is not on this machine, and lets a harvest run to the last page. Published runs live in benchmarks/, one folder each, ending with the issues that run faced and what each turned out to be. The first, bench-1: 68 pass, 1 fail, 1 error, 2 known gaps, click harvested whole under PyPI's 8.5.0 in 36 seconds.

The unpublished runs that built the suite found a crawl returning one page of forty, a search answering a query no page contained, a fonts stylesheet offered as documentation, a rate limit reported as empty pages, and a resolver that stored a Kubernetes CLI under click. Each is a regression test now.

Serving a purpose

A technology of any size documents itself in more than one place: a manual, a specification, a generated API reference, often on different hosts. DocsForge finds those from the link evidence it gathers while crawling, confirms each new host through the same identity gate it uses for resolution, and classifies them by shape (how to fetch it) and kind (what it is for).

intent then decides which enter scope:

Intent	Takes	Leaves
`resolve-import`	`api`, `sdk`	guides, cookbooks, changelogs
`implement`	`api`, plus guides and examples	changelogs, meta
`learn`	`guide`, plus language and spec	changelogs, meta
`operate`	`operations`, plus guides and API	meta
`reference` (default)	everything except `meta`	—

Two rules keep this honest:

Corpora left out are declared, never dropped. Each is listed with its size and marked not requested.
Choosing a corpus is scope; dropping pages inside one is filtering, and is forbidden. A selected corpus is harvested whole or reported partial.

When several corpora could be what you meant, it does not guess — it returns NEEDS SELECTION with the options, and remembers your answer so the question is asked once. forget_selection clears it.

learn_technology(name="stripe", intent="resolve-import", strict=True)

strict=True adds usable_for_planning, which is true only when every corpus your intent requires came back complete. It is the figure a downstream system can refuse to act on.

One subject, whole

topic keeps a harvest to what the caller is working on, without handing back shards:

learn_technology(name="go", topic="web development")
# -> 21 pages: Writing Web Applications (and its code), the RESTful API tutorial,
#    the JSON tutorial, plus getting started, install and Effective Go.
#    40 pages judged outside the topic, listed by section: /doc/database/ (9),
#    /doc/modules/ (7), ... "Ask again with a broader topic, or none, to take those too."

Every page is judged by its title, address, headings and prose against the topic's words, widened by a small stated vocabulary (for "web": http, server, handler, router, middleware, template, websocket… weighed above words that merely travel with it, like json or request). Pages that introduce the technology are always kept. A topic crawl follows only the pages it kept, so it stays on the subject instead of fetching the whole site to discard most of it. The harvest is filed as <name>-<topic>, so it is never mistaken for the whole manual, and the answer says which terms it used and what it left out.

Answering from what is stored

search_knowledge_base returns passages, not pages. Results are chunked on h2/h3 and ranked as sections, each carrying its heading path so you can cite it — Error Handling > Retrying rather than "somewhere on the retry page". One page of a generated API reference can be 20,000 tokens spent answering a one-line question.

search_knowledge_base(query="exponential backoff", kind="api")

kind keeps tutorial prose out of an answer that wants a signature. Relevance is applied at read time only: what is stored stays whole, and only what is handed back is narrowed.

Reasoning, if you want it

Four moments decide whether a harvest is correct, and all four are settled by arithmetic at exactly the point the arithmetic is weakest: a template none of the nine selectors recognise, a corpus whose kind the URL does not state, a new host that may be a different project sharing a name, and a page answering 200 while rendering an error.

DOCSFORGE_REASONING=on lets a model settle those instead — under a budget that is the whole point of the feature:

Twelve calls per harvest. A hard cap, not a target. Spent means fall back, never stall.
Cached per template and per host, never per page. A 700-page site with three templates costs at most three calls. There is no per-page model call anywhere, by construction — that is the bill this design exists to avoid.
Validated before trusted. A proposed selector that matches nothing on the page is discarded, and the algorithmic answer stands.
Only ever stricter at the identity gate. A host the gate refused is never re-admitted by asking; reasoning can veto a wrong admission and cannot create one.
Recorded. Every consultation, its question and its answer appear in the result beside the coverage note. A judgement nobody can audit is worse than a heuristic.

It needs a configured provider as well as the flag, and with it off every path behaves exactly as it does without it — which is what the test suite runs.

Known limits

Kept here rather than only in System Files/Issues.md, because a tool whose pitch is calibrated confidence cannot be selective about its own.

Multi-word names resolve about 8 times in 20, up from 1 before the name-shape lap. apache airflow, ruby on rails, open telemetry, shadcn ui and visual studio code now work. spring boot, next auth, framer motion, aws lambda and the cloud-provider names still refuse — several for honest reasons, such as documentation living under a deep path on an enormous vendor portal that no name shape reaches. Refusing beats guessing.
A resolved name is not always the right project. Verification confirms a page is about something with that name, which is not the same as confirming the project. On 56 names nobody had tuned for, 91% resolved correctly on the first try (held-out accuracy). The misses are contested names (sqlx is a Go library and a Rust one), projects GitHub's search does not surface under their own name, and sites that render only in a browser. Check the URL in the result before trusting a harvest, pass ecosystem= or language= when a name is shared, and use forget_resolution when it is wrong.
Resolution leans on GitHub's search, which allows ten requests a minute without a token. A burst beyond that waits up to a minute rather than answer without it; set GITHUB_TOKEN to lift the limit.

flask and polars, which used to reach a to-do app at flask.io and a third-party site, now resolve to their own documentation: a domain that stands on owning the name alone is held until the registries have been asked (R10).

What was closed, each after being caught live: a domain probe no longer keeps its ownership claim when it redirects onto a code host (mojo.dev lands on a Java library); a language can claim the lang domain it publishes from (ziglang.org, nim-lang.org); the claim a host makes on a name is now graded rather than counted, so a first-come <name>.github.io label and a country-code community mirror no longer outrank the project's own domain (tensorflow reached the Rust bindings' rustdoc and pytorch the Korean user group before this); a sub-project documented on its parent's host can be identified at all, through a registry-nominated URL whose path names it (langgraph); and djangoproject.com counts as Django's own domain, so django reaches docs.djangoproject.com rather than the weblog. See System Files/Issues.md (the closed entries teach as much as the open).
A scoped npm name cannot pass the gate. @tanstack/react-query is refused: the site writes "TanStack Query" and never the scoped name, and the registry's agreement is one strong signal where two are required. Whether the registry binding a scoped name to a homepage should count as identification is an open decision (Issues.md R7), not a bug.
A site that rate-limits you sets the pace. Read the Docs answers 429 to an address that harvests one of its sites three times in an hour. A harvest waits out Retry-After, stops after three refusals in a row, and lists the refused pages as refused by the site — not as missing, and not, as it once did, as "nothing on them reads like documentation".
A page under an unrecognised template is refused, not stored. Extraction tries nine selectors and then scores the page by text-to-link density; where nothing reads like documentation it stores nothing and says so, rather than silently keeping the navigation. That is the right default — it is what stopped a sidebar being filed as a manual — but it costs a real page whenever an API reference is mostly method links. Bounded reasoning can propose a selector for such a template, validated against the page before it is used.
A page that answers 200 while rendering an error can still be stored. A 4xx or 5xx is refused outright, but a soft 404 — a genuine 200 whose body reads "Page not found" — is indistinguishable from documentation without reading it. Bounded reasoning catches these when it is switched on; with it off, they are stored.
An interrupted harvest leaves nothing readable. Pages are durable as they are fetched, and whatever was already stored is untouched — but a harvest that never finished stays invisible until it does, rather than being served as a partial copy. That is deliberate: a partial corpus presented as a whole one is the failure this project exists to refuse. Whether it should be readable behind an explicit opt-in is an open question.
A site that is only an app is slow to harvest. A page with no readable HTML, no Markdown copy and no sitemap entry is found by rendering, one page at a time on one browser — correct, and minutes where a static site takes seconds.
One sidebar for a whole product family says nothing narrower. Firebase's navigation spans every Firebase product, so a harvest from firebase.google.com/docs/firestore keeps to /docs/. Pass scope="/docs/firestore/", or a topic, to keep to Firestore.
A topic is judged by its words. Relevance is title, address, headings and prose against a stated vocabulary, not understanding: a borderline page can land on either side. The answer names the terms used and lists what was left out by section, so a wrong call is visible and can be widened.
A language's edition is found only where the site shows it. The page's own code, or the site filing it under the language's name. A project that documents every language on one page (Stripe) is harvested with all of them, and says so.
Background harvests do not survive a restart. The pages do — they are written as they are fetched — and the report of one is now visible from any process, so a harvest started by the MCP subprocess shows up in the panel and in list_knowledge_base. What still does not survive is the harvest itself: it runs on a daemon thread and dies with its process. When that happens the tracker says the harvest stopped reporting rather than leaving it spinning, because a dead process and a slow one differ only in their silence.

System Files/ is the record of the system as built. Start with Architecture.md — how the pieces fit, in diagrams, including the acquisition ladder and what the identity gate will and will not believe. PRD.md is what this is for and who for, each requirement tied to the benchmark case that measures it; Design.md is the decisions and the incident behind each; Workflow.md follows a name to an answer and says how the system is run, locally, offline and hosted; Issues.md is the live register, including every limit above; Audit.md is the current verdict, with the runs behind it.

Security

DocsForge fetches URLs chosen by whoever is talking to it, which in the MCP and web paths can be a language model. Every request leaves through one function, Fetcher.get, and these guards apply to all of them:

SSRF — requests to private, loopback, link-local, reserved, multicast and unspecified addresses are refused: by the address a hostname resolves to (NAT64-mapped addresses unwrapped first), in every inet_aton spelling (2130706433, 0x7f000001, 0177.0.0.1, 127.1) without consulting the platform's resolver, at every redirect hop — redirects are followed by DocsForge, one at a time, each target guarded — and inside the browser when rendering, where every request the page makes passes the same check. Set DOCSFORGE_ALLOW_PRIVATE=1 (or --allow-private) to scrape docs on your own network. The benchmark tries loopback, the cloud metadata address, a DNS name that resolves to loopback, and a 302 to each, on every run.
Path traversal — save_docs cannot write outside DOCSFORGE_OUT_ROOT.
The bearer gate — a hosted /mcp requires Authorization: Bearer; main.py --http refuses to bind a non-loopback address without a token.
Deletion is opt-in — forget_documentation exists only with DOCSFORGE_ALLOW_DELETE=1; the offline runner turns it on because that store is disposable.
Politeness — a per-host delay and concurrency cap, Retry-After honoured on a 429 (bounded), and a harvest that stops after three refusals rather than answering a site's "slow down" with thirty more requests.

Rendered Markdown in the web chat is sanitized with nh3 before it reaches the page, since it mixes model output with scraped HTML. Bind the web chat to 127.0.0.1 (the default) unless you have put authentication in front of it.

Deploying

Two shapes. Both put the corpus in Postgres and serve the same routes: the site at /, /tools, /connect, a /health check, and the MCP endpoint at /mcp behind a bearer token. The web chat, tests, scripts, the published benchmarks, System Files/ and .env are never shipped.

Variable	Required	What it does
`DOCSFORGE_MCP_TOKEN`	yes	Clients send it as `Authorization: Bearer …`. Generate it (`openssl rand -hex 32`); never reuse a database password.
`DOCSFORGE_DB` or `DATABASE_URL`	yes	The Postgres DSN, `?sslmode=require` on Aiven. On a host with no persistent disk a file store would vanish.
`DOCSFORGE_HARVEST_DEADLINE`	no	Seconds a harvest may take inside one tool call before it goes to the background (25).
`DOCSFORGE_PUBLIC_URL`	no	The https address OAuth discovery advertises. Vercel's production domain is used when unset.
`PORT`	no	Container only: what the platform routes to (8765).

Vercel (serverless)

docsforge/server/vercel.py exposes the app; pyproject.toml names it under [tool.vercel] and vercel.json sets the function's maxDuration. Import the repository in Vercel, set the variables above in the project's Environment Variables (the token as sensitive), deploy. Then:

claude mcp add --transport http docsforge https://<project>.vercel.app/mcp   --header "Authorization: Bearer <token>"

What a serverless host changes, and the code says so rather than pretending:

Stateless. /mcp runs the SDK's stateless mode — every request carries everything, answers are plain JSON. Nothing lives between requests.
Ephemeral. Nothing runs after a response. A harvest that finishes inside DOCSFORGE_HARVEST_DEADLINE is stored as usual; one that does not is discarded, and the result says exactly that instead of "it continues in the background". Nothing partial is ever published. Large harvests are done from a long-lived DocsForge — python main.py on your machine with the same DOCSFORGE_DB — and are readable through Vercel at once. Raise the deadline (to below maxDuration, 300 s by default here) if your client waits that long on a tool call.
Only /tmp is writable, per instance. Caches (resolution cache, logs) go there and are lost between instances; that costs a repeat lookup, nothing more. Harvest status is kept in the database as well, so harvest_status answers the same from whichever instance takes the call.
Misconfiguration is visible. With no database or no token the site still serves and /mcp answers 503 with the reason; /health reports degraded when the configured database could not be reached.
Put the function next to the database. Vercel's default region is iad1 (Washington DC); a store-backed call makes 8–30 round trips, and measured against a database in Singapore that is 4–15 seconds per call, against under a second offline. "regions": ["sin1"] (or wherever your database is) in vercel.json is the one-line fix; a Hobby plan may pick one region.

A container (long-lived)

For a host that keeps a process alive — where background harvests work as designed — the Containerfile builds main.py --http from pip install .:

podman build -t docsforge -f Containerfile .
podman run -p 8765:8765 -e DOCSFORGE_MCP_TOKEN=… -e DOCSFORGE_DB=… docsforge

Run one replica: streamable-HTTP sessions and running harvests live in the process. Redeploy when nothing is harvesting — a redeploy kills a harvest in flight and its status record with it.

The site pages under docsforge/server/site/ were designed in Stitch from the web chat's palette. scripts/site_exports/ holds Stitch's raw exports and python scripts/build_site.py scripts/site_exports turns them into the served pages — one nav, real links, Tailwind compiled to a static stylesheet, the version chip filled in from the package at request time. Needs Node for the compiler. Edit in Stitch, download, re-run.

Tests

DOCSFORGE_DB="" DATABASE_URL="" python -m pytest tests/ -q   # 1,310 offline tests, no network

# The 37 Postgres tests skip unless you point them at a throwaway database,
# which makes a green run look more complete than it is — set this before
# trusting one. The variable is deliberately NOT DOCSFORGE_DB, and the database
# it names must not be the one holding real harvests: these tests create and
# drop technologies.
DOCSFORGE_TEST_DB=postgresql://postgres:pw@127.0.0.1:5432/DocsForgeTest python -m pytest tests/ -q

Set the two store variables empty rather than unsetting them: DocsForge reads .env on import and fills in only what is absent, and a test that built a store while they were absent once ran migrate() against the real database from inside the suite.

The live suite is python -m scripts.benchmark.run --offline — see Measuring it live. The older live checks need the network, and the last two need GROQ_API_KEY:

python scripts/smoke_mcp.py         # spawns main.py, the MCP server, over stdio
python -m docsforge.server.app --port 8123 &
python scripts/smoke_web.py 8123    # one real Groq turn, end to end
python scripts/smoke_multiturn.py 8123
python scripts/shoot_ui.py 8123     # screenshots every UI state

shoot_ui.py stubs the model stream by default, so it costs no tokens and is deterministic; pass --live to drive a real turn instead.

Output

By default, each source produces its own Markdown file in the output directory, named from the host, path, and a short hash of the URL — so pages from different sites never overwrite each other. Every file starts with a comment header noting the source URL, detected type, and timestamp. Use --single-file to merge all documents into one file separated by horizontal rules.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.impeccable		.impeccable
Logo		Logo
System Files		System Files
benchmarks		benchmarks
docsforge		docsforge
scripts		scripts
tests		tests
.containerignore		.containerignore
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.vercelignore		.vercelignore
CONTRIBUTING.md		CONTRIBUTING.md
Containerfile		Containerfile
DESIGN.md		DESIGN.md
LICENSE		LICENSE
README.md		README.md
main.py		main.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
vercel.json		vercel.json

Latest commit

History

Folders and files

Repository files navigation