Add language documentation shortcuts (#25): Python, MDN, Node.js, Ruby, Go, Rust, Java, PHP, TypeScript, C/C++, and .NET - #26
Conversation
First three languages from #25. Each is one definition in clis/: - oc py library json, oc py doc reference/datamodel (the .html is appended, matching how the aws shortcuts spell pages) - oc mdn js Array/map, oc mdn css grid-template-columns, and a generic oc mdn doc <path> for everything else under /en-US/docs - oc node api fs Python renders search client-side (Sphinx), MDN serves its search page as a client-rendered shell with only the nav in the HTML, and nodejs.org has no search, so all three get the DuckDuckGo site: filter fallback the aws and gcp shortcuts already use. Short names py, mdn, and node join the alias table; python, mozilla, nodejs, and the full domains resolve as before through the generic rules.
docs.python.org has no search server: its search page downloads the whole Sphinx index (searchindex.js, ~900KB gzipped) and ranks matches in the browser. oc now does the same ranking itself, so 'oc py search' answers from the site's own index instead of DuckDuckGo. The index is cached on disk for a day and never printed; what renders is a numbered result list that rides the normal open path, so 'do <n>' follows a result. A query that names a symbol exactly (json.dumps) links straight to its anchor. A site definition opts in with a 'sphinx' field naming the docs root, so any Sphinx site can use the backend later.
|
Pushed a prototype of a Sphinx search backend. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL flagged the tag strip in plainTitle: requiring a closing '>' means a title whose markup never closes keeps its '<script' fragment. The '>' is optional now, so every '<' starts a strip and none reaches the results page, and the output is trimmed. Both call sites already HTML-escape the result; this hardens the flattening itself. Covered by a new test.
CodeQL cannot see that the optional '>' already keeps every bracket out of the flattened title, and it re-flags each shape of the regex. The character walk does the same thing in a form whose completeness is plain to read: nothing between or including brackets survives, closed or not.
|
CodeQL flagged the title flattening in the Sphinx backend (js/incomplete-multi-character-sanitization): the original regex required a closing '>', so a title whose markup never closes could keep a literal '<script' fragment. Both call sites already HTML-escape the flattened title, so nothing reached the results page unescaped, but the flattening itself was incomplete. Fixed in d56ddb3 and e9d5ad4: titles are now flattened by a character walk that keeps only text outside angle brackets, so no '<' or '>' survives, closed tag or not, plus a regression test for the unterminated-tag case. All checks are green again and the alert is closed as fixed. |
MDN renders its search results client-side, so 'oc mdn search' has gone through DuckDuckGo with a site: filter. But the page it renders gets its results from a public JSON endpoint, /api/v1/search, which returns the site's own ranking with a summary per result in one ~11KB response. A new 'api' shortcut shape names such an endpoint and which response fields hold the list, title, URL, and snippet; the response becomes the same synthetic results page a Sphinx search produces and rides the normal render path, so 'do <n>' follows a result. First view is ~520 tokens and works for any site whose search answers as JSON.
|
MDN's search no longer goes through DuckDuckGo. Its results page is client-rendered, but the page gets its results from a public JSON endpoint, /api/v1/search, so 5706aab adds a second search backend beside the Sphinx one: an 'api' shortcut shape that names a JSON search endpoint and which response fields hold the list, title, URL, and snippet. The response becomes the same numbered results page, so do/find/read/next work unchanged. Live: 'oc mdn search array map' prints MDN's own ranking (2,696 matches, top 10 with one-line summaries) for ~520 tokens, and 'oc do 1' lands on Array.prototype.map(). The shape is declarative, so any site whose search answers as JSON can use it from its clis/ definition alone. Of the three language sites in this PR, only Node.js still searches via DuckDuckGo: nodejs.org ships no search index or endpoint at all. |
nodejs.org has no search results page: the site's search box is a JavaScript modal asking a third-party service, so the search verb went through DuckDuckGo. But the API docs publish their entire reference as one static JSON file, all.json, much the way a Sphinx site publishes its search index, so a new nodedoc backend ranks that file locally: every module, class, method, property, and event heading becomes a result linking to its own anchor, and oc node search prints them as a normal numbered page. The file is ~8MB (~1MB over the wire) and static, so the day cache the Sphinx backend used moves to a shared cache module both backends call: one directory per backend, one file per host, parsed before written so a block page never poisons it. A typical result list costs under 100 tokens and answers from disk in under 100ms once cached.
|
Node.js is off DuckDuckGo too, via a third shape: nodejs.org has no search results page at all (the search box is a JavaScript modal asking a third-party service), but the API docs publish their whole reference as one static JSON file, all.json, much the way a Sphinx site publishes its index. A new ~84 tokens per result list, ~80ms from cache. Of the three language references on this branch, none relies on DuckDuckGo now: Python ranks its Sphinx index, MDN asks its own search API, Node.js ranks its own docs corpus. |
RDoc publishes a site's whole search index as one static JS file, the same way Sphinx and the Node.js docs do, so 'oc ruby search' fetches docs.ruby-lang.org's index (en/3.4, where the file exists), caches it under the shared day cache, and ranks it with the same scorer the Node backend uses. Class methods print as Class.name(params), instance methods as Class#name(params), and every entry links straight to its method anchor. 'oc ruby class Array' opens a class page directly. cli.js now dispatches the local search backends through one map, and the site-resolution guard keys on the query field, which only a search shape ever carries.
The rest of the languages from #25, each probed for the cheapest shape that works: - go: pkg.go.dev renders search on the server, so 'search' is a plain URL and 'pkg <path>' opens a package page. - php: manual-lookup.php sends an exact function name straight to its manual page, so 'fn array_map' lands on the function doc; fuzzy search goes through DuckDuckGo since the suggestions render behind the manual's own TOC. - cpp: cppreference's new static site has clean /cpp/ and /c/ paths; its native search page is bare links oc refuses as unreadable, so 'search' goes through DuckDuckGo. - rust: std and doc page opens; rustdoc's search-index.js format is version-fragile, so 'search' goes through DuckDuckGo. - java: Javadoc opens pinned to javase/26, the current GA (27 redirects there); the typeSearchIndex file lacks the module names a URL needs, so 'search' goes through DuckDuckGo. - ts: handbook page opens; the site searches via Algolia client-side, so DuckDuckGo again. - learn gets a 'dotnet <api>' verb, since .NET API docs already live there. README table and prose and the skill notes cover the new sites.
|
Second wave from #25 is in: Go, Rust, Java, Ruby, PHP, TypeScript, C/C++, and a Ruby got a real local search backend: RDoc ships the whole search index as one static JS file, so The rest were probed for the cheapest shape that works:
137 tests pass, all verbs checked live, CI green. That covers every language on the #25 list. |
The skill is paid for on every load, so the note naming which search verbs run locally now says it in one clause instead of four.
|
Docs are now in sync with the full branch across the repos:
|
only-cli
changed the title
|
The follow-up benchmark landed in only-cli/benchmarks (25ec3c1): a docs suite of eleven graded lookups, one per language shortcut in this PR, run end to end through both Claude Code headless and codex exec against the agents' built in web tools, with this branch shimmed onto PATH. Headline numbers:
Full tables in the benchmarks README and results/agent-latest-docs{,-codex}.md. |
2 participants
First three languages from #25: Python, MDN, and Node.js.
Each is one JSON definition in
clis/, the same shape as the cloud docs shortcuts:All three sites render fine through
oc opentoday (checked live: the Python library page, the MDN reference pages, and the Node API pages all come back as real content under budget), so this is only about not making the agent guess URL shapes.Search: Python renders search client-side with Sphinx, MDN serves its search page as a client-rendered shell (the HTML carries only the nav), and nodejs.org ships no search at all, so each
search <query>goes through DuckDuckGo with a baked-insite:filter, the fallback the aws and gcp shortcuts already use.Names:
py,mdn, andnodejoin the alias table;python,mozilla,nodejs, and the full domains already resolve through the generic rules. README table, llms.txt, and the skill's site list are updated, and the new definitions are covered in tests/sites.test.js alongside a check that a doc path keeps its slashes.Settles the open question in #25 in favor of one CLI per language: it matches how
clis/works today and each file stays a screenful.