A package indexer with BETA 997 snapshots, a scanner that reads more of each package, and a shorter Documentation Development section

.gitattributes

@@ -17,6 +17,11 @@
 # cannot be run.
 wisdom/extract/workflow.mjs text eol=lf
 
+# The package indexer's snapshots are checked out byte for byte: a sync compares the
+# api.json it writes (LF) with the committed one, and a store seeded from the snapshots
+# has to start from the bytes a sync writes, or its first diff is every line.
+indexer/snapshots/** -text
+
 # The help add-in's resources are checked out byte for byte: every build compares its
 # copy of the symbol index with the LF file it writes (builder/addin-index.mjs).
 add-in/Resources/** -text

.github/actions/run-gates/action.yml

@@ -148,6 +148,15 @@ runs:
     - name: Unit-test the help add-in's project file (test/addin-project.test.mjs)
       shell: bash
       run: node --test test/addin-project.test.mjs
+    # Unit tests for the package indexer, indexer/lib/: the VB6 preprocessing,
+    # the files of a package, the api.json built from them (sorted by code
+    # unit, nested types by their path, values and fields in declaration order,
+    # the same bytes from the same files) and the differ that compares two of
+    # them. Reads indexer/sample.twinpack; no network, no git, no install. No
+    # browser, no built tree, well under a second.
+    - name: Unit-test the package indexer (test/indexer.test.mjs)
+      shell: bash
+      run: node --test test/indexer.test.mjs
     # A regex that backtracks exponentially does not fail a build, it stops
     # one: the corpus passes until a page happens to contain the trigger, and
     # then a render worker sits inside String.replace forever. Nothing else

WIP.Authoring.md

@@ -83,17 +83,18 @@ The attribution rule is **per-page**, determined by content provenance — not b
 
 ### Authoring-only frontmatter
 
-Three keys on a package's `index.md` are **provenance for the authoring pass**. None of
+Four keys on a package's `index.md` are **provenance for the authoring pass**. None of
 them reaches the HTML --- `grep -r indexed_from docs/_site` returns nothing, because nothing
 in `template.mjs` iterates frontmatter generically; every consumer reads a named field. The
 build reads one of them, `exclude_from_docs`, for one purpose: the symbol index's count of
 public symbols that no page documents leaves those names out (`reportableGaps` in
 [builder/symbols.mjs](builder/symbols.mjs); `--symbol-gaps` writes the list). Leave all
-three in place.
+four in place.
 
 | Key | Means |
 |-----|-------|
-| `indexed_from` | The twinBASIC build whose package download the documentation was indexed against. Currently `beta-x-0983` on all six. |
+| `package_name` | The packages in the indexer's store that the section documents, by their names there, when the folder's name is not theirs: `WebView2Package` for `WebView2`, `CustomControls, CustomControlsPackage` for `CustomControls`, `cefPackage145` for `CEF`. Absent, the folder's name is the package's. `indexer/lib/doc-pages.mjs` reads it. |
+| `indexed_from` | The twinBASIC build whose package download the documentation was indexed against, as its release tag. Currently `beta-x-0997` on all thirteen, the build whose `indexer/snapshots/` the pages are compared with. |
 | `exclude_from_docs` | Modules or classes in that package deliberately left undocumented --- internals, private helpers, API shims. |
 | `exclude_kinds` | Member *kinds* deliberately left undocumented. Only `WinEventLogLib` uses it, for `Declare`. |
 
@@ -102,14 +103,16 @@ newer build, and anything new that is not named in `exclude_from_docs` or `exclu
 a documentation gap.** Without them a re-index cannot tell "not yet written" from
 "deliberately omitted", which is the whole point of recording them.
 
-The six packages carrying `indexed_from` are `AppGlobalClassObject`, `CustomControls`,
-`TwinBasicAssertions`, `WinEventLogLib`, `WinNamedPipesLib` and `WinServicesLib`.
-`exclude_from_docs` is on three of them (`InternalStuff`; `EventLogHelperPrivate` +
-`EventLogAPIs`; `ServicesConstantsPublic`).
+All thirteen package landing pages carry `indexed_from`. `exclude_from_docs` is on three
+of them (`InternalStuff`; `EventLogHelperPrivate` + `EventLogAPIs`;
+`ServicesConstantsPublic`). The three CEF builds are one section with one snapshot; why, and
+how the indexer ties a section to its snapshots: [indexer/README.md](indexer/README.md#documented-packages).
 
 **Bump `indexed_from` whenever a package is re-indexed against a newer build**, in the same
-commit as the content it adds. A stale value is worse than none: it asserts a completeness
-check was run against a build that no longer matches the package.
+commit as the content it adds, and write the new snapshots into `indexer/snapshots/` in it
+(`node indexer/twin-index.mjs sync`), with `builder/package-api.json` for a built-in or
+default package (`node indexer/twin-index.mjs derive`). A stale value is worse than none: it asserts a
+completeness check was run against a build that no longer matches the package.
 
 One other inert key is a different thing: `has_children` on four pages is a
 just-the-docs leftover. tbdocs derives the nav tree itself and never reads it. Nobody
@@ -322,6 +325,22 @@ system remains a genuine consumer of the permalinks.
 4. **Add the page** to `Reference/Statements.md` or `Reference/Procedures and Functions.md` if it's a statement or callable and not already listed there.
 5. **Run the [site integrity check](WIP.md#site-integrity-check)** after the batch and before committing.
 
+**Finding the symbols to write.** After a `sync`, `node indexer/twin-index.mjs analyze --package <name>`
+lists what is public in the package and the section's pages lack, in
+`indexer/.packages/report.json`. A section with `indexed_from` is compared with the snapshot of
+that build: symbols **added** (create a page), **changed** (update it; the report gives the old
+and the new signature) and **removed** (flagged: reword or remove the page by hand). A section
+without it is audited: each public symbol is matched by name with the `title:` of the section's
+pages, so a member documented inline on its container's page is listed as missing, and a name
+that several types share counts as documented once any page has it. A symbol is a type or a
+name a type declares, so a property's Get, Let and Set and the overloads of a name are one.
+Anything deliberately undocumented belongs in the landing page's `exclude_from_docs` or
+`exclude_kinds` (see [Authoring-only frontmatter](#authoring-only-frontmatter)), not left in
+the list. When the batch is done, bump `indexed_from` and commit the new snapshots and
+`derive`'s `builder/package-api.json` with it; the
+rules and the reasons are in [WIP.md](WIP.md#the-package-indexer) and
+[WIP.Indexer.md](WIP.Indexer.md).
+
 ## twinBASIC deviations from VBA to flag
 
 Add a `> [!NOTE]` callout or rewrite the affected section when source diverges. Known cases:

WIP.Build.md

@@ -93,7 +93,7 @@ yet designed; the plan holds measurements and open questions for them.
 
 ### Tooling is JavaScript, and the three remaining `.py` files each have a reason
 
-Everything under `scripts/`, `builder/`, `lib/`, `book/`, `eval/` and `wisdom/` is Node.js.
+Everything under `scripts/`, `builder/`, `lib/`, `book/`, `eval/`, `wisdom/` and `indexer/` is Node.js.
 One trap: **a tool that rewrites a file must preserve its line endings
 byte-exactly.** Python's `Path.read_text` / `write_text` round-trip applies
 universal-newline translation, rewriting any LF file it touches to CRLF on Windows
@@ -196,6 +196,8 @@ project file](#the-help-add-ins-project-file)).
 
 `wisdom/` — Discord knowledge-harvesting tool (three-phase: export → process → extract). Plans in `wisdom/PLAN-{1,2,3}.md`; implementation under `wisdom/`. Uses only Node.js built-in APIs. Running it is [WIP.Wisdom.md](WIP.Wisdom.md).
 
+`indexer/` — package indexer: reads the twinBASIC packages (the built-in and default ones from the local install, or with `--release` from the GitHub release; the community ones from TWINSERV), parses their `.twin` sources and writes an `api.json` snapshot per package into a local git store. The build does not use it. Uses only Node.js built-in APIs. Its layout and commands are in `indexer/README.md`, and why it is built as it is in [WIP.Indexer.md](WIP.Indexer.md). It is under the lint, format and regex-safety gates like the other tooling folders; `sample.twinpack` and the committed snapshots are data, not scripts; the sync state lives in the gitignored store, not in the repository.
+
 `eval/` — use-case evaluation of the developer documentation. `build_corpus.mjs` mirrors the
 repository with every non-prose file stubbed unreadable, so "documentation only" is a property
 of the tree rather than an instruction. `site_search.mjs` replays the site's real lunr index
@@ -679,9 +681,8 @@ Three decisions about the build side, each with the alternative it rules out:
   with no page and put the docs' own layout (Array filed under Information, the Styles
   classes documented though declared `Private`) in the wrong place.
 - **The snapshot is committed, like `inter-metrics.json`, because making it needs a
-  twinBASIC install.** `scripts/build_package_api.mjs` exports the packages through
-  `scripts/lib/tb-packages.mjs` (shared with `census_attributes.mjs`), scans them with
-  `scripts/lib/twin-api.mjs`, and writes ~255 KB. `package-api.json` is a build *input*, so `check_tree_fresh.mjs`
+  twinBASIC install.** `node indexer/twin-index.mjs derive` reads the packages' `.twinproj`
+  files as `sync` does, scans them with `scripts/lib/twin-api.mjs`, and writes ~255 KB. `package-api.json` is a build *input*, so `check_tree_fresh.mjs`
   watches it; `symbol-baseline.json` is an output and is in its `IGNORED_FILES`.
 - **The heading scan is `indexOf`, not a regex.** `/<h([1-6])\b([^>]*)>([\s\S]*?)<\/h\1>/`
   is cubic in the regex gate's census, and it runs over every reference page; the scan
@@ -707,8 +708,18 @@ out `-text`: a CRLF checkout of the copy would differ from the LF the build writ
 build.
 
 `scripts/check_symbol_index.mjs` is the gate on all three pieces, in `test.bat` and both
-CI workflows: forty-six probes on fixtures, no tree, no install. The scanner's probes are
-the traps the packages' `.twin` sources contain; the derivation's are each a rule the real
+CI workflows: over a hundred probes on fixtures, no tree, no install. The scanner's probes are
+the traps the packages' `.twin` sources contain, the community packages' included, and what
+a declaration's record says (signature, attributes with their arguments, description, return
+type, accessor, the project name). A block of them holds the forms an audit of `parseTwin`
+found it losing without a word (lists after `Inherits` and `Implements`, a qualified CoClass
+interface, `[Hidden(False)]`, modifier-named and initialised fields, brackets in a Type, a
+result type before `Overrides`, an unmodified `Declare`, and the `#If` branches as
+alternatives), each with the compiler's answer in its comment, checked with `tbrun --tests`
+on BETA 1005. A second block holds the second step's: declarations joined by colons,
+identifiers with characters past ASCII (the compiler takes every one as part of a name, a
+no-break space included, so `\b` is no word boundary for them), a type-declaration
+character on a name, and the condition each `#If` branch gives its members. The derivation's are each a rule the real
 site exercises only once or twice (`symbols:`, a section heading named like a member, the
 ellipsis the typographer puts in a Core H1).
 
@@ -926,7 +937,7 @@ polynomial. Every one of the constructions classifies `safe`.
 ### The regex-safety gate
 
 [scripts/check_regex_safety.mjs](scripts/check_regex_safety.mjs) parses every
-`.mjs` under `builder/`, `scripts/`, `lib/`, `book/`, `eval/` and `wisdom/` with
+`.mjs` under `builder/`, `scripts/`, `lib/`, `book/`, `eval/`, `wisdom/` and `indexer/` with
 acorn, takes the regex literals *and* every `new RegExp(...)` whose arguments the
 source decides, and refuses any that can backtrack exponentially. In `test.bat`
 and both CI workflows; ~10 s, no browser, no built tree. Exit 0 no exponential

WIP.Harness.md

@@ -33,7 +33,7 @@ makes it the strongest available evidence for what is legal syntax.
 
 The tooling unpacks and packs in process, through `unpackProject` and `packTree` in
 [scripts/lib/tb-project.mjs](scripts/lib/tb-project.mjs), which call impexp's
-`exportProject` and `importProject`: the census and `build_package_api`, `tbbuild --build`,
+`exportProject` and `importProject`: the census, `tbbuild --build`,
 `tbrun`, `check_examples`, `sweep_attributes`, `bug_repro`'s `pack`, and the add-in and IDE
 lanes. The compiler executable's `export` verb writes the same tree: on BETA 995 the two
 unpack the install's sixteen packages and thirty-two samples to identical trees, 1,633 files
@@ -139,7 +139,13 @@ comment in the file:
 - An **escaped identifier is spelled like an attribute**: `[_HiddenModule].Foo`,
   `[_MAX] = 0`. The tail after the `]` is what separates them. An Enum member may *be*
   one --- `Report.twin` declares `[ ]`, `[A4 Portrait]`, `[Letter Landscape]` as member
-  names.
+  names --- and **a bracket group alone on its line in an Enum body is always a member,
+  whatever is inside the brackets** (BETA 1005): `[Hidden]` is the member `Hidden`,
+  `E.[Hidden]` and `E.Hidden` are 0, `[Description("x")]` is a member of that name, and
+  `[NoSuchAttribute]` too. Code after the group (`[Hidden] Red = 5`) is refused, TB5182 at
+  `Red`. So the census records no attribute for such a group (`scripts/lib/census-scan.mjs`,
+  probes in `check_twin_parsers.mjs`); it once did for the 35 `[_]` members VBCCR's
+  `LongPtr` Enum declares in the samples.
 - **A comment can sit anywhere**: inline `/* voffset &H00A8*/ Property Get X()` before a
   declaration (DAO.twin), a trailing `' NOTE: ...` after a `]`, a whole `'` line *between*
   two attribute groups (VBA/Strings.twin; 142 sites), or a `#If` between an attribute and
@@ -195,13 +201,15 @@ other 193 are refused at every site.
   tool that takes a hyphenated option once with each of them before trusting it.
 - **A control must be refused for a site to mean anything.** An unknown name is built at every
   site, and a site whose control *compiles* is voided. An Enum body accepts any own-line
-  `[...]` --- `[ClassId("guid")]` and `[Hidden(True)]` included, neither of which can be a
-  member name --- so what it does with the line is unchecked, and nothing accepted there is
-  evidence. Inline, `[Name] X = 1`, the compiler refuses every attribute, `[Hidden]` included,
+  `[...]` --- `[ClassId("guid")]` and `[Hidden(True)]` included --- because the line is a
+  member, not an attribute (BETA 1005: `E.[Description("x")]` compiles and is 0, so such a
+  group is a legal member name), and nothing accepted there is evidence. Inline,
+  `[Name] X = 1`, the line is the member `[Name]` followed by text the parser has no
+  handler for (TB5182, at `X`), so the compiler refuses every attribute, `[Hidden]` included,
   though the page documents it on an Enum member. An Enum member target is therefore reported
-  as one the sweep cannot test, which is true. (The mechanism is not established; only the
-  behaviour was measured.) A local variable is different: an own-line `[Name]` in a Sub is a
-  *call* statement, so that variant is dropped and only the inline one is kept.
+  as one the sweep cannot test, which is true. A local variable is different: an own-line
+  `[Name]` in a Sub is a *call* statement, so that variant is dropped and only the inline one
+  is kept.
 - **A probe that draws what the control draws is a refusal, whatever the code.** The signature
   compared has the attribute's name (whole word, any case) and the probe's own generated names
   (`S000062`, `S000062_U`) taken out, or the control's message never equals the probe's.

WIP.HelpAddin.md

@@ -729,10 +729,9 @@ WinNativeCommonCtls' 738, 294 --- inherited members that pages name in a sentenc
   documents `=`, `<>` and four more. A page's first heading `# \ and \= operators` must be
   written `\\=`, since `\=` is a markdown escape.
 - **From the packages:** [builder/package-api.json](builder/package-api.json), which
-  [scripts/build_package_api.mjs](scripts/build_package_api.mjs) writes from the packages of
-  an install, through the export of [scripts/lib/tb-packages.mjs](scripts/lib/tb-packages.mjs)
-  (shared with `census_attributes.mjs`) and a declaration scanner,
-  [scripts/lib/twin-api.mjs](scripts/lib/twin-api.mjs). It is committed, like
+  `node indexer/twin-index.mjs derive` ([indexer/lib/package-api.mjs](indexer/lib/package-api.mjs))
+  writes from the packages of an install, read and parsed as the indexer's `sync` reads them,
+  through the declaration scanner [scripts/lib/twin-api.mjs](scripts/lib/twin-api.mjs). It is committed, like
   `inter-metrics.json`, because CI has no install. 1,379 types in 16 exports, 255 KB. It
   supplies each symbol's package, container and kind, lists the public symbols that have no
   page (documentation coverage as a side effect), and supplies each class's default
@@ -758,8 +757,9 @@ WinNativeCommonCtls' 738, 294 --- inherited members that pages name in a sentenc
     to `VBA.Collection` and `tbIDE.IToolWindowsV1` to `tbIDE.ToolWindows`, 37 in all, which
     is P5's hover answer turned into a lookup. A lookup that takes the compiler's word has
     to map `_Collection` to `Collection`.
-- **Regenerate** `package-api.json` with `node scripts/build_package_api.mjs` when the
-  reference is re-indexed against a newer build, and commit it with the pages.
+- **Regenerate** `package-api.json` with `node indexer/twin-index.mjs derive` when the
+  reference is re-indexed against a newer build, and commit it with the pages and the
+  indexer's snapshots.
 
 **The gates.** `builder/symbol-baseline.json` lists every published URL, and a build that
 loses one fails --- the case it exists for is a reworded member heading, which moves an