Skip to content

docs(sql): generate the keywords page from the engine instead of maintaining it by hand - #348

Draft
fupelaqu wants to merge 1 commit into
mainfrom
feature/generated-keywords-page
Draft

fupelaqu wants to merge 1 commit into
mainfrom
feature/generated-keywords-page

Conversation

@fupelaqu

Copy link
Copy Markdown
Contributor

Closes #347

Makes the keyword tables of documentation/sql/keywords.md generated from the engine, and adds the
guard that keeps them that way. The hand-written explanation on the page stays hand-written.

Why

The page said "A list of reserved words recognized by the parser" and was not that list. Measured on
main, entry by entry against Parser.reservedKeywords (145 keyword lines on the page):

direction count examples
reserved words the page did not list 26 EXISTS, ALL, UNION, EXCEPT, MATCH, TRUE/FALSE, GROUP, ORDER, BY, INNER/LEFT/RIGHT/FULL/CROSS/OUTER, PI, TO, COLUMN, FIRST, LAST, AGAINST, PARSE_DATE, PARSE_DATETIME, FORMAT_DATE, FORMAT_DATETIME
words the page listed that are recognised but not reserved 26 CEILING, UCASE, LCASE, CONVERT, TRY_CAST, SAFE_CAST, RLIKE, REGEXP, OVER, POINT, ST_DISTANCE, POSITION, POWER, REVERSE, CURDATE, CURTIME, the eight DATE*/DATETIME* aliases

Both lists are reproduced by the new generator. The page also omitted 186 recognised words entirely
(145 of 331) and had 17 date/time functions commented out with [//]: #.

It drifted in both directions because it was a grammar-keyword list wearing a reserved-word title
one dimension where the engine has two. Story 22.2 made that load-bearing: EXISTS is reserved, so
SELECT exists FROM t is a parse error, while ANY and SOME were deliberately left unreserved so a
column named any keeps parsing.

And two dimensions turn out not to be enough either. Executing every row through the parser — which
is what the generator now does — found four words that are neither reserved nor usable: CURDATE,
CURTIME, NULL and RANDOM are shadowed by a zero-argument function or a literal. SELECT curdate FROM t returns the current date, silently, instead of your curdate column. The old page listed
two of them under "Date/Time functions" and said nothing; a yes/no page would have printed no and
been actively misleading. They now carry no (shadowed).

What is generated, and how to regenerate

Only the region between <!-- BEGIN GENERATED KEYWORDS --> and <!-- END GENERATED KEYWORDS -->.
Everything outside is prose, read back verbatim and preserved — including the framing PR #346 wrote.

sbt regenerateKeywordsPage

Discoverable from three places: the generated banner at the top of the region, the guard's failure
message, and the page's own "How to read the table" section — and KeywordsPageSpec pins the command
string against the alias build.sbt actually declares, so the instruction cannot become a lie.

Sources of truth, none copied: SQLKeywords for the words, Parser.reservedKeywords for the reserved
flag, the parser itself for the shadowed flag. The Kind column is derived from the token's
package, never from the word, so a new keyword needs no edit to the generator — and an unmapped
package throws, naming itself.

Result: 331 keywords (134 reserved, 4 shadowed), 11 compound phrases, 1 symbolic operator. The
52-entry divergence is subsumed: once every word carries its true flag, the two directions of drift
cannot exist.

Three words carry the kind reserved only rather than being guessed at or dropped —
CURRENT_DATETIME, FORMAT_DATE, FORMAT_DATETIME are reserved but backed by no grammar surface (the
same three SQLKeywordsSpec already pins). Nothing else resisted classification.

The guard, and its RED proof

KeywordsPageSpec, 25 tests, no allow-list, no assume (an assume-cancelled suite exits 0, and this
suite IS the mechanism). It owns four failure modes that cannot see each other: the page is stale; the
generator is short or inventive; the render misrepresents a correct model; or the page makes a
claim the parser contradicts.

Falsified in two rounds. Every mutation was run, restored by bytes, and the restore asserted.

Round 1 — 12 mutations, all RED: flip a reserved cell · flip a shadowed cell · delete a row · drop
the :: row · delete the WITHIN GROUP construction · restore a hand keyword list outside the markers
· rename the sbt alias · re-add the duplicate reserved word · invent a row · collapse a kind label ·
drop the per-row kind sort · convert the page to CRLF.

Round 2 — the sharp one: six generator faults, with the page regenerated to agree with them.

broken generator caught by
invents a row emit no row that no source of truth backs, label every row with the kinds its registry surfaces imply
drops statementWords emit a row for every word in SQLKeywords + 2 more
collapses a kind label label every row with the kinds its registry surfaces imply
drops the per-row kind sort emit no row twice, and exactly the known kinds
render drops the Reserved column 4 tests, incl. both parser oracles
shadow detection disabled agree with the parser about every shadowed cell it prints

The guard cannot be satisfied by regenerating. That round is also how the guard's one real blind
spot was found: the kind-vocabulary assertion stayed green when a whole surface was relabelled,
because another surface still contributed the label. The kind mapping is now pinned in both
directions, derived entirely from the registry.

And the oracle caught a flaw in itself: SELECT true FROM t renders identically whether true is a
literal or a column, so a round-trip test in the value position reports TRUE/FALSE as usable bare
names. The reserved half of the oracle therefore runs in the alias position (SELECT c AS <word>),
which regexAlias gates from the same list and which admits no literal reading; the shadow half stays
in the value position, where shadowing is what happens.

Idempotency: starting from an emptied generated region, two separate sbt invocations (two JVMs)
produce byte-identical files equal to the committed page — sha256 eb2a2dec…. Nothing is emitted in
Set iteration order; every table is sorted, and every toUpperCase/toLowerCase in the new code
passes Locale.ROOT.

Every SQL claim the prose makes is executed: SELECT exists FROM t asserted to be rejected;
SELECT any, some FROM t WHERE any = 1, SELECT "exists" FROM t, SELECT `exists` FROM t,
SELECT t.from FROM t, SELECT doc.count FROM t, SELECT foo-bar FROM t asserted to parse;
SELECT in-stock FROM t asserted to be rejected (a bare name that merely starts with a reserved word
is also rejected — measured, and now documented); the four multi-keyword constructions asserted to
parse; and the normalisation claims asserted on the whole rendered statement, never with include
include("IN (SELECT") is satisfied by NOT IN (SELECT, i.e. by the inverse of the claim.

Beyond the page: three things in the diff that are not the page

  • Parser.reservedKeywords privateprivate[sql]. Nothing outside the sql package can see it
    and no production code reads it; the alternative was scraping Parser.scala as text, which is
    assume-guarded where it already exists and cannot be the basis of a generated artefact. ⚠️ It does
    add a public accessor to the Parser trait's interface — binary-incompatible for a mixer, acceptable
    on the already-incompatible 0.24.0-SNAPSHOT. Verified it disturbs nothing reflective:
    HelpCorpusSpec's two Parser.getClass.getMethods scans filter on a PackratParser return type.
  • reservedKeywords had "on" twice. Harmless as a regex alternative, invisible once it reaches a
    Set — but this list is a documentation source now, so the duplicate is removed and the guard asserts
    the list is duplicate-free.
  • SQLKeywordsSpec's anchor scan is hardened to demand exactly one occurrence. Two traps were hit
    and fixed while widening the visibility, both now recorded in the scaladoc at the declaration: a
    type ascription between the name and the = makes the raw-text scan miss, and a comment quoting
    the anchor whole
    gets matched instead of the declaration. The second failed with Set() was empty, which names neither cause.

documentation/sql/keywords.md also gains a -text rule: it is now asserted byte for byte, and
checkout-time eol conversion on a core.autocrlf=true clone would redden the guard for the platform
rather than the content, invisibly to macOS/Linux CI. The guard asserts the LF invariant too, because
regenerating a CRLF page yields a permanently mixed file that then passes everything else.

Merge order — this PR must land AFTER #346

#346 (docs/22-subqueries) edits this same file and must merge first. This branch is cut from main
and carries #346's keywords.md prose verbatim, so the two conflict on documentation/sql/keywords.md
and the resolution is take this branch's version — it already contains #346's text. Rebase onto main
after #346 lands, re-run sbt "sql/testOnly *KeywordsPageSpec", then merge.

⚠️ One consequence for reviewers, stated because it has already been mis-filed once: the page links
known_limitations.md#subqueries-and-derived-tables, and that heading is created by #346. Checked
against main the anchor looks dead; checked against origin/docs/22-subqueries it resolves
(known_limitations.md:76). Do not retarget or remove that link — the pre-#346 section it would
point at instead is renamed by #346 and says subqueries do not work, which is the claim #346 exists to
delete. Every other link on the page was verified against main: dql_statements.md#quoted-identifiers
(heading at :91), README.md, and the in-page #compound-phrases / #multi-keyword-constructions
anchors all resolve. documentation/sql/README.md:16 and operators.md:5 link this page by filename
only, so deleting the old ## Main clauses / ## Aggregates headings broke no inbound anchor.

No Closes line for #345#346 owns it; this PR only carries the keywords.md half of its prose forward.

Web twin — deliberately NOT in this PR

softclient4es-web has no keywords page today and nothing links to one, so nothing is left broken.
Generating the MDX twin is a materially different job, not a formatting variant: it needs Starlight
frontmatter, a sidebar entry in astro.config.mjs, absolute /sql/<page>/ routes instead of relative
.md links, and MDX escaping — and, decisively, it is a second repository with no Scala build, so an
elasticsql test can neither write it nor guard it. A generator that writes across a repo boundary is
unguardable from either side, which is the failure this PR exists to end.

Hand-copying the list into MDX is explicitly rejected: it would create exactly the second drifting copy
this change removes. The follow-up that does work — for a separate issue — is for this generator to also
emit a machine-readable artefact (e.g. keywords.json) that the web repo's own build consumes, so the
MDX page is generated there from the same source of truth.

Verification

  • sbt "+ compile" — green on 2.13.16 and 2.12.20 (main sources; CI's cross gate).
  • sbt "sql/test" — 1269/1269. sbt "core/test" — 1057/1057. Both 2.13, as CI runs them.
  • sbt "++ 2.12.20 sql/Test/compile" — green (covers the two 2.12-only lexer traps even though CI no
    longer compiles 2.12 test sources). The one 2.12 red in core is the pre-existing
    SchemaCacheTtlApiSpec 19/20, green on 2.13.
  • sbt headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck — green (CI's lint job verbatim).
  • sbt "sql/clean" regenerateKeywordsPage — the documented command works on a cold tree.
  • No ES / Docker path is touched.

…taining it by hand

`documentation/sql/keywords.md` called itself a reserved-word list and was not one. Measured
against `Parser.reservedKeywords`: 26 reserved words were absent from it, 26 words it listed are
recognised but not reserved, and 186 recognised words appeared nowhere. It drifted in both
directions because it was a grammar-keyword list with a reserved-word title - one dimension where
the engine has two, which story 22.2 made load-bearing.

The keyword tables are now generated between two markers from `SQLKeywords` (words),
`Parser.reservedKeywords` (reserved flag) and the parser itself (shadowed flag); the prose outside
the markers is hand-written and survives regeneration. `sbt regenerateKeywordsPage` rebuilds it.

Executing the oracle rather than restating the list found a third answer the page could not give:
CURDATE, CURTIME, NULL and RANDOM are neither reserved nor usable - a zero-argument function or a
literal shadows them silently - so they carry `no (shadowed)`.

`KeywordsPageSpec` (25 tests, no allow-list, no `assume`) guards staleness, a short OR inventive
generator, a render that misrepresents a correct model, and every claim the prose makes. Proven by
18 mutations, six of them applied to the generator with the page regenerated to agree: the guard
cannot be satisfied by regenerating.

Also here, because the page now depends on them: `Parser.reservedKeywords` widened to
`private[sql]` and de-duplicated (`"on"` appeared twice); `SQLKeywordsSpec`'s raw-text anchor scan
hardened to demand exactly one occurrence; and a `-text` rule for a file that is now asserted byte
for byte.

Closes #347
@fupelaqu
fupelaqu force-pushed the feature/generated-keywords-page branch from b5b4fd9 to 2c9039a Compare September 16, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keywords documentation page is hand-maintained and cannot express which words are reserved

1 participant