Conversation
…taining it by hand `documentation/sql/keywords.md` called itself a reserved-word list and was not one. Measured against `Parser.reservedKeywords`: 26 reserved words were absent from it, 26 words it listed are recognised but not reserved, and 186 recognised words appeared nowhere. It drifted in both directions because it was a grammar-keyword list with a reserved-word title - one dimension where the engine has two, which story 22.2 made load-bearing. The keyword tables are now generated between two markers from `SQLKeywords` (words), `Parser.reservedKeywords` (reserved flag) and the parser itself (shadowed flag); the prose outside the markers is hand-written and survives regeneration. `sbt regenerateKeywordsPage` rebuilds it. Executing the oracle rather than restating the list found a third answer the page could not give: CURDATE, CURTIME, NULL and RANDOM are neither reserved nor usable - a zero-argument function or a literal shadows them silently - so they carry `no (shadowed)`. `KeywordsPageSpec` (25 tests, no allow-list, no `assume`) guards staleness, a short OR inventive generator, a render that misrepresents a correct model, and every claim the prose makes. Proven by 18 mutations, six of them applied to the generator with the page regenerated to agree: the guard cannot be satisfied by regenerating. Also here, because the page now depends on them: `Parser.reservedKeywords` widened to `private[sql]` and de-duplicated (`"on"` appeared twice); `SQLKeywordsSpec`'s raw-text anchor scan hardened to demand exactly one occurrence; and a `-text` rule for a file that is now asserted byte for byte. Closes #347
fupelaqu
force-pushed
the
feature/generated-keywords-page
branch
from
September 16, 2026 09:57
b5b4fd9 to
2c9039a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #347
Makes the keyword tables of
documentation/sql/keywords.mdgenerated from the engine, and adds theguard that keeps them that way. The hand-written explanation on the page stays hand-written.
Why
The page said "A list of reserved words recognized by the parser" and was not that list. Measured on
main, entry by entry againstParser.reservedKeywords(145 keyword lines on the page):EXISTS,ALL,UNION,EXCEPT,MATCH,TRUE/FALSE,GROUP,ORDER,BY,INNER/LEFT/RIGHT/FULL/CROSS/OUTER,PI,TO,COLUMN,FIRST,LAST,AGAINST,PARSE_DATE,PARSE_DATETIME,FORMAT_DATE,FORMAT_DATETIMECEILING,UCASE,LCASE,CONVERT,TRY_CAST,SAFE_CAST,RLIKE,REGEXP,OVER,POINT,ST_DISTANCE,POSITION,POWER,REVERSE,CURDATE,CURTIME, the eightDATE*/DATETIME*aliasesBoth lists are reproduced by the new generator. The page also omitted 186 recognised words entirely
(145 of 331) and had 17 date/time functions commented out with
[//]: #.It drifted in both directions because it was a grammar-keyword list wearing a reserved-word title —
one dimension where the engine has two. Story 22.2 made that load-bearing:
EXISTSis reserved, soSELECT exists FROM tis a parse error, whileANYandSOMEwere deliberately left unreserved so acolumn named
anykeeps parsing.And two dimensions turn out not to be enough either. Executing every row through the parser — which
is what the generator now does — found four words that are neither reserved nor usable:
CURDATE,CURTIME,NULLandRANDOMare shadowed by a zero-argument function or a literal.SELECT curdate FROM treturns the current date, silently, instead of yourcurdatecolumn. The old page listedtwo of them under "Date/Time functions" and said nothing; a
yes/nopage would have printednoandbeen actively misleading. They now carry
no (shadowed).What is generated, and how to regenerate
Only the region between
<!-- BEGIN GENERATED KEYWORDS -->and<!-- END GENERATED KEYWORDS -->.Everything outside is prose, read back verbatim and preserved — including the framing PR #346 wrote.
Discoverable from three places: the generated banner at the top of the region, the guard's failure
message, and the page's own "How to read the table" section — and
KeywordsPageSpecpins the commandstring against the alias
build.sbtactually declares, so the instruction cannot become a lie.Sources of truth, none copied:
SQLKeywordsfor the words,Parser.reservedKeywordsfor the reservedflag, the parser itself for the shadowed flag. The
Kindcolumn is derived from the token'spackage, never from the word, so a new keyword needs no edit to the generator — and an unmapped
package throws, naming itself.
Result: 331 keywords (134 reserved, 4 shadowed), 11 compound phrases, 1 symbolic operator. The
52-entry divergence is subsumed: once every word carries its true flag, the two directions of drift
cannot exist.
Three words carry the kind
reserved onlyrather than being guessed at or dropped —CURRENT_DATETIME,FORMAT_DATE,FORMAT_DATETIMEare reserved but backed by no grammar surface (thesame three
SQLKeywordsSpecalready pins). Nothing else resisted classification.The guard, and its RED proof
KeywordsPageSpec, 25 tests, no allow-list, noassume(anassume-cancelled suite exits 0, and thissuite IS the mechanism). It owns four failure modes that cannot see each other: the page is stale; the
generator is short or inventive; the render misrepresents a correct model; or the page makes a
claim the parser contradicts.
Falsified in two rounds. Every mutation was run, restored by bytes, and the restore asserted.
Round 1 — 12 mutations, all RED: flip a reserved cell · flip a shadowed cell · delete a row · drop
the
::row · delete theWITHIN GROUPconstruction · restore a hand keyword list outside the markers· rename the sbt alias · re-add the duplicate reserved word · invent a row · collapse a kind label ·
drop the per-row kind sort · convert the page to CRLF.
Round 2 — the sharp one: six generator faults, with the page regenerated to agree with them.
statementWordsReservedcolumnThe guard cannot be satisfied by regenerating. That round is also how the guard's one real blind
spot was found: the kind-vocabulary assertion stayed green when a whole surface was relabelled,
because another surface still contributed the label. The kind mapping is now pinned in both
directions, derived entirely from the registry.
And the oracle caught a flaw in itself:
SELECT true FROM trenders identically whethertrueis aliteral or a column, so a round-trip test in the value position reports
TRUE/FALSEas usable barenames. The reserved half of the oracle therefore runs in the alias position (
SELECT c AS <word>),which
regexAliasgates from the same list and which admits no literal reading; the shadow half staysin the value position, where shadowing is what happens.
Idempotency: starting from an emptied generated region, two separate sbt invocations (two JVMs)
produce byte-identical files equal to the committed page —
sha256 eb2a2dec…. Nothing is emitted inSetiteration order; every table issorted, and everytoUpperCase/toLowerCasein the new codepasses
Locale.ROOT.Every SQL claim the prose makes is executed:
SELECT exists FROM tasserted to be rejected;SELECT any, some FROM t WHERE any = 1,SELECT "exists" FROM t,SELECT `exists` FROM t,SELECT t.from FROM t,SELECT doc.count FROM t,SELECT foo-bar FROM tasserted to parse;SELECT in-stock FROM tasserted to be rejected (a bare name that merely starts with a reserved wordis also rejected — measured, and now documented); the four multi-keyword constructions asserted to
parse; and the normalisation claims asserted on the whole rendered statement, never with
include—
include("IN (SELECT")is satisfied byNOT IN (SELECT, i.e. by the inverse of the claim.Beyond the page: three things in the diff that are not the page
Parser.reservedKeywordsprivate→private[sql]. Nothing outside thesqlpackage can see itand no production code reads it; the alternative was scraping
Parser.scalaas text, which isassume-guarded where it already exists and cannot be the basis of a generated artefact.add a public accessor to the
Parsertrait's interface — binary-incompatible for a mixer, acceptableon the already-incompatible
0.24.0-SNAPSHOT. Verified it disturbs nothing reflective:HelpCorpusSpec's twoParser.getClass.getMethodsscans filter on aPackratParserreturn type.reservedKeywordshad"on"twice. Harmless as a regex alternative, invisible once it reaches aSet— but this list is a documentation source now, so the duplicate is removed and the guard assertsthe list is duplicate-free.
SQLKeywordsSpec's anchor scan is hardened to demand exactly one occurrence. Two traps were hitand fixed while widening the visibility, both now recorded in the scaladoc at the declaration: a
type ascription between the name and the
=makes the raw-text scan miss, and a comment quotingthe anchor whole gets matched instead of the declaration. The second failed with
Set() was empty, which names neither cause.documentation/sql/keywords.mdalso gains a-textrule: it is now asserted byte for byte, andcheckout-time eol conversion on a
core.autocrlf=trueclone would redden the guard for the platformrather than the content, invisibly to macOS/Linux CI. The guard asserts the LF invariant too, because
regenerating a CRLF page yields a permanently mixed file that then passes everything else.
Merge order — this PR must land AFTER #346
#346 (
docs/22-subqueries) edits this same file and must merge first. This branch is cut frommainand carries #346's keywords.md prose verbatim, so the two conflict on
documentation/sql/keywords.mdand the resolution is take this branch's version — it already contains #346's text. Rebase onto
mainafter #346 lands, re-run
sbt "sql/testOnly *KeywordsPageSpec", then merge.known_limitations.md#subqueries-and-derived-tables, and that heading is created by #346. Checkedagainst
mainthe anchor looks dead; checked againstorigin/docs/22-subqueriesit resolves(
known_limitations.md:76). Do not retarget or remove that link — the pre-#346 section it wouldpoint at instead is renamed by #346 and says subqueries do not work, which is the claim #346 exists to
delete. Every other link on the page was verified against
main:dql_statements.md#quoted-identifiers(heading at
:91),README.md, and the in-page#compound-phrases/#multi-keyword-constructionsanchors all resolve.
documentation/sql/README.md:16andoperators.md:5link this page by filenameonly, so deleting the old
## Main clauses/## Aggregatesheadings broke no inbound anchor.No
Closesline for #345 — #346 owns it; this PR only carries the keywords.md half of its prose forward.Web twin — deliberately NOT in this PR
softclient4es-webhas no keywords page today and nothing links to one, so nothing is left broken.Generating the MDX twin is a materially different job, not a formatting variant: it needs Starlight
frontmatter, a sidebar entry in
astro.config.mjs, absolute/sql/<page>/routes instead of relative.mdlinks, and MDX escaping — and, decisively, it is a second repository with no Scala build, so anelasticsql test can neither write it nor guard it. A generator that writes across a repo boundary is
unguardable from either side, which is the failure this PR exists to end.
Hand-copying the list into MDX is explicitly rejected: it would create exactly the second drifting copy
this change removes. The follow-up that does work — for a separate issue — is for this generator to also
emit a machine-readable artefact (e.g.
keywords.json) that the web repo's own build consumes, so theMDX page is generated there from the same source of truth.
Verification
sbt "+ compile"— green on 2.13.16 and 2.12.20 (main sources; CI's cross gate).sbt "sql/test"— 1269/1269.sbt "core/test"— 1057/1057. Both 2.13, as CI runs them.sbt "++ 2.12.20 sql/Test/compile"— green (covers the two 2.12-only lexer traps even though CI nolonger compiles 2.12 test sources). The one 2.12 red in
coreis the pre-existingSchemaCacheTtlApiSpec19/20, green on 2.13.sbt headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck— green (CI's lint job verbatim).sbt "sql/clean" regenerateKeywordsPage— the documented command works on a cold tree.