perf(sql): restore packrat memoisation - 500 grammar productions from def to lazy val

[fupelaqu]

Closes #338

PackratParsers memoises on (parser INSTANCE, input position) and the library's own scaladoc requires each nullary production to be a lazy val. A def builds a fresh instance on every reference, so each reference site gets its own memo entry, no alternative can hit another's, and identical work is re-done at the same position — memoisation silently off for the whole grammar.

This PR changes declaration form only. 500 nullary PackratParser productions become lazy val. No alternation is reordered, no production body is edited. The visible line count is larger than 500 because lazy val is five characters longer than def , so scalafmt re-wrapped many bodies; git diff -w over sql/src/main shows nothing but the declarations and those re-wraps.

What was converted, what was left alone

	count	why
nullary PackratParser productions → lazy val	500	the fix
abstract PackratParser declarations left as def	2	derivedTableBodyInner and derivedTable are declared on trait Parser and implemented elsewhere. A lazy val may implement a deferred def, not the reverse, so the declarations stay def and both implementations are now lazy val
parameterised productions left as def	1	relationCriteria(relation: String) — each call is a different parser, so it cannot be memoised on identity
nullary Parser[...]-typed productions left as def	19	measured, bought nothing — see Candidates that bought nothing

Contribution by file group

Cumulative, measured after each batch with the committed ParseCostProbe (3 JVM runs per intermediate row, 5 for the endpoints). Marginal contributions are order-dependent — a later group can only claim what earlier groups have not already taken — so the order is stated, not implied.

#	group	productions	sum of medians (µs)	vs control	marginal
—	origin/main control	—	10,321	—	—
1	parser/WhereParser.scala	47	10,129	−1.9 %	−1.9 %
2	parser/Parser.scala	165	4,144	−59.8 %	−59.1 %
3	parser/type/package.scala	41	3,861	−62.6 %	−6.8 %
4	parser/function/** (time, aggregate, math, geo, string, cond, convert)	161	3,344	−67.6 %	−13.4 %
5	parser/time, parser/http, parser/operator/math	55	3,193	−69.1 %	−4.5 %
6	clause parsers (From, OrderBy, GroupBy, Select, Limit, Having)	31	3,173	−69.3 %	−0.6 %

Two rows worth reading honestly:

• Group 1 is nearly flat (−1.9 %) and that is the expected result, not a disappointment: the two genuinely hot WhereParser productions (any_identifier, criteria) were already lazy val on main. The other 47 were not where the cost was.
• Group 2 is essentially the whole win. Parser.scala holds the identifier, value and literal productions that every other production funnels through.

Before / after, against a control re-measured in the same session

Absolute parse timings drift between sessions, so the origin/main control was re-run on the same quiesced machine immediately before the branch numbers, using the same probe binary and the same JVM settings — 5 JVM runs per side, each reporting the median of 1,000 timed parses after 300 discarded warm-up iterations.

statement	control (µs)	branch (µs)	change
bare SELECT	193.9 [189–202]	59.8 [57–65]	−69 %
SELECT list + LIMIT	727.9 [692–754]	243.3 [232–248]	−67 %
WHERE-heavy	2,672.1 [2512–2947]	850.5 [800–889]	−68 %
GROUP BY + HAVING	1,899.4 [1828–1924]	628.6 [608–663]	−67 %
window function	580.6 [571–584]	200.2 [188–205]	−66 %
JOIN	762.6 [733–773]	262.4 [250–275]	−66 %
derived table	934.2 [900–967]	326.0 [301–337]	−65 %
WHERE subquery	988.5 [932–1004]	360.8 [351–409]	−64 %
CREATE TABLE	357.1 [349–368]	142.6 [136–157]	−60 %
rejection (malformed)	1,047.0 [1026–1049]	90.7 [82–99]	−91 %
sum of medians	10,163.3	3,165.0	−68.9 %

Every per-statement range is non-overlapping between the two sides.

Secondary instrument, ParserSpec wall time (ScalaTest "Run completed in", 5 runs per side, 315 tests green on both):

	median	range
control (origin/main)	1.774 s	1.660 – 1.868 s
branch	1.456 s	1.425 – 1.527 s

−17.9 %, ranges non-overlapping. The suite moves less than the probe because most of its wall time is fixture construction and ScalaTest overhead rather than parsing — which is exactly why per-statement latency is the number that matters to a user: every query they send pays it.

Zero behaviour change — demonstrated, not asserted

A branch-only suite cannot establish "no narrowing": the inputs a grammar change breaks are by construction the ones nobody wrote a test for. So the claim is backed by a differential replay, and the harness ships with the PR (GrammarDiffProbe).

• 15,601 inputs, frozen to a file so both sides see identical bytes: every SQL-shaped cell of the four corpus CSVs, every examples[].sql and syntax[] line of the help corpus, every SQL-shaped string literal in the sql test sources, every whitespace-boundary prefix of all of those, and targeted mutations (stray ( / ), dangling AND / OR, trailing comma, unterminated quote, misspelt WHERE / FROM / SELECT).
• Replayed through the parser built from origin/main and from this branch, recording per input: verdict, the ParserError message on rejection, and on success both the AST render (Statement.sql) and its toString.
• Result: the two output files are byte-identical — same SHA-256. 0 differing verdicts, 0 differing renders, 0 throws on either side. Distribution: 2,144 accepted / 13,457 rejected, identical on both sides.
• The probe was shown to fire. Removing one alternative (identifierWithArithmeticExpression) from any_identifier on the control tree produced 14 differing rows. A zero from an instrument never demonstrated to be non-zero is worth nothing.

New concurrency guard

Sharing one parser instance is the one hazard this change introduces, and no existing test covered it. ParserConcurrencySpec adds three:

1. 8 threads against a genuinely COLD object Parser, loaded in a child-first URLClassLoader with a null parent so every production is uninitialised when the threads hit it. This leg carries its own integrity gate — it asserts the class came back from that loader and fails loudly if delegation ever made it a second copy of the warm leg (falsified: pointing the loader's parent at the ambient classloader turns the leg red with that message).
2. The same race against the ambient Parser, compared to the single-threaded answers.
3. Cold and warm parsers must agree statement by statement.

Every leg is bounded by a timeout, so an initialisation deadlock fails the suite instead of hanging CI. The structural argument it checks is recorded in the file: a lazy val on an object initialises under that object's single monitor (no lock-ordering inversion possible between productions), and parser2packrat takes its argument by name, so a production's body is evaluated on first application, not on initialisation — which is what keeps the mutually recursive productions safe.

New parse-cost probe

ParseCostProbe is a runnable, non-asserting main under sql/src/test:

sbt "sql/Test/runMain app.softnetwork.elastic.sql.perf.ParseCostProbe"
sbt "sql/Test/runMain app.softnetwork.elastic.sql.perf.ParseCostProbe 300 1000 out.tsv"

It times ten statement shapes (bare SELECT, SELECT list, WHERE-heavy, GROUP BY + HAVING, window, JOIN, derived table, WHERE subquery, DDL, and a deliberate rejection), printing runs / median / min / p95 / max. It discards a global warm-up plus 300 further iterations per statement before timing, and it is locale-independent so the table stays machine-readable.

There is no timing assertion anywhere in the suite, by design. A wall-clock ceiling in CI is a flake liability — the previous ParseCostProbeSpec had to have its 1 ms ceiling relaxed to 10 ms (#269/#270) after it reddened two PRs whose diffs touched no parse path. This probe prints; it never fails.

Candidates that bought nothing (measured, then reverted)

The 19 nullary Parser[...]-typed productions (arithmeticExpressionLevel1/2, identifierWithValue, case_condition, ident, …). Converting them to lazy val avoids re-allocating the combinator tree per reference but enables no memoisation, since the packrat cache only ever sees a PackratParser. Measured at −0.9 % against the converted branch, with fully overlapping ranges — noise. Reverted: it widens the blast radius into the hottest shared productions for a number that cannot be measured. Recorded here rather than carried.

Two consequences worth stating

1. Parser$ gains 45 new public methods, and loses none. They are all name-mangled trait-private accessors (app$softnetwork$…$MathParser$$abs() and friends): a trait-private def compiles to a private method, while a trait-private lazy val needs a mangled but public accessor in the mixing class. Verified by diffing javap output of Parser$ built from both trees — 45 added, 0 removed, 0 changed, and every added entry carries $$. So this is additive and no source-level API moves. Anything that mixes in the parser traits (nothing in this repo or the sibling repos does — they use the Parser object) would need a rebuild.
2. One test fixture had to move with it, and it is a genuine coupling rather than an incidental edit: HelpCorpusSpec's reflection scan enumerates productions returning a sealed trait, and WhereParser's trait-private subqueryBody became visible to getMethods for the reason in (1). It is added to the expected set with that explanation; it names no new statement leaf (its body is derivedTableBodyInner in parentheses) so the AST package walk's coverage is unchanged. DialectCensus's source anchors, which quote declaration heads verbatim, were retargeted the same mechanical way (def x: → lazy val x:) — that file's "anchor resolves exactly once" assertion is what caught them.

Verification

Run on the final tree, 2.13 only, exactly as CI does (+ compile is main sources on both legs; CI never compiles or publishes 2.12 test sources):

gate	result
sql/test	1211 passed
core/test	1049 passed
macrosTests/test	24 passed
softclient4es-sql-bridge/test (template)	206 passed
es6bridge/test	206 passed
softclient4es9-sql-bridge/test (under the JDK-17 wrapper)	206 passed
+ sql/compile, + core/compile	green on 2.12.20 and 2.13.16
headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck	green, with the scalafmt cache cleared and the file enumeration verified non-empty (2 sbt / 91 + 91 main / 65 + 47 test sources actually checked)

Elasticsearch integration tests were not run, and that is deliberate. This PR changes no emitted query, no client code and no module outside sql/src/main/scala/.../parser plus three test files; the bridge suites above are the layer that pins the generated Elasticsearch JSON, and they are green in both copies. The differential replay is a strictly stronger statement about the grammar than any integration run could make. CI's own matrix covers the rest.

sql/Test/headerCheck still reports 31 files without licence headers — all pre-existing, none of them added here (the three new test sources carry headers). CI runs the Compile-scoped headerCheck, which is green.

Follow-ups spotted and NOT taken

• Parser.apply does an unconditional Console.err.println(msg) on every rejection, from the sql module — noisy stderr inside an embedded host. Pre-existing, unrelated to this diff, untouched here.
• Parser.apply discards the NoSuccess reader, so the offset that would let a client show a caret is thrown away one frame before any caller could use it. Pre-existing.
• The 19 Parser[...]-typed productions above remain def. If a future change makes any of them hot, the measurement to redo is in this PR body.