Uncorrelated WHERE subqueries (IN / EXISTS / scalar / quantified) — elasticsql - #337
Merged
Merged
Conversation
…/ quantified Story 22.2. The four ANSI-92 uncorrelated WHERE-subquery forms parse, render back to themselves, and execute ES-natively at every venue: the inner statement runs first at the single `SingleSearch -> ElasticQuery` seam and the predicate is rewritten into the literal form the bridges already emit (`terms`, a literal comparison, `match_all`, `match_none`). Correlated subqueries are refused with a named message; ANSI NULL semantics hold for IN / NOT IN, the scalar form and every quantifier, including the two opposite empty-set rules (ANY over nothing is FALSE, ALL over nothing is TRUE). Also in this change, both lead-directed: * An aggregate over ZERO documents now answers ANSI NULL on every major. Measured on real clusters, Elasticsearch answers identically on the wire and ES 8 alone converted the null to `0.0`, because that module reads the typed response and a primitive `double` swallows the null inside the vendor's model. The null is recovered from the document count, in core, so one rule serves every client. COUNT stays 0; SUM keeps Elasticsearch's 0.0 by recorded decision. * Packrat memoisation was defeated grammar-wide: the library memoises on the parser INSTANCE and every `WhereParser` production was a `def`. Declaring `any_identifier` and `criteria` as `lazy val` takes `ParserSpec` from 3.490 s to 1.240 s median (-64 %), verified to change no verdict over 4,824 inputs parsed by both trees. Closes #335 Closes #336 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fupelaqu
marked this pull request as ready for review
September 15, 2026 05:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uncorrelated WHERE subqueries —
IN,EXISTS, scalar and quantified — now parse and execute ES-natively at every venue, including the plain REPL. The inner statement runs first at the singleSingleSearch -> ElasticQueryseam (SearchApi.resolveWithSchema) and the predicate is rewritten into the literal form the bridges already emit.Two further changes ride along, both directed by the lead: aggregates over zero documents now answer ANSI NULL on every major, and a grammar-wide packrat-memoisation defect is fixed (parse cost −64 %).
Closes #335
Closes #336
What ships
<id> [NOT] IN (<body>),[NOT] EXISTS (<body>),[NOT] <id> <op> (<body>)for= <> != >= > <= <, and<id> <op> ANY|SOME|ALL (<body>). Bodies may end in WHERE or HAVING. Nothing is newly reserved —WHERE any = 1andSELECT some FROM tstill parse; what makes the quantified form win is the alternation ORDER.InSubquery/ExistsSubquery/ScalarSubquery/QuantifiedSubquery, plus the resolved sentinelsMatchAllCriteria/MatchNoneCriteria.= ANY|SOMErenders canonically asINand<> ALL/!= ALLasNOT IN;SOMEcanonicalises toANY. Every accepted shape is a parser fixed point (Parser(stmt.sql) == Right(stmt)) with its rendered text pinned.index.max_terms_count(65,536). A bare-column body runs as a bounded terms aggregation; atextordateinner column, and everything else, runs as written. Nested subqueries resolve recursively. An inner failure propagates with ITS status and cause.INignores inner NULLs,NOT INover a NULL-bearing set returns no row, a scalar over zero rows matches nothing,ANY/SOMEover an empty set is FALSE whileALLover an empty set is TRUE, and a NULL makes everyALLform — and every negated form — UNKNOWN.UNION ALLand FROM-less bodies, more than one projected column in IN / quantified / scalar position, a scalar body that is neither metric-only norLIMIT 1, and subqueries in HAVING / CASE / JOIN ON / MATERIALIZED VIEW / WATCHER.Routing does not move: such a statement is passthrough, and the licensed row cap still binds on the OUTER rows — the inner statement is an internal execution step and is not quota-checked.
Release notes
maincarried the released0.23.0, so this branch opens0.24.0-SNAPSHOT.sql): newCriteriasubtypes (an exhaustive match downstream needs new arms) and newoperatortokens (EXISTS,ANY,SOME,ALL).SingleSearcharity is unchanged. Downstream rebuild required.WHERE x = ANY (…)/= SOME (…): these used to parseANY/SOMEas a column name and then fail; they are now the quantified form. No word is newly reserved, and a column namedanyorsomekeeps working.> ANYreduces to> min,> ALLto> max(and symmetrically for>=,<,<=);= ALLholds only when the inner set is a single value;<> ANYis its negation. Over an EMPTY subqueryANY/SOMEis FALSE andALLis TRUE. A NULL among the inner values makes everyALLform and every negated form match nothing.SELECT MAX(x) FROM t WHERE <no match>previously answered0.0on ES 8.18 and an absent column on 6.8 / 7.17 / 9.0; it now answers NULL everywhere, as an explicit null column.COUNTis unchanged: 0, never NULL.SUMis unchanged: it keeps Elasticsearch's own0.0— a deliberate divergence from ANSI (which says NULL), because the engine computes that value on every supported major and synthesising NULL would override the engine everywhere and silently flipSUMfor every existing consumer. Any statement, dashboard or downstream consumer that relied on the old0.0forMIN/MAX/AVGwill see different results. The rule lives in one place,ClientAggregation.nullOverEmptyInput.SearchApi.resolveWithSchemanow executes inner statements synchronously on both the sync and the async path — on the thread that constructs the outer request, bounded at one round trip for a projection / scalar / EXISTS body and at ~66 scroll pages for a row-shaped one. Inner row-shaped scrolls materialise on the separate scroll-routing system, so there is no self-deadlock at any nesting depth.ElasticBridge.queryrefuses an unresolved subquery node with a namedIllegalArgumentException(was the genericUnsupported filter type).NESTED(…)/CHILD(…)/PARENT(…)now validate the criteria they wrap. Rules that were silently skipped inside a relation — includingIN's and an expression's own type checks — now apply, so a statement that parsed before may be rejected.formataccepts no ISO alternative used to be forwarded verbatim and rejected by Elasticsearch; it is now re-rendered in the column's own pattern.WHERE ts = '2026-06-04T00:00:00'against a"format": "yyyy-MM-dd HH:mm:ss"column now works. One contract pin was retargeted accordingly.too_many_buckets/max_terms_counttranslation is now on every search path and its wording covers both causes, so a plain high-cardinalityGROUP BYgets a clearer message than before.▶ Row query …INFO line andElasticResponse.sqlon the scroll route now carry the statement AS WRITTEN, assearchalways did — a resolved subquery can carry up to 65,536 literals.IN (SELECT …)is rejected by the parser. The prose sweep is held for the epic's documentation story; the in-repo help corpus (SELECTtopic) is updated here.Parse cost: 3.490 s → 1.240 s median (−64 %)
ParserSpec, 10 timed runs per configuration, same quiesced machine, median and range:main, pristine (control, same session)main+ the SAME two-word fix (control)PackratParsersmemoises on (parser INSTANCE, position) and the library's own scaladoc requires productions to belazy val; all 49WhereParserproductions weredefs, so no alternative could ever hit another's memo entry. Declaringany_identifierandcriteriaaslazy valis the whole fix.The
main-with-the-fix row is the one that matters: the win is a pre-existing, grammar-wide cost that every statement the engine has ever parsed was paying — not a credit this story's grammar earns. Against that control this story's four new alternatives cost +0.95 % (1.228 → 1.240), with fully overlapping ranges, i.e. below the suite's measurement resolution. No timing assertion was added.Folding the subquery forms into
equality/comparison/inLiteralas RHS shapes was considered and deliberately not done: it targets ~1 % that cannot be measured above noise, at the cost of a large diff into the three hottest shared productions.Narrowing check. Because this touches shared productions, the parser was built from BOTH trees and fed the same 4,824 distinct inputs — every census and help-corpus statement from
main, every prefix truncation at token boundaries (a broad refusal corpus), plus stray-paren, dangling-ANDand malformed-keyword variants — and the verdicts diffed: 0 differing, 0 narrowed, 0 widened, 0 throws.Verification
sql1208,core1049, bridge template 206, es6 bridge 206, es9 bridge 206,macrosTests24 — all green;+ sql/compileand+ core/compilegreen on both Scala legs.headerCheck scalafmtSbtCheck scalafmtCheck Test/scalafmtCheckgreen.WhereSubqueryCompletenessSpec(3-shard two-index fixture, closed-form oracles) 16 tests — one row version-gated because ES 6.8 does not enforce a per-indexmax_terms_count; the sibling completeness guards (GroupBy,Select,WindowPartition,Limit,Scroll) 73 tests; andReplGatewayIntegrationSpec70 tests, including the new WHERE-subquery section.