feat(sql,core): correlated subqueries — cross-scope resolution and routing to the relational engine (story 22.3, elasticsql half)

[fupelaqu]

🔴 Release gate (spec PD-9), discharged rather than forgotten. This PR ships 22.3a and task B1 together, on the lead's OQ-2 ruling. B1 flips a correlated subquery from a parse-time rejection to an engine-routed statement, and the arrow half that executes it has not merged — so between this merge and arrow's 22.3b a correlated statement at a REPL with extensions is refused with a message telling the user to install a jar they already have. That is acceptable only because no release is cut in that window: the lead has ruled that no 0.24.0 goes out before the end of epic 22 and its usual train, by which time arrow's 22.3b has merged. Details in the release notes below.

Closes #340

What this is

Story 22.3, elasticsql scope only: all of the spec's [a] (A1–A5) plus [b] task B1. The arrow and jdbc halves of [b] — DuckDB decorrelation, the classifier, the pushdown, the capability flags — are separate work in softclient4es-arrow and softclient4es-jdbc.

WHERE EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id) and its family used to be rejected at parse with "story 22.3 … not executed yet". They now parse and route to the relational engine; a venue without the engine refuses them loudly, naming the construct.

The design in one paragraph

One scope model in sql — SubqueryScope.Scope / chain / resolve / ScopeSource — as a post-pass over the AST the parser already annotated, not a new Identifier field. Annotating in update() would have cost an arity change on a 13-field case class read by three extensions files, a second place where tableAlias/table are set, and an arm whose input (the enclosing statement) is not in hand there at all. The unresolved-qualifier-left-in-name signature already is the annotation. The flip itself is two lines: SubqueryCriteria.commonChecks loses its correlated Left, and SingleSearch.relationalClosureRequired gains || hasCorrelatedSubqueries. Every venue inherits with zero edit — CoreDqlExtension, the searchAs macro abort, JoinExtension.canHandle, and the resolveWithSchema seam, which already read the statement-level predicate (so OQ-5's "one identifier" turned out to be zero).

Three defects found on the way, each measured

1. correlationNames was blind to a collapsed alias — a silent wrong answer. It read From.tableAliases, a ListMap keyed by TABLE that has already collapsed two sources sharing an aliasKey (21.2 AD-6 / BIDC-8). On FROM orders o JOIN UNNEST(o.orders) AS i it is ListMap(orders -> i): alias o is simply gone, so a body's o.region was not reported as correlated, the statement did not route, and Elasticsearch read o.region as an object path — zero rows, HTTP 200. It now derives from scopeOf, which reads the lossless aliasesToTable, which is what makes "one resolver" true rather than aspirational. Found by the independent review. Pinned by a regression row and by the converse agreement property — every reference the resolver places in an enclosing scope must have been reported by the detector; the forward direction alone is satisfied by a detector that reports nothing.

2. The LATERAL walk is load-bearing for B1, not a nicety. A derived table nested inside a WHERE-subquery body was caught only as a side effect of the correlated rejection this PR deletes. Extending lateralOffenders to descend into subquery bodies is what keeps the deletion from turning a loud rejection into an accepted statement.

3. LATERAL never reached the user. GatewayApi.excerpt (#262) caps a parse-rejection reason at 200 characters and keeps head(120) + ... + tail(77) — it elides the middle. lateralMessage was 241 characters and the elision landed exactly on "this is LATERAL, which is not supported". No unit test could see this: none of them goes through GatewayApi. Found by executing against real Elasticsearch 8.18. The message now leads with the construct name, and DerivedTableSpec (22.1) / WhereSubquerySpec pins were retargeted — contract unchanged, order fixed. Generalises: the load-bearing terms of any validate() message must fit in the first 120 characters.

Also fixed here

++ 2.12.20 core/Test/compile was red on main — verified with a control worktree at 10f5e4d5, same seven errors. client.StringValue / client.BooleanValue shadow the sql ones and Scala 2.12 prefers an enclosing-package member over an explicit wildcard import; no CI job and no documented build line compiles core test sources on 2.12, which is how it survived 22.2. Fixed with aliased imports.

Parse cost (no regression)

Measured against an interleaved same-session control, never a quoted number. Two worktrees, alternating, 5 runs each, first (cold) run discarded:

pass	control (main @ 10f5e4d)	branch
1	994 / 959 / 910 / 1003 — median 976 ms	866 / 860 / 822 / 867 — median 863 ms
2	891 / 872 / 825 / 852 — median 862 ms	907 / 877 / 825 / 841 — median 859 ms

Pass 1's control is a warm-up artefact of a freshly compiled tree; pass 2 is the honest reading: 862 vs 859 ms, −0.3 %, deep inside noise.

Verification

sql 1237 · core 1057 · macrosTests 24 · bridge + testkit compile · + sql/compile + core/compile ++ 2.12.20 sql/Test/compile ++ 2.12.20 core/Test/compile · CI lint line, with scalafmtCheck falsified (a deliberate mis-format reddens it) rather than trusted. Real clusters: ES 8.18 71 REPL + 16 completeness, ES 7.17 71 REPL. ES 6.8 / 9.0 left to CI — the diff touches no client module.

Every new rejection was proved able to fail, one mutation at a time: dropping the WHERE-subquery lateral descent reddens 2 rows; reverting the closure predicate reddens 10; removing the resolver's defensive arm reddens 1; removing the unresolved-qualifier check reddens 1. Every rejection row also asserts not startWith Parser.InternalParseFailure, so a restored throw cannot masquerade as a grammar rejection (21.4's lesson).

Release notes

• ⚠️ Behaviour change. A correlated WHERE subquery now parses. At a venue with the relational engine it will execute (once arrow's 22.3b lands); at every other venue it is refused with HTTP 400 naming "A correlated subquery" and softclient4es-arrow-extensions, where it used to be refused with "Correlated subquery … story 22.3 … not executed yet". Anything matching the old text breaks. DELETE / UPDATE by query with a correlated WHERE get the same 400 — they never ran.
• ⚠️ The window until arrow 22.3b merges. MEASURED on arrow main: JoinExtension.canHandle still reads s.from.enrichmentRequired (22.4 has not merged there either), so a correlated no-JOIN statement is not claimed by the join extension at all — it falls through to CoreDqlExtension and the user is told to put softclient4es-arrow-extensions on a classpath that already has it. Loud, but a misleading dead end at the very venue meant to run the shape. (This differs from the 22.3 spec's PD-9 prediction of No JOIN found in query, which assumed 22.4 had landed in arrow.) Acceptable only because no release ships in this window.
• ⚠️ The LATERAL rejection text changed — it now leads with LATERAL is not supported:. Any downstream fixture pinning the old wording needs updating.
• ⚠️ Binary-incompatible sql changes ⇒ downstream rebuild on the open 0.24.0-SNAPSHOT train: SubqueryScope.correlatedMessage deleted (dead once the rejection moves), and new public types SubqueryScope.{Scope, ScopeSource, PlainSource, DerivedSource, UnnestSource, Resolution, Resolved, Ambiguous, Unresolved}. No arity change to any existing case class; GenericIdentifier stays at 13. SearchApi.bareNameCorrelation became private and scopeCorrelation took its place — both private[client], so no published surface moved.
• A qualified name inside a subquery body that resolves in no scope, and is not a mapped object field of the inner index, is now a 400 that names the scopes it searched. An empty inner mapping never triggers it — a mapping gap is not evidence a column is absent.
• Docs: select.json's notes[] and known_limitations.md (PD-7). The web mirror is a hand-off, not opened here.

Not in scope / hand-offs

• arrow 22.3b (B2–B8) and the jdbc row + supportsCorrelatedSubqueries. 🔴 The one fact the arrow half must know: hasCorrelatedSubqueries is relative to a scope chain, so asking an isolated body gives the wrong answer — read the top-level node's correlatedRefs, or walk with scopes accumulated as correlatedReferences does.
• The two-level-out integration row is deliberately pending on arrow#144 (two legs over one index) when 22.3b lands — named, never silently green.
• Criteria.embeddedStatements / mapEmbeddedStatements do not exist yet (22.5 has not landed), so the body walks go through 22.2's whereSubqueries / inner. No third Criteria walk was added. 22.5 must pin subqueries.map(_.query) == embeddedStatements when it lands.
• known_limitations.md's "What a not-yet-supported query looks like" block is still stale (it says a WHERE subquery is rejected by the parser — false since 22.2). That state was explicitly accepted by the lead; owner: 22.7.

🤖 Generated with Claude Code