fix(sql): bound the LATERAL walk to the derived table, and resolve names case-insensitively

[fupelaqu]

Closes #342
Closes #343

Two defects in core 0.24.0-SNAPSHOT, both introduced by story 22.3a (PR #341, merged today) and
not released. The version line is untouched.

---

#342 — a false-positive LATERAL rejection made story 22.3's own AC 11 unreachable

Reproduction (measured on origin/main 36f0884e, before any edit)

SELECT d.id FROM (SELECT o.id FROM orders o
                  WHERE EXISTS (SELECT 1 FROM returns r WHERE r.oid = o.id)) d

Left(ParserError(LATERAL is not supported: a derived table cannot reference an outer alias.
'o.id' inside derived table 'd' reads the enclosing FROM (SQL-92 §7.6). Move the condition to the
outer WHERE, or write it as a correlated WHERE subquery.))

SubqueryScope.correlationNames(outer) = Set(d)
SubqueryScope.lateralOffenders(outer) = List((d, o.id))

o is declared by d's own body. The offender set says so — it holds only d — and the walk
reported o.id anyway.

Cause

lateralOffenders reused correlatedReferences, whose recursion accumulates the enclosing names as
it descends (deeper = outerScopes ++ innerNames). Accumulating is right for the CORRELATION
question — a body reading any enclosing scope must be reported at the outermost node, where routing
is decided — and wrong for the LATERAL question, where the boundary is the derived table and must
not move. By the time the walk reached the WHERE subquery nested inside d's body, d's own o
had been folded into the "outer" set one level up.

Fix

One walk, escapingReferences(body, outerScopes, accumulate); the boundary rule is the parameter, so
the two questions cannot drift into two traversals. correlatedReferences is accumulate = true
(unchanged behaviour); lateralOffenders is accumulate = false, where the set only ever SHRINKS by
the names each level declares.

What stays rejected, and the pin that proves it

shape	verdict
JOIN (SELECT o.cid FROM orders o WHERE o.cid = c.id) d	still LATERAL, same message
FROM (… WHERE o.cid = customers.id) d, customers	still LATERAL, same message
a derived body nested in a WHERE-subquery body reading c.id	still LATERAL, same message
two derived levels in (… (SELECT y FROM (… WHERE o.cid = c.id) e) d)	still LATERAL, attributed to e
a body reading the derived table's OWN enclosing alias (… WHERE u.k = d.k) d)	still LATERAL
AC 11's nested-derived shape	parses, relationalClosureRequired == true

DerivedTableSpec's two pre-existing LATERAL rows needed no retargeting — they are the
"still rejected" direction and stayed green byte-for-byte.

---

#343 — SubqueryScope.resolve narrowed derived-projection matching to case-sensitive

Reproduction (measured on origin/main 36f0884e)

Retiring the arrow planner's resolveUnqualified (which matched with _.equalsIgnoreCase(name)) in
favour of resolve (which matches with Seq.contains) narrowed resolution in two ways:

SELECT Total FROM orders o JOIN customers c ON o.id = c.id
                           JOIN (SELECT SUM(x) AS total FROM t) d ON o.id = d.total
  resolve(Total) = Ambiguous                               -- the reported "Ambiguous column"

SELECT Total FROM bi_events JOIN (SELECT SUM(amount) AS total FROM bi_events) AS d
                            ON bi_events.id = d.total
  resolve(Total) = Resolved(0, PlainSource(bi_events,bi_events), Total)   -- the WRONG leg, silently

SELECT D.total FROM (SELECT amount AS total FROM t) d
  resolve(D.total) = Unresolved                            -- `Scope.byName` compared with `==`

The halves already disagreed: the consumer maps the answer back to a TableInfo with
alias.equalsIgnoreCase(source.alias), so the ALIAS side was case-insensitive while the PROJECTION
side was not.

Fix, and the quoted-identifier decision

Scope.byName and the projection lookup both match case-insensitively, restoring what the retired
helper did. derivedScopeCheck now takes its column head from Resolved.column instead of
re-deriving it from Identifier.name — without that, the widened byName would have compared the
QUALIFIER (D of D.total) against the projection and refused a statement resolve had just
resolved.

A quoted name is matched the same way, deliberately. ANSI would compare a delimited identifier
exactly, but (a) the projection side carries no quoting information — DerivedTable.outputNames is a
Seq[String] — so an exact rule could only be applied to one half, and "Total" would then resolve
against Total exactly while total resolved case-insensitively; and (b) the helper this restores
was case-insensitive for every operand, quoted or not, so exempting quoted names would be a NEW
narrowing with no mandate. Nothing is rewritten by the choice: Resolved.column carries the caller's
own spelling, so the column reaches DuckDB / Elasticsearch exactly as written — this decides WHICH
SOURCE owns a name, never how it is spelled. Pinned.

What stays rejected

resolve(nope) over two plain legs is still Ambiguous; SELECT d.nope FROM (SELECT amount AS total FROM t) d still gets "Column 'nope' is not projected by derived table 'd' (it projects: total)".

---

Third fix — an UNNEST leg changed what a bare name resolved to (no separate issue: same regression as #343)

🔴 This carries no Closes #N of its own. It is the same retirement regression as #343 — story
22.3 replaced the planner's resolveUnqualified with SubqueryScope.resolve and the replacement was
narrower in three places, of which #343 names the case-sensitivity. Folded in here on the lead's
ruling rather than minted as a third issue, so the PR's issue coverage stays honest: two issues, three
regressions, all from one change.

Reproduction (measured on this branch with the first two fixes already in)

FROM (SELECT * FROM x) d JOIN UNNEST(d.items) i
  sources     = List(DerivedSource(d,None), UnnestSource(i,items))
  resolve(a)  = Ambiguous

FROM (SELECT * FROM x) d                        -- the same shape without the UNNEST
  resolve(a)  = Resolved(0, DerivedSource(d,None), a)

resolve's own scaladoc already promised that an UnnestSource "takes part in NO un-qualified
rule". Rule (0) — a lone source owns every bare name — counted every source, and scopeOf
emits the UNNEST as a source of its own; the retired helper counted TableInfos, and an UNNEST is
part of one. The code contradicted its own documented contract, in a resolver three repos read.

It was not only a lost resolution. A derived table beside an UNNEST stopped being scope-checked at
all, so the two halves of a twin pair disagreed:

SELECT a FROM (SELECT b FROM x) d                            -> "Column 'a' is not projected by
                                                                 derived table 'd' (it projects: b)"
SELECT a FROM (SELECT b FROM x) d JOIN UNNEST(d.items) i     -> accepted (nothing resolved, so
                                                                 nothing checked it)

Fix

One line: the UNNEST sources are dropped before rule (0), not only inside rules (1)–(3). The
prose was already correct, so it is unchanged — that was the point.

Both directions pinned

shape	after
FROM (SELECT * FROM x) d JOIN UNNEST(d.items) i, bare name	resolves to d — same as the UNNEST-free control
SELECT a FROM (SELECT b FROM x) d JOIN UNNEST(d.items) i	refused identically to its UNNEST-free twin
bare name over two plain legs + an UNNEST	still Ambiguous
bare name over a plain leg + an opaque derived leg + an UNNEST	still Ambiguous
correlationNames / byName / unresolvedMessage	UNNEST still a source everywhere else — pinned

The "name on both sides" row the brief asked for is not representable, and that is the answer.
UnnestSource.projection is None by construction — nothing but a mapping could know the unnested
element's columns — so a bare name can never be known to exist on both. The nearest reachable shape
is pinned instead: SELECT items FROM (SELECT items FROM x) d JOIN UNNEST(d.items) i, where the
derived table projects the very column being unnested. It resolves to d, and that is the right
owner: d.items is the array, i is the alias of its element. Qualified resolution still
reaches the element (byName searches every source, i.name resolves) — only the un-qualified rules
do not.

Neighbours checked (the derivedScopeCheck lesson, applied)

Every consumer of resolve / scopeOf / Scope.sources in sql/src/main and core/src/main was
enumerated:

site	reaches rule (0)?
SingleSearch.derivedScopeCheck (sql)	yes — pinned, both directions
SearchApi.unresolvedQualifier (core)	no — it pre-filters to id.name.contains("."), so it only ever takes the qualified branch
Scope.names / correlationNames	no — built from sources.flatMap(key, alias), untouched, so story 22.2's alias-collapse pin (which requires the UNNEST alias) stays green
SubqueryScope.unresolvedMessage	no — lists every source in "Scopes searched", correctly including the UNNEST, since qualified resolution does search it
Scope.byName	no — searches every source, untouched

Nothing else re-derives a source count: the other .sources hits in the tree are
SingleSearch.sources (index names from FROM), a different member.

Falsification — the sixth perturbation

Reverting the one line reddens exactly the two new rows (SubqueryScopeSpec + DerivedTableSpec),
68 of 70 still green; restored after.

---

Falsification — every new assertion was made RED before being trusted

Each half of each fix was reverted in turn, the suites re-run, and the fix restored:

perturbation	red
lateralOffenders back to accumulate = true	2 (SubqueryScopeSpec + DerivedTableSpec)
projects back to Seq.contains	2 (both specs)
Scope.byName back to ==	1 (SubqueryScopeSpec)
derivedScopeCheck back to a case-sensitive contains	1 (DerivedTableSpec)
derivedScopeCheck back to re-deriving the head from id.name	1 (DerivedTableSpec)
resolve rule (0) back to counting UNNEST sources	2 (both specs)

Every rejection assertion also asserts not startWith Parser.InternalParseFailure (story 21.4:
a rejection test is unfalsifiable once a boundary catch exists) — the shared rejects helper.

---

Verification

leg	result
sql/test	1244 passed
core/test	1057 passed
macrosTests/test	24 passed
softclient4es7-sql-bridge/test (run alone)	206 passed
+ sql/compile / + core/compile (2.13.16 + 2.12.20)	green
++ 2.12.20 sql/Test/compile / ++ 2.12.20 core/Test/compile	green
headerCheck scalafmtSbtCheck scalafmtCheck Test/scalafmtCheck	green

Not run, deliberately: the Dockerised per-major ES suites. The diff touches only validate()
and the scope resolver — no emission path, no client module, no bridge template — so the cheapest
reproducing surface is the unit legs above, and the ES matrix is left to CI. No claim is made about
them here.

---

Blast radius

In this repo: no existing pin needed retargeting. The full sql suite (1242) is green,
DerivedTableSpec's two LATERAL rows and the SubqueryScopeSpec lateralReferences rows included.

softclient4es-arrow (PR #188, open, built against core WITH these defects): all twelve
statements its JoinPlannerSpec / JoinDetectorSpec / QueryPlannerSpec rows feed through core
were re-parsed on this branch — every one keeps core's verdict (Right). Its four
"Ambiguous column 'total'" / 'nope' / 'Total' pins and the AD-4 equivalence row all use
all-lowercase identifiers, so the case-insensitivity widening cannot reach them; the LATERAL fix only
widens what parses, and a shape that never parsed cannot have been asserted. grep -l LATERAL
over the feature/22.3 branch returns nothing but a local findings note.

Re-run with the third fix in, since JOIN UNNEST is live arrow territory: the check was widened to
every UNNEST-bearing statement its JoinPlannerSpec / JoinDetectorSpec / PredicatePushdownSpec /
WriteWithJoinExecutorSpec rows feed through core — the derived-plus-UNNEST refusal row
(SELECT d.id, i.product FROM (SELECT id, items FROM orders) AS d JOIN UNNEST(d.items) AS i), the
UNNEST-alias attribution rows, the UNNEST-alias collision row, the pushdown row and the
INSERT … JOIN UNNEST(raw.items) AS item row. All keep core's verdict. The one row that rejects,
SELECT * FROM orders JOIN UNNEST(items) AS item ("UNNEST identifier items must be a nested field"),
was control-run with all three fixes stashed and rejects identically — pre-existing, an
Unnest.validate rule this diff does not touch, and its arrow row classifies by regex without calling
Parser at all. Expected impact on #188: none — but it needs a core 0.24.0-SNAPSHOT republish before its CI sees this.

softclient4es-jdbc: no pin on either message. The single grep hit is a prose comment in
JdbcIntegrationSpec:1303 explaining the planner's "Ambiguous column" guard; the statement it
guards is unaffected.

---

For the lead

• Republish core 0.24.0-SNAPSHOT after merge — arrow fix: read local files without Hadoop in COPY INTO (JDK 23+) — R1FIX.4 / #183 #188 and the jdbc half both resolve
  against it.
• The third narrowing I surfaced is fixed in this PR per your ruling (its own section above),
  folded into SubqueryScope.resolve matches a derived-table projection case-sensitively #343 rather than minted as a third issue — say the word if you would rather it carried
  its own number.