feat(table): adopt arrow-go 18.8, fast-path shredded variant extract, restore parquet dict fallback

[nssalian]

Rationale for the change

Bumps arrow-go to v18.8, uses its new compute.VariantGet for variant extract with a zero-copy fast path for shredded fields, and re-enables the Parquet dictionary cost-fallback that v18.8 disabled for compressed columns.

Changes

Arrow-go v18.8

1. Bump github.com/apache/arrow-go/v18 to v18.8.0.
2. Accept both the canonical arrow.parquet.variant and legacy parquet.variant extension names (arrow_utils.go, internal/parquet_files.go).
3. Force WithDictionaryCostFallbackFor on every leaf so zstd columns drop a dictionary that saves nothing (parquet-mr shouldFallBack parity; v18.8 disabled this for compressed columns) (internal/parquet_files.go).

Variant extract

4. Residual extract navigates the path via compute.VariantGet (feat(extensions): add VariantGet for path extraction from variant arrays arrow-go#1206), with the per-row CastVariantLiteral walk as fallback; adds BoundExtract.VariantPath() (variant_residual.go, variant_extract.go).
5. tryShreddedTypedColumn: when a field is shredded to exactly the target type (no field-level residual), return its typed_value column directly - Iceberg's cast is then an identity (mirrors Java VariantExpressionUtil.castTo). Null rows and absent-object ancestors fold into the validity mask (zero-copy when none, one bitmap-AND per level otherwise); promotions, field-level residual, and unshredded data fall back (variant_residual.go).

Performance

BenchmarkVariantExtract measures extract-column materialization on a 131,072-row batch (the default read batch size), Apple M4 Max, go1.27. "Production" is the current code (the fast path when the field is shredded to the queried type, otherwise the compute.VariantGet fallback); "per-row" is the previous walk.

int64 field, 131,072 rows	production	per-row (previous)
clean, fully shredded (fast path)	47 ns, 1 alloc	66.9 ms, 2,358,045 allocs
1% null rows (fast path)	5.0 us, 10 allocs	66.4 ms, 2,334,461 allocs
nested $.a.b (fast path)	78 ns, 2 allocs	102 ms, 2,751,261 allocs
1% off-type rows, field-level residual (fallback)	32.3 ms	68.7 ms
unshredded (routed to per-row)	33.6 ms	33.5 ms

String targets show the same pattern.

• When the field is shredded to the queried type - the common case after feat(table): type uniformity for variant shredding inference #1846 type uniformity, including nullable columns and nested paths - extraction returns the shredded typed_value column directly. This is O(1) in the row count and effectively allocation-free: 47 ns and 1 allocation on a clean batch, versus ~67 ms and 2.36M allocations for the per-row walk. Nullable columns add a single bitmap merge (~5 us); nested paths ~78 ns.
• When the field is not cleanly shredded, extraction falls back. Partly-shredded data (field-level residual) uses the columnar compute.VariantGet, ~2x faster than the per-row walk (32.3 ms vs 68.7 ms). Unshredded columns route directly to the per-row walk, matching it (33.6 ms vs 33.5 ms) rather than paying VariantGet.
• These figures cover the extract-column materialization step only, not a full scan, which additionally includes predicate evaluation, IO, and decompression.

Testing

• TestExtractFastPathParity: fast-path output byte-identical to the per-row reference across exact/nested/absent/promotion/residual/null shapes; asserts it is wired in and zero-copy (mutation-checked) under a checked allocator.
• TestShreddedVariantExtractResidualNoLeak: extract through the real residual filter under a checked allocator (leak check).
• TestGetWritePropertiesEnablesDictCostFallback: cost fallback enabled per leaf.
• TestVariantExtractScanEndToEnd / TestVariantExtractResidualAndHeterogeneous: fast path and fallback via the public Scan().WithRowFilter(...).
• BenchmarkVariantExtract: the numbers above (int64/string x clean/nulls/residual/unshredded/nested).

AI Disclosure

• Model: Claude Opus 4.8
• Platform/Tool: Claude Code
• Human Oversight: fully reviewed
• Prompt Summary: Adopt arrow-go 18.8 and add a zero-copy fast path for fully-shredded variant extract