Arrow, Spark: Support vectorized Parquet reads for struct type

[nssalian]

Rationale

Vectorized Parquet reads currently throw UnsupportedOperationException for struct columns, so any scan that projects a struct falls back to the row reader. This adds vectorized struct reads for Spark 3.5, 4.0, 4.1, and 4.2.

A struct is assembled from its child column vectors. Whether a struct is null is decided per row from the definition level of a real leaf under it, matching the row reader - including the schema-evolution case where a struct is projected with only a field that isn't in the file (present rows return the struct with that field null; rows where the struct was null return null). Nested structs share a single presence read, and a struct whose projected leaf is a partition constant reads present without re-reading a column. ORC and Avro are unchanged.

The standalone ArrowReader still rejects structs (SUPPORTED_TYPES); that path needs a fully materialized Arrow StructVector and is left as a follow-up. Flink has no arrow vectorized-Parquet path, so it is unaffected. Related to #2485.

Changes

• arrow: VectorizedReaderBuilder.struct builds a VectorizedStructReader; presence is populated from a definition-level hook in the column/page/def-level readers.
• parquet: ParquetSchemaUtil.selectPresenceColumn and presenceField are now public (used by the arrow reader to pick the same presence leaf the row reader retains). New public API in iceberg-parquet; non-breaking, no revapi entry needed.
• Spark 3.5/4.0/4.1/4.2: add StructColumnVector, dispatch it in ColumnVectorBuilder, and allow structs in SparkBatch. The unshredded-variant lower-bounds check now also covers variants nested inside a struct.

Chose to do all the Spark versions together, since the struct reader lives in the shared arrow/parquet modules with no per-version switch, so enabling only one version leaves the others' existing nested-struct tests failing on the vectorized path.

Tests

Per Spark version: nested / nullable / multi-level / dictionary-encoded / all-null structs, projection of an added field (per-row null), and structs under position deletes. Arrow unit tests cover child-holder assembly, presence-leaf resolution, and nested presence-reader reuse.

---

AI Disclosure

• Model: Claude Opus 4.8
• Platform/Tool: Claude Code
• Human Oversight: fully reviewed
• Prompt Summary: implement vectorized Parquet struct reads across Spark 3.5/4.0/4.1/4.2 (arrow reader + presence plumbing + per-version column vector and batch-read eligibility), with per-row null presence matching the row reader.

[pvary]

@nssalian: Is this revisiting #15647?
You might want to take a look and find the best solution.

[nssalian]

Thanks for linking me to that PR @pvary. I missed it but I did take a look at the implementation. The PR stops at the arrow layer (no Spark StructColumnVector/SparkBatch changes) and uses an all-children-null heuristic for null structs, so it's incomplete for the Spark vectorized path from my vantage point. I'm fixing the test failures in this PR hence it is still in draft as I work through. Let me push up the changes to get it in a clean state and we can review it once that is done.