Repository navigation

[SPARK-60676][CONNECT] Use Arrow columns for signed-int64 pandas input - #121

Draft

dbtsai wants to merge 27 commits into

apache:masterfrom

dbtsai:codex/spark-60676-columnar-pandas

Draft

[SPARK-60676][CONNECT] Use Arrow columns for signed-int64 pandas input#121
dbtsai wants to merge 27 commits into
apache:masterfrom
dbtsai:codex/spark-60676-columnar-pandas

Commits on Oct 29, 2026
- Export Arrow tables through the C Stream interface
  dbtsai
  committed
  d7cebc3
- Use arrow-rs reader conversion for the C Stream handoff
  dbtsai
  committed
  afb69bc
- Simplify Arrow handoff tests and add live-server smoke cases
  
  dbtsai
  and
  Isaac
  committed
  d89c01b
- Skip Arrow handoff tests when protoc is unavailable
  
  dbtsai
  and
  Isaac
  committed
  fe79d54
- [SPARK-60675][CONNECT] Support PyArrow Table input through the C Stream interface
  
  dbtsai
  and
  Isaac
  committed
  c4bbfd3
- [SPARK-60675][CONNECT] Upload large Arrow inputs as cached relations
  
  dbtsai
  and
  Isaac
  committed
  ce89b8d
- [SPARK-60676][CONNECT] Use Arrow columns for signed int64 pandas inputs
  dbtsai
  committed
  4a5807e
- [SPARK-60676][CONNECT] Validate integer upload scales against mock RPC
  dbtsai
  committed
  38568fe
- [SPARK-60676][CONNECT] Cover leading null inference and non-nullable schemas
  dbtsai
  committed
  7009bb7
- [SPARK-60676][CONNECT] Exercise named schemas through mock rename relations
  dbtsai
  committed
  fee0800
- [SPARK-60676][CONNECT] Avoid Arrow configuration overhead for small NumPy frames
  dbtsai
  committed
  50b00d1
- [SPARK-60676][CONNECT] Retain IPC row fallback for NUL field names
  dbtsai
  committed
  a8774f0
- [SPARK-60676][CONNECT] Include pandas regressions in the existing Arrow CI suite
  dbtsai
  committed
  e7e3094
- [SPARK-60676][CONNECT] Preserve integer pandas schema and fallback semantics
  
  dbtsai
  and
  Isaac
  committed
  8a11a4d
Commits on Oct 30, 2026
- [SPARK-60676][TESTS] Preserve subclass column labels on pandas 3
  
  dbtsai
  and
  Isaac
  committed
  d26e45b
- [SPARK-60676][CONNECT] Merge upstream Arrow handoff to restore CI
  
  dbtsai
  and
  Isaac
  committed
  aff23b8
- [SPARK-60676][CONNECT] Preserve customized pandas columns hooks
  
  dbtsai
  and
  Isaac
  committed
  964e16e
- [SPARK-60676][CONNECT] Preserve rebound pandas index storage ordering
  
  dbtsai
  and
  Isaac
  committed
  c4284a3
- [SPARK-60675][CONNECT] Support PyArrow Table input through the C Stream interface
  
  dbtsai
  and
  Isaac
  committed
  c7e5941
- [SPARK-60675][CONNECT] Upload large Arrow inputs as cached relations
  
  dbtsai
  and
  Isaac
  committed
  97bd97b
- [SPARK-60675][CONNECT] Reduce cached Arrow upload overhead
  
  dbtsai
  and
  Isaac
  committed
  3f5fde1
- [SPARK-60675][CONNECT] Batch Arrow input settings with safe fallbacks
  
  dbtsai
  and
  Isaac
  committed
  efd5d75
- [SPARK-60675][CONNECT] Validate Arrow schema widths and test stock uploads
  
  dbtsai
  and
  Isaac
  committed
  c1b4cb5
- [SPARK-60675][CONNECT] Harden Arrow input settings and schema validation
  
  dbtsai
  and
  Isaac
  committed
  36189cc
Commits on Oct 31, 2026
- Merge master schema caching into SPARK-60676 continuation
  
  dbtsai
  and
  Isaac
  committed
  ce717ab
- Merge current Arrow input dependency into SPARK-60676
  
  dbtsai
  and
  Isaac
  committed
  a235bf6
- Preserve integer pandas fallback with missing Arrow aliases
  
  dbtsai
  and
  Isaac
  committed
  aa5f5b5

Commits

Commits on Oct 29, 2026

Commits on Oct 30, 2026

Commits on Oct 31, 2026