Skip to content

[SPARK-60680][CONNECT] Support integer positional DataFrame column indexing - #125

Draft
dbtsai wants to merge 2 commits into
apache:masterfrom
dbtsai:codex/spark-60680-positional-index
Draft

dbtsai wants to merge 2 commits into
apache:masterfrom
dbtsai:codex/spark-60680-positional-index

Conversation

@dbtsai

@dbtsai dbtsai commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Support integer positional column access such as df[0] and df[-1]. The binding currently accepts only strings, so these calls raise TypeError. Resolve integer positions through the DataFrame's column names and use Python list indexing to preserve negative positions, bool/int subclasses, and IndexError for out-of-range or arbitrarily large integers. Keep named lookup unchanged and release the GIL during schema lookup.

This is independent of schema caching PR #120. Positional access performs a schema RPC on current master. List projection and Column-filter indexing are outside this change.

Why are the changes needed?

SPARK-60680: restore the Python Connect client's positional column indexing behavior.

How was this patch tested?

  • All 19 positional regression cases live in the CI-selected test_dropin_offline.py, with unique helper and fixture prefixes. Exact selected suite: candidate 293 passed; baseline 275 passed and exactly 18 expected positional failures. This test-only followup leaves runtime and workflow bytes unchanged.
  • 19 focused tests using a separate-process mock gRPC server, passing against both the candidate and official Python client. Covers positive/negative positions, bounds and huge integers, booleans, subclasses, empty schemas, invalid types, schema errors, special column names, and named-access RPC guards.
  • 274 existing offline tests pass on both baseline and candidate with --rootdir=python --import-mode=importlib. Supplemental wire comparison confirms ten identical selected column names; candidate performs six schema RPCs versus reference Python's cached one.
  • 392 native client library tests and 95 core transport library tests passed. Full native tests compiled with default and wasm-udf features. Release wheel built with maturin build --release --locked; formatting passed.
  • Unchanged frozen client benchmark: five fresh subprocess samples per case/client across four positional cases and five guards. Candidate and official Python: 45/45 pass each. Master baseline: 25/45 pass, with exactly the 20 expected positional TypeError failures. Named-column access and all unrelated guards pass for all clients. Each candidate positional sample performs one schema RPC; named access performs none. Compatibility fix, no speedup claim.

Validation uses a scripted localhost gRPC service, with no Spark server or JVM. A workspace-wide compile attempt was blocked by the unrelated WASM example's missing wasm32-unknown-unknown target; native tests compile independently. Initial sandbox TLS failure passed with network permissions. Failed setup and baseline evidence are retained in the campaign artifacts.

This pull request and its description were written by Isaac.

dbtsai and others added 2 commits October 28, 2026 22:18
…dexing

Co-authored-by: Isaac <no-reply@databricks.com>
…fline suite

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant