Repository navigation
Conversation
…dexing Co-authored-by: Isaac <no-reply@databricks.com>
…fline suite Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Support integer positional column access such as
df[0]anddf[-1]. The binding currently accepts only strings, so these calls raiseTypeError. Resolve integer positions through the DataFrame's column names and use Python list indexing to preserve negative positions, bool/int subclasses, andIndexErrorfor out-of-range or arbitrarily large integers. Keep named lookup unchanged and release the GIL during schema lookup.This is independent of schema caching PR #120. Positional access performs a schema RPC on current master. List projection and Column-filter indexing are outside this change.
Why are the changes needed?
SPARK-60680: restore the Python Connect client's positional column indexing behavior.
How was this patch tested?
test_dropin_offline.py, with unique helper and fixture prefixes. Exact selected suite: candidate 293 passed; baseline 275 passed and exactly 18 expected positional failures. This test-only followup leaves runtime and workflow bytes unchanged.--rootdir=python --import-mode=importlib. Supplemental wire comparison confirms ten identical selected column names; candidate performs six schema RPCs versus reference Python's cached one.wasm-udffeatures. Release wheel built withmaturin build --release --locked; formatting passed.TypeErrorfailures. Named-column access and all unrelated guards pass for all clients. Each candidate positional sample performs one schema RPC; named access performs none. Compatibility fix, no speedup claim.Validation uses a scripted localhost gRPC service, with no Spark server or JVM. A workspace-wide compile attempt was blocked by the unrelated WASM example's missing
wasm32-unknown-unknowntarget; native tests compile independently. Initial sandbox TLS failure passed with network permissions. Failed setup and baseline evidence are retained in the campaign artifacts.This pull request and its description were written by Isaac.