Repository navigation
Conversation
Provide an alternative to the per-batch C Data handoff in PR apache#103 while retaining PyO3 0.28.3. Add ownership, buffer-sharing, compatibility, and public client handoff tests. Generated-by: OpenAI Codex
Apply LC's suggestion to export one boxed RecordBatchReader through arrow-pyarrow and materialize it with read_all. Remove the custom stream callbacks, unused converter crate, and dedicated converter workflow. Keep schema consistency checks, standalone schema preflight, and the NUL schema IPC fallback. Retain handoff tests in the coverage workflow and keep PyO3 at 0.28.3. Generated-by: OpenAI Codex (application version unavailable).
Keep public behavior coverage and compile mock protocol descriptors from source. Add native toArrow smoke cases, remove the unproven schema preflight, and avoid the arrow module name collision. Leave CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Document the Arrow 58.3 NUL export panic at its upstream source and wrap the architecture paragraph. Keep CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
…am interface Normalize Arrow schemas and timestamps with the existing PySpark helpers, then serialize imported Rust batches directly for local relations without Python row materialization. Co-authored-by: Isaac <no-reply@databricks.com>
Honor server local-relation settings, split Arrow IPC into independently readable chunks, and upload session-scoped cache artifacts with SHA-256 deduplication and CRC validation. Keep large input payloads out of query plans and test uploads above the gRPC message limit. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Oct 29, 2026
…mantics Use explicit LongType schema names when constructing Arrow columns. Limit the columnar path to concrete pandas/NumPy arrays and native schema descriptors, including authentic converter aliases retained by already imported modules. Keep nullable values in non-nullable fields on the native row path so local validation still precedes configuration RPCs. Support warm Arrow importers that resolve converters locally. Add mock-RPC regressions for all emitted integer cells, schema metadata and nullability, custom conversion hooks, and repeated warm construction. Co-authored-by: Isaac <no-reply@databricks.com>
Build the subclass-label fallback fixture with an explicit object Index and assert its concrete label type. Pandas 3 otherwise normalizes constructor labels to builtin strings before the unchanged integer helper sees them. The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed regression and full CI-selected Arrow/drop-in suites pass on both versions. Runtime Python and native sources remain unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Merge current Apache master d9defae while retaining all pandas integer input guards and the pandas 3 fixture repair. Keep upstream Arrow output regressions, CI protoc requirement and live smoke step; preserve fork input bridge/docs and stable-buffer assertions. Co-authored-by: Isaac <no-reply@databricks.com>
Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Oct 30, 2026
…am interface Normalize Arrow schemas and timestamps with the existing PySpark helpers, then serialize imported Rust batches directly for local relations without Python row materialization. Co-authored-by: Isaac <no-reply@databricks.com>
Honor server local-relation settings, split Arrow IPC into independently readable chunks, and upload session-scoped cache artifacts with SHA-256 deduplication and CRC validation. Keep large input payloads out of query plans and test uploads above the gRPC message limit. Co-authored-by: Isaac <no-reply@databricks.com>
Use accelerated SHA-256 and shared payload storage, and send cache artifacts in 1 MiB frames while keeping sequential uploads. Document real remote Spark benchmarking as required before merge. Co-authored-by: Isaac <no-reply@databricks.com>
Read normalization and upload settings in one optional configuration RPC, preserve server defaults, and defer chunk validation until caching. Avoid importing PyArrow for other input types and share artifact context and response validation. Co-authored-by: Isaac <no-reply@databricks.com>
…loads Reject incompatible explicit schemas before Arrow casts and hide the internal settings handoff from Rust documentation. Exercise cached Arrow input with stock local-relation settings in the live CI suite. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Preserve both Arrow input and schema-cache regression suites. Co-authored-by: Isaac <no-reply@databricks.com>
Preserve batched settings, cached uploads and master schema caching. Authenticate the new AtomicType normalization aliases before the integer fast path, with isolated fallback regressions. Co-authored-by: Isaac <no-reply@databricks.com>
Avoid loading optional Arrow label modules for fallback inputs, and decline integer Arrow conversion when the loaded importer AtomicType alias is absent. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Construct supported integer pandas DataFrames through the Arrow Table importer, using one Arrow array per Series instead of materializing Python rows. NumPy
int64and pandas nullableInt64support inferred schemas, complete string-name schemas, and explicit all-LongTypeschemas. Explicit renaming preserves integer precision, null masks, metadata and nullability.Eligibility requires canonical pandas objects, label storage/conversion hooks, native schema descriptors and converter aliases. Customized inputs retain row conversion and its local error ordering. NumPy-only frames below 1,000 rows retain the row path; nullable
Int64uses Arrow independently of frame size. Other dtypes, schema casts/DDL/atomic schemas, duplicate or unsupported names retain existing conversion.verifySchemaremains ignored for pandas Arrow input, matching Spark Connect.Dependency reconciliation: this branch incorporates current #118 at
36189cc12a54523fe518ec518464eb19c29376a7and upstream master617fa1481b3e6fc7b825928130f280506328c9b1through normal merges. The net change against #118 is four files: the pandas helper, its Rust session entry point, Arrow-suite regressions and architecture documentation. #118 continues to provide Arrow import, schema validation, settings handling and cached uploads.The reconciliation also authenticates the public and loaded-importer
AtomicTypealiases now consulted by #118. Rebinding or deleting a consulted alias causes the integer optimization to decline before importer configuration, preserving native row behavior. Seven isolated public regressions cover aliases changed before/after importer loading, invalid/rebound replacements and importer-alias deletion. Label eligibility checks also preserve optional PyArrow import avoidance by using only the already-loaded module before consulting pandas Arrow internals.Why are the changes needed?
The row path constructs Python objects for every integer cell and can lose integer typing when nullable inference begins with a null. This path retains integer buffers and null masks. Conservative eligibility checks preserve customized conversion behavior, including renamed schemas and local nullability errors.
Does this PR introduce any user-facing change?
Yes. Compatible integer pandas inputs use columnar construction. Unsupported or customized inputs retain existing conversion. This remains a signed 64-bit integer optimization.
How was this patch tested?
Fresh evidence below applies to candidate
aa5f5b59d0876d78fe1606b1de2ea5feff3984bbagainst current #11836189cc12a54523fe518ec518464eb19c29376a7:SPARK_REMOTE. All seven new alias cases pass; replacing only the helper with the oldc4284a3helper in isolated copies makes all seven fail on each version. The unchanged intermediatea235bf6wheel independently fails the deletion regression withAtomicTypeNameError. Final environments remain untouched by these controls.apache-spark-connectnative test targets and itswasm-udffeature test targets compile with--no-run; this does not claim integration execution or a WASM-target build. Formatting and diff whitespace pass. Advisory Clippy exits 101 on 12approx_constanterrors in four files byte-identical to [SPARK-60675][CONNECT] Support PyArrow Table input through the C Stream interface #118; it is not green.1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4, unchanged operations/timers/validators: 260 / 260 samples, 0 failures. Two Rust roles, both pandas versions, ten existing warm cases and three first-construction cases, five fresh processes per role/case. An adapter-only bridge translates [SPARK-60675][CONNECT] Support PyArrow Table input through the C Stream interface #118's batchedGetOptioninto the frozen fixture's internalGet, preserving its registered settings and original wire/RPC counters. Complete received-input validation runs after the original measurement and validator. Full 20-counter RPC maps, CPU, memory distributions and before/after bindings are retained.The first-construction 1,000×3 costs are pandas233 / first-construction / scale-pandas-1000x3: 1.991→3.983 ms (+1.992 ms / +100.1%; ranges 1.704–3.111 versus 3.276–17.338); pandas310 / first-construction / scale-pandas-1000x3: 1.639→3.053 ms (+1.414 ms / +86.3%; ranges 1.578–1.658 versus 1.967–3.203).
The largest adverse elapsed median is pandas233 / first-construction / scale-pandas-1000x3: 1.991→3.983 ms (+1.992 ms / +100.1%; ranges 1.704–3.111 versus 3.276–17.338).
The largest adverse CPU median is pandas233 / first-construction / scale-pandas-1000x3: 1.817→3.454 ms (+1.637 ms / +90.1%; ranges 1.699–2.170 versus 2.853–5.026).
The largest adverse held-RSS median is pandas310 / warm / type-pandas-string-10000x3: 115.438→117.219 MiB (+1.781 MiB / +1.5%; ranges 115.219–118.828 versus 116.062–118.703).
The largest adverse sampled-peak median is pandas310 / warm / type-pandas-string-10000x3: 115.438→117.219 MiB (+1.781 MiB / +1.5%; ranges 115.219–118.828 versus 116.062–118.703).
The largest adverse process-lifetime-peak median is pandas310 / warm / type-pandas-string-10000x3: 115.438→117.219 MiB (+1.781 MiB / +1.5%; ranges 115.219–118.828 versus 116.062–118.703).
The largest adverse post-GC median (including additive-validator allocation) is pandas310 / warm / type-pandas-string-10000x3: 120.578→123.438 MiB (+2.859 MiB / +2.4%; ranges 120.406–125.469 versus 121.094–123.766).
Complete input proof covers 260 unique receipts / 82,526,000 cells. 60 of 130 same-index 20-counter RPC pairs differ: eligible integer cases add one Config RPC and 380 request bytes; all other counters are unchanged. Arrow controls match. All measured inputs use inline delivery with ordinary values and empty metadata; separate candidate strict IPC tests cover nulls, nonempty metadata, renames and cached delivery.
These five-process distributions do not establish statistical significance, causation or memory neutrality. Full elapsed/CPU/all-RSS distributions, largest sampling gaps, per-process measurements, RSS observation counts and every RPC vector/difference are retained in the campaign artifacts. Timestamped RSS observation series are not retained.
Earlier unsuccessful attempts remain preserved and excluded: test namespace/protoc launcher failures, the intermediate optional-PyArrow regression fixed here, six GetOption fixture failures in the first preflight, and an auditor launch with an older system Python. The accepted preflight and final run use fresh output directories; no failed-attempt samples enter the final distributions.
First construction sets constructor warmups to zero after untimed library/session/input preparation and
spark.version; it is not a cold interpreter measurement. RSS is the externally sampled process peak with a 2 ms target interval, including input/warmup allocations. Process-lifetime peak is separate; post-GC RSS includes the additive validator. Measurements use scripted local gRPC fixtures, without JVM/live Spark execution. The official interpreter is an untimed fixture server, with no official-client timing comparison.The Config bridge exercises registered settings only; missing/unset values, null responses and GetOption default semantics are not exercised. Fixture-server CPU, including translation, is outside measured client CPU; construction wall time and original RPC/request-byte costs include the fixture roundtrip. Complete Arrow equality is certified by received-input validator receipts; the separate stdlib audit independently rehashes saved payloads and checks routing/provenance, without re-decoding Arrow values.
Historical evidence retains its original revision/comparator scope: the original 197× result predates these guards and is not a fresh-head claim. The
c4284a3versusaff23b84hook-guard comparison reported first-wide-construction costs of +30.3% on pandas 2.3 and +13.0% on 3.1, and a +10.78 MiB sampled-peak increase in its million-row case. Its official pandas 3.1 preflight timeout remains unresolved and excluded. Earlier advisory Clippy failures are not claimed green. Fresh measurements above replace those historical timings for assessment of the reconciled head; overlapping ranges alone do not establish zero overhead or memory neutrality.Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
This pull request and its description were written by Isaac.