Repository navigation
Conversation
Provide an alternative to the per-batch C Data handoff in PR apache#103 while retaining PyO3 0.28.3. Add ownership, buffer-sharing, compatibility, and public client handoff tests. Generated-by: OpenAI Codex
Apply LC's suggestion to export one boxed RecordBatchReader through arrow-pyarrow and materialize it with read_all. Remove the custom stream callbacks, unused converter crate, and dedicated converter workflow. Keep schema consistency checks, standalone schema preflight, and the NUL schema IPC fallback. Retain handoff tests in the coverage workflow and keep PyO3 at 0.28.3. Generated-by: OpenAI Codex (application version unavailable).
Keep public behavior coverage and compile mock protocol descriptors from source. Add native toArrow smoke cases, remove the unproven schema preflight, and avoid the arrow module name collision. Leave CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Document the Arrow 58.3 NUL export panic at its upstream source and wrap the architecture paragraph. Keep CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
…am interface Normalize Arrow schemas and timestamps with the existing PySpark helpers, then serialize imported Rust batches directly for local relations without Python row materialization. Co-authored-by: Isaac <no-reply@databricks.com>
Honor server local-relation settings, split Arrow IPC into independently readable chunks, and upload session-scoped cache artifacts with SHA-256 deduplication and CRC validation. Keep large input payloads out of query plans and test uploads above the gRPC message limit. Co-authored-by: Isaac <no-reply@databricks.com>
Read the three mutable Arrow normalization settings in one Get request per nonempty construction while retaining empty-input and transport behavior. Co-authored-by: Isaac <no-reply@databricks.com>
…rrow Conservatively accept ordinary object-column frames with exact built-in dates and explicit DateType schemas. Preserve native row conversion for customized schema descriptors and unsupported values. Co-authored-by: Isaac <no-reply@databricks.com>
Use the measured 10000-row threshold to avoid config RPC overhead on smaller inputs. Cover the fast-path boundary and load fresh-process test fixtures by exact source path for CI portability. Co-authored-by: Isaac <no-reply@databricks.com>
Use the shared protocol fixture so fresh-process descriptor regressions skip consistently with the Arrow suite when protoc is unavailable. Co-authored-by: Isaac <no-reply@databricks.com>
…mantics Use explicit LongType schema names when constructing Arrow columns. Limit the columnar path to concrete pandas/NumPy arrays and native schema descriptors, including authentic converter aliases retained by already imported modules. Keep nullable values in non-nullable fields on the native row path so local validation still precedes configuration RPCs. Support warm Arrow importers that resolve converters locally. Add mock-RPC regressions for all emitted integer cells, schema metadata and nullability, custom conversion hooks, and repeated warm construction. Co-authored-by: Isaac <no-reply@databricks.com>
Require concrete pandas NumPy storage and unchanged converter aliases before selecting the date Arrow path. Preserve custom array values and native fallback error ordering. Co-authored-by: Isaac <no-reply@databricks.com>
Keep astype, notna and where class replacements and instance hooks on the native row path. Authenticate ordinary pandas normalization methods before columnar date conversion. Co-authored-by: Isaac <no-reply@databricks.com>
Build the subclass-label fallback fixture with an explicit object Index and assert its concrete label type. Pandas 3 otherwise normalizes constructor labels to builtin strings before the unchanged integer helper sees them. The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed regression and full CI-selected Arrow/drop-in suites pass on both versions. Runtime Python and native sources remain unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Merge current Apache master d9defae while retaining all pandas integer input guards and the pandas 3 fixture repair. Keep upstream Arrow output regressions, CI protoc requirement and live smoke step; preserve fork input bridge/docs and stable-buffer assertions. Co-authored-by: Isaac <no-reply@databricks.com>
Reconcile the merged Arrow export master without rewriting the pandas date branch. Retain every date input guard and custom schema/converter regression, plus the final upstream output guards, stable-buffer and CI protoc checks. Co-authored-by: Isaac <no-reply@databricks.com>
Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>
Merge the reviewed integer columns follow-up while retaining all date guards and regressions. Co-authored-by: Isaac <no-reply@databricks.com>
Decline customized column label storage before date column inspection so native row conversion retains hook values, exceptions, and normalization order. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The date fast path builds Arrow date32 arrays for ordinary NumPy-backed pandas object-column DataFrames with at least 10,000 rows and an explicit all-DateType StructType. It accepts exact built-in datetime.date or None cells, preserves explicit names/metadata/nullability, and uses the existing Arrow importer. Smaller inputs, subclasses, alternate missing objects, inferred/name-only/DDL/cast schemas, unsupported names and nulls in nonnullable fields retain row conversion. Authentic frame/Series/NumPy storage, schema/converter/importer identities and direct normalization hooks remain guarded.
The new change calls the already-reviewed
_has_canonical_columns(frame.columns)before date dtype/cell inspection or Config RPCs, returning to row conversion for customized label hooks. The native row path calls columns.tolist() before normalization; the date fast path previously could skip its exception or mutation. The predicate is exactly PR121c4284a3V5 and covers Index identity/tolist/_values/_data plus ordinary concrete NumPy or pandas string storage and their directly delegated hooks. The candidate adds exactly two runtime lines in prepare_date_table; there are no Rust/Cargo changes. Scope remains ordinary pandas methods and the concrete guarded hooks, without a promise of arbitrary global/internal monkeypatch parity.Depends on #121, #128 and #134, transitively #118; #108 is merged upstream. Current follow-up normally merges PR121
c4284a3into published PR138 380221a and adds the two-line date guard. The date-only patch adds no workflow or Rust/Cargo change. Source comparison: dbtsai/spark-connect-rust@380221a...20e8ec6Why are the changes needed?
Direct date conversion avoids pandas normalization and Python row materialization for supported inputs. Customized column-label hooks must still run in the original row-path order, preserving their exceptions and mutations before cell normalization or configuration RPCs. This follow-up is a correctness change; no new speedup, performance-neutrality or memory-neutrality claim is made.
Historical initial optimization evidence remains scoped to prior revisions. At original implementation
896ca1athe minimal exact dependency-base date 10k×3 median was 5.304→2.303ms (56.6%), and supplemental full-stack 4.406→2.603ms (40.9%), with195/195 minimal and135/135 full-stack unchanged-validator samples. The later2ea5ebfcorrectness follow-up separately passed 45 main + 90 date guard samples, with date median1.883→2.021ms (+7.3%). These are preserved historical measurements, not current candidate speedup claims. Initial exact dependency heads were PR121e7e3094, PR128b8940edand PR134911e23con minimal549242a. Published380221ais the later upstream reconciliation, retaining the date implementation. Current candidate additionally merges c4284a3 shared labels guard and adds the two-line date call.Does this PR introduce any user-facing change?
Supported date DataFrames use the existing Arrow importer, including its configuration/cached-upload behavior, with explicit schema values preserved. Customized labels fall back to native normalization and error ordering. Unsupported cases and smaller frames retain row conversion. Exact-cell rejection can scan before falling back; historical subclass/NaN studies measured0.612–1.355ms added median cost at 10k×3. Current guard/fallback costs are disclosed below.
Current candidate validation
_valueserrors (2),_dataordering (2), and ordinary canonical storage (6). Passing cases check full received date values/epoch days/null masks, date32/schema/renames/field metadata/nullability and fallback Config/upload behavior.9cbeabe54acd3e026d623c8d83487296a1f8a7b7695af43a41bb0b3818c81a01.Baseline/candidate wheel SHA256:
ebe11132799f6f8b756f2213d8f03c64bbaf5800cb4bcf5871254c13d35f25ca/c2c68cf358fb48ba8289f15e801aad94180520dc840e76ec35f1edb1a1650286. Current exact tested candidate is20e8ec67b0df58aee5c7e12de188a760415f4341.Matched measurement scope and adverse costs
The unchanged frozen benchmark harness head is
1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4. Main measurements use five fresh processes per client/case with the original rotating order and timers. pandas 2.3 has 18 warm cases × three roles × five samples and three first-construction cases × three roles × five samples (315); pandas 3.1 is a separately labeled three-warm/three-first Rust pair only (60). Total 375 passed / zero failed, final launcher exit0. No original main workload was repeated for proof recovery.Both matched date sides already include integer dependency
c4284a3: baseline2c43ef417a92c5d54e05c6ba84692c672345d1b0, candidate20e8ec67b0df58aee5c7e12de188a760415f4341. These measurements isolate the two-line date gate and exclude the total published380221a→new dependency cost. The normal baseline merge has parents published date380221a83a89cba558185ddaf663ec00f295a49eand published integerc4284a3e0b71c8d624a1253c52337419761b2131; candidate is its normal child commit. Both source trees are frozen and clean.pandas 2.3 reference and Rust roles share Python 3.12.14, pandas 2.3.3, NumPy 2.5.4, PyArrow 26.0.0, grpcio 1.84.1 and psutil 7.2.2; the separate Rust pair uses pandas 3.1.0. Source/wheel/executable/import origins/dependencies/directURL/native inventories are frozen. Original per-profile before/after bindings are equal. Enhanced captures around recovery also match exactly before/after for the official pandas 2.3 reference and all four Rust roles, ignoring only the top-level observation time (
cost-runtime-retention-completion.json). Source, wheel, installed module inventory, executable/import origins, dependencies, directURL and native bindings are unchanged.All 135 same-index Rust RPC map pairs match across all 20 counters (27 paired case/profile groups × five). The two return workloads keep the unchanged frozen endpoint/shape validators; complete received-input receipts are a distinct proof scope.
first-constructionmeans warmups 0 after imports, input/session preparation and an untimed spark.version; it is not a cold-interpreter measurement. postGC RSS includes additive validation allocations; elapsed/CPU/held/sampled/process peak and RPC snapshots keep their original boundaries before that proof. RSS sampling may miss short peaks. All CPU-heavy work used the shared campaign lock and approved deadline wrapper; no JVM/live Spark server was used.There is no performance or memory neutrality claim and no causal attribution. Main adverse medians include pandas 2.3 warm1k×3 1.298459→2.339958ms (+80.21%), wide first 6.454917→7.394000ms (+14.55%; CPU4.329→4.460ms), and pandas 3.1 date first 3.968875→4.354250ms (+9.71%; CPU2.952→3.005ms). Their five-process ranges overlap. Date warm is 2.523625→2.582208ms (+2.32%) on pandas 2.3 and 2.196584→2.140709ms (−2.54%) on pandas 3.1; pandas 2.3 first date is 8.581375→5.962583ms (−30.52%) with broad overlapping ranges. Descriptive ranges do not establish statistical equivalence.
Largest held and process-peak median increases are DDL +1.328125MiB and scale100k +1.312500MiB. Largest sampled-peak increase is scale1M 248.093750→255.375000MiB (+7.281250MiB / +2.93%), with overlapping244.781250–263.562500 /247.109375–263.625000MiB ranges. The complete 189-row CSV records all 27 pairs × seven metrics, medians/min/max/delta/%/overlap, including baseline and postGC RSS; it is not restricted to favorable observations.
Separately, historical PR121 wide first-construction dependency costs were 5.492→7.158ms (+30.3%) on pandas 2.3 (CPU4.389→4.657) and 4.589→5.186ms (+13.0%) on pandas 3.1 (CPU3.256→3.621). That prior dependency study's largest peak increase was 10.78MiB /4.44%, held +0.547MiB. These historical costs are not the new date gate's matched costs. There are no official pandas 3.1 timings in the date follow-up; the earlier official pandas 3.1 timeout remains unresolved/excluded.
Input proof and retained-envelope limitation
The original 330 in-memory input validation receipts record 72,149,500 cells. All 240 Rust request aliases remain byte-hash matched. The 90 original official final and 18 original official preflight request envelopes were overwritten by later same-session clean_cache ExecutePlan calls; they remain lost. Recorded receipt equality is not a claim that these original 108 envelopes were retained.
Root accepted reuse of 12 PR130 official case/profile groups, binding 72 older date receipts, because exact case dictionaries, the official reference, all 17 original harness files and all 8 semantic fields match: received payload SHA/size, Arrow schema, Spark schema, rename columns, full-column hashes, top-level null-mask hashes and cell count. Exactly six uncovered official singles then completed exit0 with immutable original
.request.pbfiles: warm DDL10k×3, wide10k×32, scale1M×3, scale100×3; first wide10k×32 and date 10k×3. These six check 3,700,300 fresh cells, each matching all 8 fields of its six older receipts. Fresh-six timings are excluded from all matched performance comparisons. Reused/fresh payload evidence neither reconstructs old envelopes nor restores original rotation/timing provenance.Three narrowly reviewed additive retention corrections were activated as manifest
frozen-cost-controls-retention-v3.json(b87ccf5dc006f41769048ca5ee7812e7528985575fc6723b066003e65cf375e1): immutable request copying after original snapshots in integer_validate.py; copying exact received date IPC bytes and checking full schema/values after the original boundary timer; unique persisted payload/receipt paths and hash/size checking in the boundary runner. All original 14 controls/manifest and intermediate controls/diffs are preserved. The exact original date worker prefix through three warmups, twenty measured calls and original epoch checks is unchanged. No reserialized table is labeled as received bytes.The earlier 36 boundary preflight samples have JSON/log/exit proof and 863,976 recorded cells, but no retained original IPC/full-schema proof; they were preserved and not rerun. The separately approved final 180 boundary samples completed exit0 with 180 original
.received.ipcfiles + 180 receipts, 4,319,880 recorded cells, 18 paired groups, equal original Config/upload maps and equal four-role before/after package bindings. The independent final decoder/cost/package audit passes (independent-date-final-review.json, SHA256bc3f5eac9052127e9504febccb59696a07018156569e46f63944fee7df36b3f4). It independently checks all 240 retained original Rust requests and 50,633,000 decoded cells, the exact mapping of all 108 older official receipts to the separate shared/fresh proof, all 180 final boundary payloads and 4,319,880 cells, all 189 main metric distributions and the current source/wheel/runtime bindings. Original official envelopes and the original 36 receipt-only boundary samples remain unavailable as stated above. Mock transport and received IPC establish client delivery proof, not actual server execution.Boundary and fallback measurements
These boundary results use three warmups and twenty measured constructions per process, five fresh processes per role/group. They are elapsed-only warm-loop measurements, with the original Config request lists/upload counts; they provide no CPU/RSS/cold-interpreter result and are not the main full20-counter RPC proof. All baseline/candidate ranges overlap. Largest positive medians: pandas 2.3 subclass_first +5.59%, pandas 3.1 nan_last +5.46%, pandas 3.1 subclass_first +4.07% and pandas 3.1 exact10k +2.66%. Independent decoder review confirms full-cell/date32/epoch/null/schema/metadata proof for all retained final 180 payloads.
Preserved failures and limits
Preserved attempts include: an official
python -cruntime probe that entered Spark doctest/check_dependencies and required SPARK_HOME (corrected by executing a real standalone probe file); initial sandbox localhost-bind denial before any benchmark client sample/timer (all 20 partial files retained, approved retry passed75); advisory Clippy 101; and official request-retention loss described above. The attempted launcher SIGSTOP aborted before signaling because an obsolete child was gone; by next inspection the original main launcher had completed all 375. No signals were sent. No result/failure was erased or replaced by fresh-proof timings.No official pandas 3.1 timing comparison is claimed, and five-process descriptive ranges do not establish statistical equivalence. Received mock IPC proof does not establish live server execution. Historical implementation evidence keeps its original source, dependency and environment scope.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
This pull request and its description were written by Isaac.