Repository navigation
Conversation
Provide an alternative to the per-batch C Data handoff in PR apache#103 while retaining PyO3 0.28.3. Add ownership, buffer-sharing, compatibility, and public client handoff tests. Generated-by: OpenAI Codex
Apply LC's suggestion to export one boxed RecordBatchReader through arrow-pyarrow and materialize it with read_all. Remove the custom stream callbacks, unused converter crate, and dedicated converter workflow. Keep schema consistency checks, standalone schema preflight, and the NUL schema IPC fallback. Retain handoff tests in the coverage workflow and keep PyO3 at 0.28.3. Generated-by: OpenAI Codex (application version unavailable).
Keep public behavior coverage and compile mock protocol descriptors from source. Add native toArrow smoke cases, remove the unproven schema preflight, and avoid the arrow module name collision. Leave CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Document the Arrow 58.3 NUL export panic at its upstream source and wrap the architecture paragraph. Keep CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
…am interface Normalize Arrow schemas and timestamps with the existing PySpark helpers, then serialize imported Rust batches directly for local relations without Python row materialization. Co-authored-by: Isaac <no-reply@databricks.com>
Honor server local-relation settings, split Arrow IPC into independently readable chunks, and upload session-scoped cache artifacts with SHA-256 deduplication and CRC validation. Keep large input payloads out of query plans and test uploads above the gRPC message limit. Co-authored-by: Isaac <no-reply@databricks.com>
Read the three mutable Arrow normalization settings in one Get request per nonempty construction while retaining empty-input and transport behavior. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Oct 29, 2026
Reconcile the merged Arrow export master without rewriting the batching branch. Retain all input config batching regressions and stronger buffer lifetime checks, and preserve the upstream output schema/NUL guards and CI protoc checks. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nonempty Arrow input currently reads
timestampType, session timezone, anduseLargeVarTypesin three sequential Config RPCs. Fetch them together through the existing coreget_configstransport, retaining fresh reads on every construction and the separate upload-options request. Total Config RPCs fall from four to two; empty input still makes zero.Depends on #118 at
ce89b8dd496bbb1529f1b5531b1eaaa6d739bef0. #108 Arrow export is now merged to upstream masterd9defae6f691671d8bad4927f58a7283cd0a0b50; this branch includes that master through a normal merge. The batching implementation remains unchanged. The reconciliation preserves both the input batching tests and final upstream output guards, CI protoc checks and live smoke step. #118 remains a dependency.Fresh validation at
5737f1d511f989ad52706d10df7cb62b3eadb80b: 394 client + 95 core unit tests pass, all native/WASM integration targets compile, and the complete offline/Arrow selection passes 331 tests / 11 live-server skips on each of pandas 2.3.3 and 3.1.0. One fresh release wheel is installed in separate isolated environments; all 258 Python modules and the native library are verified against source/wheel bytes. The rebuilt native binary is byte-identical to the prior validated wheel. All checks hold the campaign CPU lock and inner cutoff guard. Only exporter comments, tests, docs and the upstream workflow step differ from the predecessor. The benchmark evidence below retains its original revision scope; this reconciliation adds no timing claim.Five fresh samples per client/case against a frozen localhost mock harness:
The small warmed cases have disjoint observed timing ranges. The 1M-row cases show overlapping ranges. Cold 30k cache-threshold construction has a lower median (4.397→3.509ms, no artifact RPCs), but overlapping ranges; no reliable cold-case speedup is claimed. All 420 samples passed. Unaffected guards retain their RPC counts and overlapping timing ranges. PR 121 is a measurement-only dependency on both pandas sides and is excluded from this change.
Original validation: 394 Rust library tests pass; integration tests compile with/without
wasm-udf; rustfmt and diff checks pass. Python selection: 324 pass, 11 live-server skips. Eighteen new cases in the existing CI-selected Arrow suite cover request keys/order, reversed/missing responses, errors/retry, config mutation, empty input, and independent timestamp/timezone/large-offset IPC/schema/cell expectations. The baseline passes the corresponding semantic checks and fails the new batching assertion as expected. Both combined pandas environments pass 72 tests, including independent nullable int64 IPC cell/schema checks.No JVM/live-server parity run. Clippy is unavailable in the configured offline toolchain. Results establish local mock-RPC construction performance, not end-to-end Spark execution speed.
This pull request and its description were written by Isaac.