Repository navigation
Conversation
Provide an alternative to the per-batch C Data handoff in PR apache#103 while retaining PyO3 0.28.3. Add ownership, buffer-sharing, compatibility, and public client handoff tests. Generated-by: OpenAI Codex
Apply LC's suggestion to export one boxed RecordBatchReader through arrow-pyarrow and materialize it with read_all. Remove the custom stream callbacks, unused converter crate, and dedicated converter workflow. Keep schema consistency checks, standalone schema preflight, and the NUL schema IPC fallback. Retain handoff tests in the coverage workflow and keep PyO3 at 0.28.3. Generated-by: OpenAI Codex (application version unavailable).
Keep public behavior coverage and compile mock protocol descriptors from source. Add native toArrow smoke cases, remove the unproven schema preflight, and avoid the arrow module name collision. Leave CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Document the Arrow 58.3 NUL export panic at its upstream source and wrap the architecture paragraph. Keep CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
…am interface Normalize Arrow schemas and timestamps with the existing PySpark helpers, then serialize imported Rust batches directly for local relations without Python row materialization. Co-authored-by: Isaac <no-reply@databricks.com>
Honor server local-relation settings, split Arrow IPC into independently readable chunks, and upload session-scoped cache artifacts with SHA-256 deduplication and CRC validation. Keep large input payloads out of query plans and test uploads above the gRPC message limit. Co-authored-by: Isaac <no-reply@databricks.com>
Read the three mutable Arrow normalization settings in one Get request per nonempty construction while retaining empty-input and transport behavior. Co-authored-by: Isaac <no-reply@databricks.com>
…tion Avoid loading general converters and rebuilding canonical nullable int64 tables while preserving the general path for customized types, metadata, casts, and loaded converter hooks. Co-authored-by: Isaac <no-reply@databricks.com>
Reconcile the merged Arrow export master without rewriting the canonical input branch. Retain every canonical input guard and config batching regression, plus the final upstream output guards, stable-buffer and CI protoc checks. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Creating a DataFrame from an already canonical nullable int64 Arrow table still imports the general conversion/error hierarchy and runs redundant casts and renaming. Return that table unchanged when its exact native schema, names, nullability and metadata match. Loaded converters, modified native descriptors/type bindings, and all other inputs retain the general path. Config reads remain fresh; native IPC serialization and upload behavior are unchanged.
Depends on #134 (current head
5737f1d511f989ad52706d10df7cb62b3eadb80b) and #118. #108 has merged upstream. The original narrow implementation is the three-file Python diff against the original #134 head. A normal merge of upstreamd9defae6f691671d8bad4927f58a7283cd0a0b50produced current headef2e76ceab55bd090bfd654fb1d7f4c07f437617. This reconciliation retains the input implementation byte for byte and brings upstream workflow/docs, an exporter comment, and handoff tests into the branch.Historical performance evidence at original candidate
2e1b60445137cdad9234b5d7ee3acc35a5537fb4and baseline911e23cbe21409fbb530894939b4f0134e69d6af(plus the separately frozen integrated v4 comparison): five fresh processes per client/case, identical native binaries and dependencies within each comparison. These timings were not rerun for the source-equivalent upstream reconciliation. Cold construction milliseconds, median [min–max]:All 255 samples pass. Every raw baseline/candidate pair has identical RPC counters, including request and artifact bytes. The unchanged benchmark harness is pinned at
1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4; an explicit external workload extension supplies metadata-freepa.tableinputs from threenp.arange(n, dtype=np.int64) + column_indexarrays and nullable LongType fields. Inputs are prepared outside timing, cold cases warm only the channel, and the cache threshold is 1 MiB. Seven original guard workloads are unchanged. Warm and 1M-row ranges overlap; no repeatable benefit is claimed there.Fresh validation at current head: 369 Python tests pass / 11 live-server skips on pandas 2.3.3, and 369 pass / 11 skips on pandas 3.1.0; 394 Rust client and 95 core library tests pass; all integration targets compile with and without
wasm-udf; rustfmt and diff checks pass. All 258 Python modules match the fresh source, wheel and both isolated installed environments. The native module is byte-identical to the prior candidate in both environments. The original 38 focused integrated regressions remain historical evidence. Existing CI-selected suites cover exact native IPC/cache hashes and every cell/schema at 10k/30k/60k/1M, nullable extrema, slices/chunks, empty input, metadata/nullability/casts, custom types/hooks, fresh imports, config errors, size limits and upload failures.Fresh wheel SHA256:
6dcec72d04cbe7d743729e514f93b4193b47519c6bc8722c6438449cc4fdee80. No JVM/live Spark execution was used; these are local mock-RPC construction measurements. Clippy is unavailable in the configured offline toolchain. The optimization deliberately falls back once the general converter is loaded.This pull request and its description were written by Isaac.