[SPARK-60686][CONNECT] Use Arrow columns for supported pandas timestamps by dbtsai · Pull Request #130 · apache/spark-connect-rust

dbtsai · 2026-10-29T06:14:02Z

What changes

createDataFrame materializes supported pandas timestamps as Python objects and converts each value separately. This PR hands explicit TimestampType columns to the existing Arrow importer when every column is datetime64[ns] or [us] and all values are microsecond-aligned. It preserves aware instants; naive timestamps require an actual UTC process (TZ=UTC) and UTC session. Explicit names, metadata, nullability and NaT are preserved. Nonnullable NaT fails locally before timezone configuration, including when Config is unavailable.

Eligibility requires authentic pandas DataFrame/Series/datetime storage, native schema access and converter aliases. The latest followup also checks canonical column-label storage before timestamp eligibility reads the labels. Customized tolist, label descriptors and string-storage hooks retain native row conversion, including hook effects, exceptions and call order. Ordinary NumPy, Python-string and Arrow-string labels remain eligible. The timestamp-specific product change is the shared two-line canonical-column guard; public regressions exercise createDataFrame delivery and errors rather than private eligibility assertions.

Inference, names/DDL schemas, TimestampNTZ, mixed/object/Arrow extension dtypes, other units, empty frames, submicrosecond values and unsupported naive timezones retain the row fallback. Timestamp helpers use local native schema imports and support importers that resolve converters locally.

Source and dependency scope

Final candidate: 7b53529d261b190130f36cf0a28070e5ea2de206. It descends from the previously published PR130 head 4b92da42bd765b812052a3ec116b2d006789b78d, normally merges the reviewed PR121 integer dependency c4284a3e0b71c8d624a1253c52337419761b2131, then adds the timestamp column-label guard and its regressions. The dependency supplies the shared canonical-label helper and integer hook/storage fixes; its Python regressions are consolidated into the existing Arrow CI test suite. This PR depends on #121 and transitively #118. #108 has merged upstream.

The matched guard baseline is 6312f39dec1bb6cd083650fc7fd2570c4f708bde, which separately merges that same exact c4284a3 dependency into 4b92da4. The new paired costs isolate the timestamp guard with this dependency present on both sides. They do not measure the complete published 4b92da4 → final-candidate change. The only product difference between these matched sources is the guard; their other difference is the new public tests. Historical speedups below retain their original source/environment scope and are not replaced by these guard-cost measurements.

Current candidate validation

Both independently built sources pass formatting, 394 client Rust unit tests, 95 core Rust unit tests, and compilation of every native and WASM test target. The complete Python inventory is test_dropin_offline.py, test_arrow_handoff.py and test_arrow_e2e.py:

Installed source	pandas	Passed	Failed	Live-server skips
Final candidate	2.3.3	648	0	11
Final candidate	3.1.0	648	0	11
Matched unguarded baseline	2.3.3	614	2 expected semantic failures	11
Matched unguarded baseline	3.1.0	612	4 expected semantic failures	11

The two baseline failures common to both versions are explicit integer _data read-order cases: the unguarded timestamp eligibility path reads labels before row normalization. The additional two pandas 3.1 failures preserve values/nulls/schemas and zero Config RPC but fail exact string-backing to_numpy mutation order. The candidate fixes all four. A first baseline invocation mistakenly named nonexistent test_pandas_input.py and exited 4 without running tests; that failed invocation and all subsequent expected baseline failures are preserved separately. Passing native builds from that attempt were retained.

The candidate adds 20 public negative cases and 12 canonical-storage positive cases. Independent standalone probes pass 20/20 on both pandas versions, as does an explicit native-row control. The unguarded baseline passes 2/20 and fails the other 18 semantically, with zero setup failures: eight raising tolist hooks bypassed, eight mutating hooks bypassed, and two _data order failures. The two _values probes already pass on the baseline and establish no regression sensitivity. Every successful delivery checks all 10,000 values/nulls and complete physical and logical schemas.

Fresh release wheels are source-bound to all 259 Python modules and the installed native extension in four isolated Rust environments, with exact executable/prefix, imports/spec origins, direct URL and dependency checks. Wheel SHA256: baseline 925a238c54a59872a93ed85665e50e5c620db15241a3b1b979986ddaa95f101a; candidate a10d8ecbaf78793466618b57213558934e96ad730f8a52afe0eaafe07bed4a2e. Both native binaries are byte-identical: 9ef8c0accbafc7fb1f7c1b2bc76201e4c20957c30ce2bc270f3d45fb4fca56fe.

Matched timestamp-guard costs and correctness

All 540 final samples passed: 300 main samples plus 240 focused samples, five per role/case/profile. The pandas 2.3 main catalog uses the original 13 warm cases × three clients, plus three first-construction guards × three clients. The pandas 3.1 main supplement uses the Rust pair for three cases and both profiles. Focused supplements cover ns/us accepted input, a representative instance mutation fallback and an expected hook error on both pandas versions/profiles.

Independent saved-byte review reconstructs all 430 received IPC requests / 12,400,000 cells, including complete typed values/nulls, payload hashes and full physical/logical schemas; it also verifies 80 expected-error receipts and 30 frozen return samples. All 46 case/profile comparisons have identical complete 20-counter RPC maps at each paired sample index (230 matched RPC-map pairs). This equality does not mean every case has zero RPCs: accepted timestamps can request Config; all measured error counters are zero.

The final costs show observable guard costs and noise. For example, pandas 2.3 warm timestamp wall median is 2.121 → 2.342 ms and CPU 1.183 → 1.479 ms; pandas 3.1 first timestamp wall is 4.149 → 4.766 ms and CPU 2.539 → 2.965 ms. The warm pandas 2.3 string, array and date wall/CPU medians also increase. The largest positive held/sampled-peak/process-peak RSS median difference is +1.562 MiB for warm pandas 2.3 strings; its post-GC difference is +1.188 MiB. All measured wall/CPU ranges and every positive RSS median movement are disclosed below. There is no latency-neutral, memory-neutral or zero-overhead claim.

Measurement boundaries and limits:

These are client scripted-fixture costs, with no JVM or live Spark execution. The frozen harness revision is 1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4; original main operations, timing boundaries, validators and RSS sampler are unchanged. Additive proof and explicit focused fixtures are separately preserved. Builds/tests/measurements ran serially under the campaign lock and an inner deadline guard.
Warm uses one constructor warmup. First construction follows imports, input creation, session creation and an untimed version request; it excludes those preparation costs and is not a cold interpreter measurement. No total cold-start claim is made.
Accepted ns/us compares the correct matched baseline with the candidate. Fallback/error compares the candidate with an in-memory native-row control in the same candidate installation, disabling both eligibility helpers. The unguarded baseline wrongly bypasses those hooks and is excluded as a valid fallback/error performance control. These control deltas include all eligibility checks, rather than isolating the new two lines. Other hook sites are covered by public correctness tests, not this representative timing fixture.
Expected-error timing includes the same catch/tuple wrapper in both roles. The error-path candidate-minus-control elapsed median differences are 9.417–26.334 µs; CPU median differences are 10–27 µs. The 2 ms RSS sampler can miss short peaks.
Each final main input receipt exclusively saves the exact already-read request bytes after timing. This proof copy occurs after elapsed/CPU/held/sampled peak/process peak/RPC measurements; it may affect post-GC RSS. Focused Rust inputs retain their per-session raw requests. RSS values are absolute process values, not isolated allocations by the guard.
Official reference uses Python 3.12.14, pyspark-client 5.0.0.dev0 from Apache Spark source ca1514a2b4e72f9529871f34d09881cd4921df82, pandas 2.3.3, NumPy 2.5.4, PyArrow 26.0.0, grpcio 1.84.1 and psutil 7.2.2. Its actual runtime proof checks all 366 Python modules, executable/prefix, direct URL and import origins. There is no official pandas 3.1 timed comparison; the prior official pandas 3.1 timeout is unresolved and preserved.
The original one-sample preflight reported 108 in-process successes (86 IPC receipts, 16 errors, six returns). Independent review then found that later same-session official ML clean_cache ExecutePlan requests overwrote all 14 original official input capture files. Those original 636,000 cells cannot be reconstructed from the old raw bytes; only 72 original captures / 1,844,000 cells remain reconstructable. Original receipts/results/failures are preserved. An additive untimed immutable request copy was reviewed before final execution. A separate fresh official-only 11-warm + three-first proof supplement passes 14/14 and validates 636,000 cells, with hashes checked after worker exit/cleanup. It does not recover the original lost bytes or establish paired performance. No successful Rust/focused preflight workload was repeated.

All 46 final wall and CPU distributions, including adverse movements

Each value is median[min,max] of five fresh samples. B is the matched Rust baseline; N is the same-install native-row control. All units are milliseconds except expected-error rows, which use microseconds. Official wall is shown only where an official process was timed. Role medians are not medians of the paired sample differences; overlapping ranges do not establish neutrality or significance.

main / pandas 2.3 / warm

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
input-tuples-explicit-10000x3	B	ms	2.092[1.819,5.080]	15.923[15.535,17.283]	16.008[15.166,16.138]	15.926[15.539,17.279]	15.987[15.170,16.142]
input-pandas-explicit-10000x3	B	ms	1.159[1.048,1.307]	17.364[17.023,18.473]	17.340[16.492,18.125]	17.322[17.027,18.470]	17.325[16.497,17.735]
input-pandas-inferred-10000x3	B	ms	1.087[1.057,1.272]	16.696[15.479,17.571]	16.550[16.240,18.771]	16.699[15.482,17.578]	16.553[16.243,18.774]
input-arrow-explicit-10000x3	B	ms	0.649[0.538,0.880]	1.319[0.985,1.403]	1.188[0.994,2.516]	0.552[0.467,0.701]	0.553[0.477,1.042]
return-timestamp-toArrow	B	ms	2.891[2.849,2.946]	0.896[0.713,6.301]	0.816[0.734,1.396]	0.698[0.509,0.918]	0.589[0.515,0.896]
return-timestamp-toPandas	B	ms	4.045[3.382,4.198]	1.097[1.007,6.926]	0.999[0.989,1.030]	0.874[0.770,1.237]	0.808[0.802,0.840]
input-scale-pandas-1000x3	B	ms	0.856[0.735,1.338]	1.437[1.173,7.017]	1.513[1.232,4.685]	0.764[0.658,1.337]	0.756[0.691,1.079]
input-scale-pandas-100000x3	B	ms	1.401[1.274,1.585]	5.895[1.436,7.248]	1.702[1.489,5.611]	1.228[0.939,1.442]	1.022[0.949,1.167]
input-type-pandas-string-10000x3	B	ms	1.757[1.705,2.051]	21.403[20.992,21.983]	22.155[21.417,24.845]	21.406[20.995,21.986]	22.103[21.421,24.842]
input-type-pandas-timestamp-10000x3	B	ms	1.211[1.049,1.345]	2.121[1.850,7.460]	2.342[1.818,7.104]	1.183[1.155,1.686]	1.479[1.125,1.822]
input-type-pandas-array-10000x3	B	ms	6.473[6.185,8.194]	42.357[41.389,47.595]	43.374[39.607,46.318]	42.360[41.393,47.599]	43.296[39.612,46.321]
input-type-pandas-date-10000x3	B	ms	1.908[1.797,2.328]	32.286[31.418,34.857]	32.948[32.132,33.901]	32.290[31.421,34.857]	32.945[32.136,33.891]
input-type-pandas-decimal-10000x3	B	ms	6.179[5.663,6.937]	25.668[25.107,26.385]	24.945[24.694,26.434]	25.672[25.111,26.388]	24.948[24.699,26.403]

main / pandas 2.3 / first-construction

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
input-scale-pandas-1000x3	B	ms	2.051[1.886,2.523]	3.949[3.709,4.717]	3.930[3.828,4.399]	3.003[2.916,3.144]	3.068[3.036,3.485]
input-type-pandas-string-10000x3	B	ms	2.990[2.965,3.950]	22.448[21.252,22.664]	22.647[21.674,23.292]	21.857[21.187,22.604]	22.060[21.628,23.229]
input-type-pandas-timestamp-10000x3	B	ms	2.753[2.380,4.838]	7.833[4.928,8.799]	5.613[4.750,13.043]	4.084[3.707,4.431]	4.624[3.594,6.032]

main / pandas 3.1 Rust pair / warm

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
input-scale-pandas-1000x3	B	ms	—	1.549[1.314,1.731]	1.466[1.359,1.488]	0.779[0.717,0.947]	0.764[0.743,0.825]
input-type-pandas-string-10000x3	B	ms	—	21.794[21.246,23.368]	21.570[21.085,23.605]	21.798[21.250,23.309]	21.573[21.089,23.578]
input-type-pandas-timestamp-10000x3	B	ms	—	1.910[1.736,7.537]	2.234[1.532,10.125]	1.132[0.977,1.593]	1.132[0.988,1.453]

main / pandas 3.1 Rust pair / first-construction

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
input-scale-pandas-1000x3	B	ms	—	3.468[3.046,3.869]	3.447[3.016,4.216]	2.299[2.183,2.674]	2.277[2.043,2.706]
input-type-pandas-string-10000x3	B	ms	—	24.005[23.553,24.558]	23.759[22.845,24.038]	23.846[23.378,24.239]	23.629[22.838,23.877]
input-type-pandas-timestamp-10000x3	B	ms	—	4.149[3.074,5.596]	4.766[3.406,8.527]	2.539[2.227,3.604]	2.965[2.487,3.072]

focused / pandas 2.3 / warm

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
timestamp-columns-ns-accepted-10000x1	B	ms	—	1.770[1.580,6.091]	1.715[1.564,7.458]	0.933[0.916,1.312]	0.938[0.910,1.614]
timestamp-columns-ns-fallback-mutate-10000x1	N	ms	—	14.694[14.347,15.589]	14.474[14.143,15.300]	14.698[14.350,15.592]	14.478[14.146,15.295]
timestamp-columns-ns-hook-error-10000x1	N	µs	—	12.292[11.417,14.083]	22.667[20.167,23.958]	16.000[13.000,17.000]	26.000[23.000,28.000]
timestamp-columns-us-accepted-10000x1	B	ms	—	1.466[1.398,1.638]	1.750[1.618,1.895]	0.788[0.767,0.840]	0.858[0.795,0.965]
timestamp-columns-us-fallback-mutate-10000x1	N	ms	—	14.337[13.854,15.212]	15.246[14.067,15.881]	14.341[13.858,15.201]	15.201[14.072,15.735]
timestamp-columns-us-hook-error-10000x1	N	µs	—	24.833[14.208,25.750]	51.167[28.958,92.000]	29.000[18.000,32.000]	56.000[32.000,65.000]

focused / pandas 2.3 / first-construction

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
timestamp-columns-ns-accepted-10000x1	B	ms	—	10.058[7.372,14.978]	7.238[6.897,19.155]	6.258[5.718,7.373]	5.686[5.493,8.465]
timestamp-columns-ns-fallback-mutate-10000x1	N	ms	—	14.870[14.406,15.526]	15.038[14.569,15.895]	14.873[14.411,15.495]	15.028[14.574,15.891]
timestamp-columns-ns-hook-error-10000x1	N	µs	—	18.541[15.208,27.708]	33.333[27.167,38.125]	21.000[19.000,31.000]	37.000[30.000,42.000]
timestamp-columns-us-accepted-10000x1	B	ms	—	11.122[7.023,11.687]	7.553[6.763,12.349]	5.505[5.234,5.971]	5.205[5.169,5.797]
timestamp-columns-us-fallback-mutate-10000x1	N	ms	—	15.705[14.314,17.459]	15.564[14.362,15.846]	15.666[14.318,17.423]	15.568[14.364,15.845]
timestamp-columns-us-hook-error-10000x1	N	µs	—	16.542[14.333,49.375]	35.417[32.292,90.500]	20.000[17.000,56.000]	38.000[36.000,96.000]

focused / pandas 3.1 / warm

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
timestamp-columns-ns-accepted-10000x1	B	ms	—	1.912[1.521,6.031]	1.657[1.549,6.541]	1.044[0.870,1.692]	0.953[0.935,1.504]
timestamp-columns-ns-fallback-mutate-10000x1	N	ms	—	16.281[16.015,17.069]	16.495[15.982,17.256]	16.284[16.017,17.041]	16.497[15.985,17.243]
timestamp-columns-ns-hook-error-10000x1	N	µs	—	15.708[10.417,28.833]	25.125[23.041,34.000]	18.000[14.000,31.000]	30.000[27.000,38.000]
timestamp-columns-us-accepted-10000x1	B	ms	—	1.524[1.498,2.165]	1.546[1.512,1.923]	0.866[0.798,1.293]	0.857[0.805,0.895]
timestamp-columns-us-fallback-mutate-10000x1	N	ms	—	16.306[15.702,16.582]	16.314[15.795,16.875]	16.310[15.706,16.566]	16.317[15.798,16.879]
timestamp-columns-us-hook-error-10000x1	N	µs	—	11.542[10.167,14.625]	23.709[22.500,93.750]	15.000[14.000,18.000]	27.000[26.000,97.000]

focused / pandas 3.1 / first-construction

Case	Control	Unit	Official wall	Control wall	Candidate wall	Control CPU	Candidate CPU
timestamp-columns-ns-accepted-10000x1	B	ms	—	5.394[5.298,9.967]	6.487[5.427,9.823]	4.348[4.082,5.177]	4.251[4.127,4.467]
timestamp-columns-ns-fallback-mutate-10000x1	N	ms	—	16.811[16.495,18.785]	16.944[16.483,17.894]	16.815[16.498,18.421]	16.947[16.488,17.816]
timestamp-columns-ns-hook-error-10000x1	N	µs	—	16.334[15.875,19.375]	29.792[27.333,39.666]	20.000[19.000,24.000]	32.000[30.000,43.000]
timestamp-columns-us-accepted-10000x1	B	ms	—	9.065[5.245,13.504]	6.141[5.893,14.578]	5.257[4.040,7.230]	5.009[4.345,6.567]
timestamp-columns-us-fallback-mutate-10000x1	N	ms	—	17.038[16.405,17.657]	17.214[16.977,17.886]	17.042[16.409,17.660]	17.218[16.976,17.850]
timestamp-columns-us-hook-error-10000x1	N	µs	—	17.708[15.000,18.625]	30.500[28.500,41.458]	20.000[18.000,22.000]	35.000[31.000,46.000]

Every positive RSS median movement, with both role ranges

MiB; each role is median[min,max], followed by candidate minus control role-median difference. Setup = baseline RSS before the operation; H = held RSS; S = sampled peak; P = process peak; G = post-GC RSS. H/S/P are combined only where both roles have identical complete five-sample distributions for those metrics. This table includes all positive role-median differences, however small; negative/zero movements and full same-index paired distributions are retained in the machine-readable results. These are absolute process values; varying setup RSS and overlapping ranges limit allocation attribution. Post-GC includes proof allocations.

main / pandas 2.3 / warm

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
input-tuples-explicit-10000x3	B	H/S/P	108.625[107.797,108.984]	108.828[107.328,110.531]	+0.203
input-pandas-inferred-10000x3	B	G	116.406[116.109,118.188]	116.609[116.156,118.109]	+0.203
input-arrow-explicit-10000x3	B	Setup	107.016[106.688,107.688]	107.469[106.906,108.078]	+0.453
input-arrow-explicit-10000x3	B	H/S/P	107.016[106.688,107.688]	107.469[106.906,108.094]	+0.453
input-arrow-explicit-10000x3	B	G	113.172[112.844,113.984]	113.797[113.047,114.641]	+0.625
return-timestamp-toArrow	B	Setup	103.172[102.688,103.984]	103.203[102.734,104.109]	+0.031
return-timestamp-toArrow	B	H/S/P	103.172[102.688,104.000]	103.328[102.734,104.109]	+0.156
return-timestamp-toArrow	B	G	103.781[103.281,104.594]	103.891[103.312,104.672]	+0.109
input-type-pandas-string-10000x3	B	Setup	111.641[111.391,112.141]	111.812[111.469,112.203]	+0.172
input-type-pandas-string-10000x3	B	H/S/P	111.641[111.391,112.141]	113.203[111.469,113.734]	+1.562
input-type-pandas-string-10000x3	B	G	119.297[119.047,119.422]	120.484[119.172,121.844]	+1.188
input-type-pandas-array-10000x3	B	Setup	121.391[120.859,121.969]	121.625[121.172,121.922]	+0.234
input-type-pandas-array-10000x3	B	H/S/P	121.391[120.875,125.250]	121.625[121.188,122.938]	+0.234
input-type-pandas-decimal-10000x3	B	G	120.750[120.656,121.531]	120.828[120.609,122.375]	+0.078

main / pandas 2.3 / first-construction

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
input-scale-pandas-1000x3	B	Setup	101.766[101.516,102.656]	101.891[101.641,102.297]	+0.125
input-scale-pandas-1000x3	B	H/P	104.562[104.375,105.469]	104.641[104.469,105.094]	+0.078
input-scale-pandas-1000x3	B	S	104.562[104.375,105.484]	104.641[104.484,105.094]	+0.078
input-scale-pandas-1000x3	B	G	110.281[110.188,111.250]	110.375[110.188,110.547]	+0.094
input-type-pandas-timestamp-10000x3	B	Setup	104.266[104.094,104.781]	104.312[104.203,105.312]	+0.047
input-type-pandas-timestamp-10000x3	B	H/S/P	107.984[107.828,108.484]	108.547[108.094,109.047]	+0.562
input-type-pandas-timestamp-10000x3	B	G	113.484[113.188,113.938]	114.016[113.688,114.812]	+0.531

main / pandas 3.1 Rust pair / warm

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
input-type-pandas-string-10000x3	B	Setup	114.828[114.703,115.812]	114.984[114.781,117.203]	+0.156
input-type-pandas-string-10000x3	B	H/S/P	114.828[114.703,116.000]	114.984[114.781,119.203]	+0.156
input-type-pandas-string-10000x3	B	G	119.750[119.500,121.672]	120.359[119.719,125.312]	+0.609
input-type-pandas-timestamp-10000x3	B	Setup	108.938[108.875,109.781]	109.016[108.922,110.266]	+0.078
input-type-pandas-timestamp-10000x3	B	H/S/P	108.938[108.875,110.000]	109.031[108.922,110.375]	+0.094

main / pandas 3.1 Rust pair / first-construction

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
input-type-pandas-timestamp-10000x3	B	Setup	106.031[105.953,107.266]	106.703[105.719,106.891]	+0.672
input-type-pandas-timestamp-10000x3	B	H/S/P	109.109[109.094,110.312]	109.828[108.797,110.203]	+0.719
input-type-pandas-timestamp-10000x3	B	G	113.906[113.797,115.297]	114.172[113.625,115.141]	+0.266

focused / pandas 2.3 / warm

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
timestamp-columns-ns-accepted-10000x1	B	Setup	105.938[105.828,106.672]	106.391[105.766,107.125]	+0.453
timestamp-columns-ns-accepted-10000x1	B	H/S/P	105.938[105.828,106.672]	106.391[105.766,107.141]	+0.453
timestamp-columns-ns-accepted-10000x1	B	G	111.609[111.609,112.422]	112.078[111.500,112.578]	+0.469
timestamp-columns-ns-hook-error-10000x1	N	Setup/H/S/P	101.781[101.562,102.141]	101.812[101.609,102.469]	+0.031
timestamp-columns-ns-hook-error-10000x1	N	G	101.781[101.578,102.141]	101.812[101.609,102.469]	+0.031
timestamp-columns-us-accepted-10000x1	B	Setup/H/S/P	105.766[105.656,106.609]	105.812[105.672,105.922]	+0.047
timestamp-columns-us-fallback-mutate-10000x1	N	Setup	108.047[107.719,108.672]	108.406[107.672,108.641]	+0.359
timestamp-columns-us-fallback-mutate-10000x1	N	H/S/P	108.547[108.219,109.328]	109.344[108.172,110.062]	+0.797
timestamp-columns-us-fallback-mutate-10000x1	N	G	114.391[113.641,115.203]	115.234[113.578,115.562]	+0.844
timestamp-columns-us-hook-error-10000x1	N	Setup/H/S/P	102.531[101.984,102.672]	102.656[102.547,103.094]	+0.125
timestamp-columns-us-hook-error-10000x1	N	G	102.547[102.000,102.672]	102.656[102.547,103.109]	+0.109

focused / pandas 2.3 / first-construction

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
timestamp-columns-ns-fallback-mutate-10000x1	N	Setup	101.766[101.594,102.375]	102.453[101.812,102.609]	+0.688
timestamp-columns-ns-fallback-mutate-10000x1	N	H/S/P	107.828[107.750,108.359]	108.250[107.922,108.391]	+0.422
timestamp-columns-ns-fallback-mutate-10000x1	N	G	113.859[113.688,114.844]	114.594[113.906,115.141]	+0.734
timestamp-columns-us-fallback-mutate-10000x1	N	H/S/P	107.766[107.578,109.000]	107.844[107.750,108.734]	+0.078
timestamp-columns-us-fallback-mutate-10000x1	N	G	113.781[113.562,114.906]	113.812[113.609,115.328]	+0.031
timestamp-columns-us-hook-error-10000x1	N	Setup	101.750[101.703,102.953]	102.328[101.766,102.938]	+0.578
timestamp-columns-us-hook-error-10000x1	N	H/S/P	101.766[101.719,102.969]	102.344[101.781,102.953]	+0.578
timestamp-columns-us-hook-error-10000x1	N	G	101.766[101.734,102.984]	102.344[101.781,102.953]	+0.578

focused / pandas 3.1 / warm

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
timestamp-columns-ns-accepted-10000x1	B	Setup	106.516[106.375,107.484]	106.562[106.391,108.156]	+0.047
timestamp-columns-ns-accepted-10000x1	B	H/S/P	106.516[106.375,107.516]	106.578[106.406,108.188]	+0.062
timestamp-columns-ns-accepted-10000x1	B	G	112.156[111.922,112.969]	112.297[112.016,113.203]	+0.141
timestamp-columns-ns-fallback-mutate-10000x1	N	Setup	108.938[108.781,110.000]	109.125[108.859,109.844]	+0.188
timestamp-columns-ns-fallback-mutate-10000x1	N	H/S/P	108.984[108.844,110.062]	109.203[108.922,110.219]	+0.219
timestamp-columns-us-fallback-mutate-10000x1	N	Setup	108.812[108.766,109.469]	108.859[108.781,109.344]	+0.047
timestamp-columns-us-fallback-mutate-10000x1	N	H/S/P	108.875[108.828,109.516]	108.922[108.844,109.406]	+0.047
timestamp-columns-us-hook-error-10000x1	N	Setup/H/S/P	102.656[102.500,102.781]	102.859[102.688,102.875]	+0.203
timestamp-columns-us-hook-error-10000x1	N	G	102.672[102.531,102.797]	102.875[102.703,102.891]	+0.203

focused / pandas 3.1 / first-construction

Case	Control	Metric	Control MiB	Candidate MiB	Δ median MiB
timestamp-columns-ns-fallback-mutate-10000x1	N	Setup	102.781[102.594,104.188]	102.828[102.578,103.844]	+0.047
timestamp-columns-ns-fallback-mutate-10000x1	N	H/S/P	109.047[108.766,109.734]	109.125[108.828,109.625]	+0.078
timestamp-columns-ns-fallback-mutate-10000x1	N	G	114.047[113.812,114.875]	114.141[113.766,114.469]	+0.094
timestamp-columns-us-fallback-mutate-10000x1	N	G	114.109[113.922,115.391]	114.406[113.719,115.656]	+0.297
timestamp-columns-us-hook-error-10000x1	N	Setup	102.672[102.609,102.766]	102.688[102.484,103.000]	+0.016
timestamp-columns-us-hook-error-10000x1	N	H/S/P	102.688[102.625,102.781]	102.703[102.500,103.016]	+0.016
timestamp-columns-us-hook-error-10000x1	N	G	102.703[102.641,102.797]	102.719[102.531,103.031]	+0.016

Frozen evidence identity

The final report and independent raw audit bind the exact candidate and matched baseline above. cost-distributions.json retains all 322 paired seven-metric distributions, every raw value/range, same-index paired differences and complete RPC maps. The CSV and report are derived from those retained sample artifacts.

Artifact	SHA256
`source-finalization.json`	`d0fff116a3b03a73188a0549ece61bf2cbbb0bf5c78bb9d272e13ceef1c45bb6`
Candidate `_pandas_input.py`	`976123c5acfd2fa1ecbf6277b8ab18dbd9a824252c8f77dbfd489357ad403e72`
Candidate `test_arrow_handoff.py`	`3efc940bd2af4a4f1517ed5d1cfd0e26b76f4e0cfde74c9e58f388dca79bb7ff`
`cost-distributions.json`	`58d046685493bedb4b38cae81e9fbb4028b966812142e1ce9a4483b6ceaaae65`
`cost-summary.csv`	`8c403b52ade81c9fc32c2d71985c6c5c04c64bb851edc94cb4363d2d8578983a`
`REPORT.md`	`84cc0c0bad2f8f87da165461e72160b3d3c38802ef85f2678754eb7b8c3db992`
Independent final raw audit script	`c2ec22809749d6ab33f5e27b477b4c9f8c07fbc02d7a638febee7bf37655d403`
Parent final evidence acceptance	`4e5b05bc29bd1742c71d686bab97f3432c495e2917a0d4f1cae4ce693d9f2040`

Historical performance and protocol evidence

These results belong to the revisions specified here. They establish the original timestamp path's benefit and earlier eligibility behavior; they are not new measurements of the final candidate, its dependency merge, or the complete published-head-to-candidate change.

Initial timestamp implementation

Candidate 979fadc8d2aba3dad757641f81973afa62aa7ab6 versus identical PR121 dependency baseline a8774f0bc8289856b98d11ddbb6b0c0aff95eaee, frozen harness 1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4, official source ca1514a2b4e72f9529871f34d09881cd4921df82:

Timestamp constructor, 10,000 × 3	Median ms	Range ms
Official Python Connect	1.321	1.225–1.524
PR121 baseline	62.533	61.747–63.928
Historical candidate `979fadc8`	2.126	1.933–2.199

That source measured 29.4× faster than the identical PR121 baseline. Constructor validation was unchanged; decoded IPC established correctness independently. The original 13-case × three-client × five-sample run passed 195/195; a further ten-sample integer guard passed 30/30. The small 1,000-row integer guard initially moved +0.159 ms (+12.2%); the followup measured 1.495 → 1.354 ms, with overlapping ranges in both runs. Raw noisy samples are retained. Candidate wheel SHA256 705cfe0787f713fb6dbd9dfe6d7ad2db4f00778ae4ed22f8ef5c50d762c3981f; baseline e6a84acc30979fa171ef94bbdc2b575ae4f0ac70579e3282f27337603193c879.

At that source, 394 Rust library tests passed, all workspace release/locked test targets compiled, and 366 Python tests passed with nine existing skips. Independent official/baseline/candidate processes verified 17 complete-IPC scenarios per client: UTC/non-UTC host/session zones, aware/naive values, DST folds/transitions, NaT, negative/boundary values, precision and schema fallbacks.

Earlier eligibility followup

Candidate 5f31a4c012300e7f18c17829223102a186084fef retained original PR121 dependency a8774f0 on both measured Rust sides. Its 63 bounded samples passed: timestamp baseline 60.276 → candidate 2.144 ms (28.1×), official 1.255 ms. Against prior PR130 979fadc8, it measured 2.438 → 2.189 ms with overlapping ranges. Integer/fallback/Arrow input/Arrow return guard ranges also overlapped. Its wheel SHA256 is 1effffcc6eef70fdec891757d29c142f4ab68d756e092ac79c09a96626dc29ea.

At that source, 443 Python tests passed with nine existing skips; native/WASM test targets compiled and native bytes remained unchanged. Reference/prior/followup processes passed 17 full-IPC scenarios each (51 runs), checking every timestamp/null, physical name/type/nullability and complete logical metadata. Candidate/prior physical schemas were byte-identical; the permitted official difference was redundant Arrow metadata encoding. The public converter-alias regression failed on the prior wheel as expected. Tests also covered custom datetime storage, schema descriptors, captured converter aliases, pre-Config null errors and local converter imports.

Previously published upstream reconciliation

A normal merge of upstream d9defae6f691671d8bad4927f58a7283cd0a0b50 into 5f31a4c produced published PR130 4b92da42bd765b812052a3ec116b2d006789b78d, retaining the input implementation byte for byte and bringing upstream workflow/docs, an exporter comment and handoff guards. No timing rerun was claimed for that reconciliation. Fresh source/wheel checks passed 394 client / 95 core Rust units, all native/WASM test-target compilation, and 451 Python tests / 11 skips on each pandas 2.3.3 and 3.1.0. Wheel SHA256 33f3a7bb5bb0993891cfa7f2406d915d408d5247c29fd37e53b868ef2f0592a0; all 259 source/wheel/installed Python modules matched and native bytes remained identical. Two restricted-sandbox attempts failed unchanged bearer-metadata TLS setup; a retained same-source run with normal system test access passed that test and the full core suite. No certificate/authentication/configuration changes were made.

Known fallback limits

The supported columnar path matches official reference microseconds at boundaries where the old pandas-object/datetime path and floating timestamp rounding lose precision, including a lower-bound ns sign/wrap case. Unsupported row fallbacks preserve pre-existing differences: naive timestamps use the host zone rather than the session zone, DST overlap/gap behavior differs, and -1 ns truncation differs. This PR does not extend those fallback semantics.

JIRA: https://issues.apache.org/jira/browse/SPARK-60686

This pull request and its description were written by Isaac.

Provide an alternative to the per-batch C Data handoff in PR apache#103 while retaining PyO3 0.28.3. Add ownership, buffer-sharing, compatibility, and public client handoff tests. Generated-by: OpenAI Codex

Apply LC's suggestion to export one boxed RecordBatchReader through arrow-pyarrow and materialize it with read_all. Remove the custom stream callbacks, unused converter crate, and dedicated converter workflow. Keep schema consistency checks, standalone schema preflight, and the NUL schema IPC fallback. Retain handoff tests in the coverage workflow and keep PyO3 at 0.28.3. Generated-by: OpenAI Codex (application version unavailable).

Keep public behavior coverage and compile mock protocol descriptors from source. Add native toArrow smoke cases, remove the unproven schema preflight, and avoid the arrow module name collision. Leave CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>

Document the Arrow 58.3 NUL export panic at its upstream source and wrap the architecture paragraph. Keep CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>

…am interface Normalize Arrow schemas and timestamps with the existing PySpark helpers, then serialize imported Rust batches directly for local relations without Python row materialization. Co-authored-by: Isaac <no-reply@databricks.com>

Honor server local-relation settings, split Arrow IPC into independently readable chunks, and upload session-scoped cache artifacts with SHA-256 deduplication and CRC validation. Keep large input payloads out of query plans and test uploads above the gRPC message limit. Co-authored-by: Isaac <no-reply@databricks.com>

…schemas

…ations

…umPy frames

…ow CI suite

… pandas input Authenticate concrete datetime storage and native schema/converter access before selecting the columnar path. Reject nonnullable NaT locally before requesting timezone configuration and keep local converter imports compatible. Co-authored-by: Isaac <no-reply@databricks.com>

…mantics Use explicit LongType schema names when constructing Arrow columns. Limit the columnar path to concrete pandas/NumPy arrays and native schema descriptors, including authentic converter aliases retained by already imported modules. Keep nullable values in non-nullable fields on the native row path so local validation still precedes configuration RPCs. Support warm Arrow importers that resolve converters locally. Add mock-RPC regressions for all emitted integer cells, schema metadata and nullability, custom conversion hooks, and repeated warm construction. Co-authored-by: Isaac <no-reply@databricks.com>

Build the subclass-label fallback fixture with an explicit object Index and assert its concrete label type. Pandas 3 otherwise normalizes constructor labels to builtin strings before the unchanged integer helper sees them. The original regression passes on pandas 2.3.3 and fails on 3.1.0; the fixed regression and full CI-selected Arrow/drop-in suites pass on both versions. Runtime Python and native sources remain unchanged. Co-authored-by: Isaac <no-reply@databricks.com>

Merge current Apache master d9defae while retaining all pandas integer input guards and the pandas 3 fixture repair. Keep upstream Arrow output regressions, CI protoc requirement and live smoke step; preserve fork input bridge/docs and stable-buffer assertions. Co-authored-by: Isaac <no-reply@databricks.com>

Reconcile the merged Arrow export master without rewriting the pandas timestamp branch. Retain every timestamp eligibility and pandas input regression, plus the final upstream output guards, stable-buffer and CI protoc checks. Co-authored-by: Isaac <no-reply@databricks.com>

Decline noncanonical columns indexes and their direct string-storage hooks before the integer fast path inspects labels or normalizes cells. Keep native tolist errors and mutations in their original order and cover public construction on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>

Decline class-level columns storage descriptors before reading labels. Keep the single native columns storage read before pandas normalization, and cover public construction with inferred, named and explicit schemas on both pandas versions. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com> # Conflicts: # python/tests/test_arrow_handoff.py

Decline customized pandas Index conversion and storage before timestamp eligibility reads labels. Keep native row conversion and errors in order. Cover public columns hooks, storage descriptors and ordinary ns/us labels. Co-authored-by: Isaac <no-reply@databricks.com>

dbtsai and others added 23 commits October 28, 2026 17:17

Export Arrow tables through the C Stream interface

d7cebc3

Provide an alternative to the per-batch C Data handoff in PR apache#103 while retaining PyO3 0.28.3. Add ownership, buffer-sharing, compatibility, and public client handoff tests. Generated-by: OpenAI Codex

Skip Arrow handoff tests when protoc is unavailable

fe79d54

Document the Arrow 58.3 NUL export panic at its upstream source and wrap the architecture paragraph. Keep CI workflows unchanged. Co-authored-by: Isaac <no-reply@databricks.com>

[SPARK-60676][CONNECT] Use Arrow columns for signed int64 pandas inputs

4a5807e

[SPARK-60676][CONNECT] Validate integer upload scales against mock RPC

38568fe

[SPARK-60676][CONNECT] Cover leading null inference and non-nullable …

7009bb7

…schemas

[SPARK-60676][CONNECT] Exercise named schemas through mock rename rel…

fee0800

…ations

[SPARK-60676][CONNECT] Avoid Arrow configuration overhead for small N…

50b00d1

…umPy frames

[SPARK-60676][CONNECT] Retain IPC row fallback for NUL field names

a8774f0

[SPARK-60676][CONNECT] Include pandas regressions in the existing Arr…

e7e3094

…ow CI suite

[SPARK-60686][CONNECT] Use Arrow columns for supported pandas timestamps

979fadc

[SPARK-60686] Include reviewed canonical integer input dependency

fd115ce

Co-authored-by: Isaac <no-reply@databricks.com> # Conflicts: # python/tests/test_arrow_handoff.py

Navigation Menu

Repository navigation

[SPARK-60686][CONNECT] Use Arrow columns for supported pandas timestamps - #130

[SPARK-60686][CONNECT] Use Arrow columns for supported pandas timestamps#130
dbtsai wants to merge 23 commits into
apache:masterfrom
dbtsai:codex/spark-60686-pandas-timestamps

dbtsai commented Oct 29, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dbtsai commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Source and dependency scope

Current candidate validation

Matched timestamp-guard costs and correctness

main / pandas 2.3 / warm

main / pandas 2.3 / first-construction

main / pandas 3.1 Rust pair / warm

main / pandas 3.1 Rust pair / first-construction

focused / pandas 2.3 / warm

focused / pandas 2.3 / first-construction

focused / pandas 3.1 / warm

focused / pandas 3.1 / first-construction

main / pandas 2.3 / warm

main / pandas 2.3 / first-construction

main / pandas 3.1 Rust pair / warm

main / pandas 3.1 Rust pair / first-construction

focused / pandas 2.3 / warm

focused / pandas 2.3 / first-construction

focused / pandas 3.1 / warm

focused / pandas 3.1 / first-construction

Historical performance and protocol evidence

Initial timestamp implementation

Earlier eligibility followup

Previously published upstream reconciliation

Known fallback limits

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dbtsai commented Oct 29, 2026 •

edited

Loading