Skip to content

[SPARK-60684][CONNECT] Fast-path exact built-in string input - #127

Open
dbtsai wants to merge 5 commits into
apache:masterfrom
dbtsai:codex/spark-60684-exact-str
Open

dbtsai wants to merge 5 commits into
apache:masterfrom
dbtsai:codex/spark-60684-exact-str

Conversation

@dbtsai

@dbtsai dbtsai commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Fast-path exact built-in Python str values in py_to_value before bool, numeric, and Decimal probing. String subclasses and failed UTF-8 extraction retain the existing generic fallback, including numeric conversion protocols and surrogate errors. The numeric conversion code is unchanged.

The offline string tests reuse _mock_connect_server and its session teardown. The shared service captures raw outbound local-relation Arrow IPC through named, documented protobuf fields; tests decode it locally after checking optional dependencies with importorskip. The server has no PyArrow dependency, and Arrow decoding errors surface directly in the tests. Coverage includes Unicode/NUL, surrogate errors, subclass protocol order, nested strings/nulls, inferred/named/explicit schemas, and 256-row tuple/pandas batches. The spawn target lives in the shared helper module, so it is independently importable.

Why are the changes needed?

Local string input currently pays generic scalar detection for every value. The original benchmark used five fresh-process samples per client/case on Apple M3 Max with arm64 Python 3.12.14 and release builds. Medians for 10,000 × 3 string inputs were:

Source Baseline Candidate Speedup
tuples 20.493 ms 3.930 ms 5.21×
pandas 22.310 ms 5.313 ms 4.20×

The Python reference measured 2.843 ms and 1.784 ms respectively. All 195 samples across 13 cases and three clients passed the unchanged validators. Numeric, Decimal, date, timestamp, array, nested-result, and RPC guards showed no material regression; small differences remained noisy with five samples.

These measurements predate the rebase and test cleanup: baseline d884e347452c6446c7bc54af151755c62801d8cd, Python reference ca1514a2b4e72f9529871f34d09881cd4921df82, frozen benchmark harness 1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4. The correctness tests inspect actual encoded IPC; the benchmark validates construction results separately.

Does this PR introduce any user-facing change?

Faster construction for exact built-in string values. Subclasses retain their existing numeric conversion behavior, and failed UTF-8 extraction follows the existing error path.

How was this patch tested?

Validation after rebasing onto master 617fa1481b3e6fc7b825928130f280506328c9b1 and cleaning up the tests:

  • 398 Rust library tests passed. Release integration-test compilation, WASM-feature integration-test compilation, cargo fmt --all --check, release PyO3 extension build, and git diff --check passed.
  • All 340 tests in python/tests/test_dropin_offline.py passed with normal imports and with --import-mode=importlib. Importlib mode includes python/tests on PYTHONPATH so the shared helper is available to spawned interpreters.
  • A separate lifecycle check verified that the shared context manager stops its session and that consecutive contexts create fresh sessions and successfully query their live mock servers.
  • Advisory workspace Clippy still fails on existing Rust diagnostics. Every reported primary error location maps to unchanged code in master and the prior PR head; the cleanup changes only Python tests and their helper.

The protocol tests use scripted localhost fixtures. Live Spark/JVM behavior was not exercised by this validation.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6)

This pull request and its description were written with OpenAI Codex (GPT-6).

@dbtsai
dbtsai requested review from HyukjinKwon and viirya October 30, 2026 18:16
@dbtsai
dbtsai marked this pull request as ready for review October 30, 2026 18:17
@dbtsai
dbtsai force-pushed the codex/spark-60684-exact-str branch from 7633018 to 7ea45dd Compare October 30, 2026 18:21
dbtsai and others added 3 commits October 30, 2026 11:38
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant