Repository navigation
Conversation
dbtsai
marked this pull request as ready for review
October 30, 2026 18:17
dbtsai
force-pushed
the
codex/spark-60684-exact-str
branch
from
October 30, 2026 18:21
7633018 to
7ea45dd
Compare
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fast-path exact built-in Python
strvalues inpy_to_valuebefore bool, numeric, and Decimal probing. String subclasses and failed UTF-8 extraction retain the existing generic fallback, including numeric conversion protocols and surrogate errors. The numeric conversion code is unchanged.The offline string tests reuse
_mock_connect_serverand its session teardown. The shared service captures raw outbound local-relation Arrow IPC through named, documented protobuf fields; tests decode it locally after checking optional dependencies withimportorskip. The server has no PyArrow dependency, and Arrow decoding errors surface directly in the tests. Coverage includes Unicode/NUL, surrogate errors, subclass protocol order, nested strings/nulls, inferred/named/explicit schemas, and 256-row tuple/pandas batches. The spawn target lives in the shared helper module, so it is independently importable.Why are the changes needed?
Local string input currently pays generic scalar detection for every value. The original benchmark used five fresh-process samples per client/case on Apple M3 Max with arm64 Python 3.12.14 and release builds. Medians for 10,000 × 3 string inputs were:
The Python reference measured 2.843 ms and 1.784 ms respectively. All 195 samples across 13 cases and three clients passed the unchanged validators. Numeric, Decimal, date, timestamp, array, nested-result, and RPC guards showed no material regression; small differences remained noisy with five samples.
These measurements predate the rebase and test cleanup: baseline
d884e347452c6446c7bc54af151755c62801d8cd, Python referenceca1514a2b4e72f9529871f34d09881cd4921df82, frozen benchmark harness1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4. The correctness tests inspect actual encoded IPC; the benchmark validates construction results separately.Does this PR introduce any user-facing change?
Faster construction for exact built-in string values. Subclasses retain their existing numeric conversion behavior, and failed UTF-8 extraction follows the existing error path.
How was this patch tested?
Validation after rebasing onto master
617fa1481b3e6fc7b825928130f280506328c9b1and cleaning up the tests:cargo fmt --all --check, release PyO3 extension build, andgit diff --checkpassed.python/tests/test_dropin_offline.pypassed with normal imports and with--import-mode=importlib. Importlib mode includespython/testsonPYTHONPATHso the shared helper is available to spawned interpreters.The protocol tests use scripted localhost fixtures. Live Spark/JVM behavior was not exercised by this validation.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
This pull request and its description were written with OpenAI Codex (GPT-6).