Repository navigation
Conversation
Share the successful server version across handles for one session, isolate new sessions, and retry failures. Retain the cached value after stop, matching Python Connect. Co-authored-by: Isaac <no-reply@databricks.com>
Move the nine version cache regression cases into the CI-selected offline suite with uniquely prefixed helpers and fixtures. Runtime code and workflows are unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Repeated
SparkSession.versionaccess sends an AnalyzePlan RPC every time. Cache the first successful server version per session, matching Python Connect's cached property. Handles for the same session share the cache;newSession()andcloneSession()have independent caches. RPC failures and missing-result responses remain retryable, including after a stopped session's first lookup. Successful values, including the empty string, remain cached afterstop(). The cache serializes concurrent first access, and the existing Python getter releases the GIL during the lookup.Validated merge commit
961d68ed5a2d2ac62486ae6f2ee13b0cf80b522cincorporates Apache master617fa1481b3e6fc7b825928130f280506328c9b1. The version-cache implementation is unchanged from the published PR. The test conflict is resolved by retaining master's complete offline suite and appending the existing 184-line version-cache block. Master's schema-cache, Arrow, mock-server, Cargo and workflow changes are preserved.Validation
tests/test_dropin_offline.py tests/test_arrow_handoff.py, run frompython/with the native extension in the source package: 353 passed, zero skipped. The same test union with the master native baseline gives 345 passed and eight expected version-cache failures. Stop-before-first-lookup intentionally also passes on master.wasm-udf. Formatting and diff checks pass.rust-objcopycould not loadlibLLVM.dylib; builds and tests completed.Focused current-master benchmark
Unchanged frozen harness and validators; 120/120 samples validated: eight cases × three clients × five fresh processes. Separate baseline/candidate environments, wheel contents, module origins, source digests, dependency freezes and all 17 harness files were verified before and after. Measurements compare master
617fa1481b3ewith the merged candidate on an arm64 macOS localhost gRPC fixture.These are medians of five process samples. First means the first version property/RPC after session setup: zero warmups, one timed read, exactly one version/AnalyzePlan RPC for every client. First-access ranges overlap (master 0.401834–1.709125 ms; candidate 0.403458–0.543083 ms), so this does not establish a first-access speedup. Repeat uses one warmup and 50 timed reads: every master sample makes 50 version/AnalyzePlan RPCs and sends 2,100 serialized request bytes; every candidate/reference sample makes zero. Repeat ranges are 0.14680750–1.16029666 ms on master and 0.000160–0.00023084 ms on candidate. Python reference repeat median is 0.00007334 ms/op.
All six guard cases preserve the full master/candidate RPC maps in every sample, including missing-column errors. Guard timings are exploratory: the adverse columns median is +23.82%, with overlapping ranges (master 0.000275–0.00035832 ms; candidate 0.00030416–0.00069166 ms). withColumn and schema-repeat ranges also overlap; these samples do not establish causality or general performance neutrality.
Client CPU medians, master → candidate, are 0.222 → 0.197 ms/op for first access and 0.0749 → 0.00026 ms/op for repeat. Median baseline/held/sampled-peak RSS in MiB is 100.906/101.156/101.156 → 100.859/101.062/101.062 for first access, and 101.031/101.062/101.062 → 101.234/101.234/101.234 for repeat. There is no lower-memory-footprint claim: tiny cases have only 1–3 RSS observations and roughly 3 ms observed polling gaps, so short peaks can be missed. CPU excludes server/observer work, lifetime RSS peaks include untimed setup, and serialized RPC payload bytes are not total network traffic.
This fixture measures client work; no JVM/live Spark suite was run. These measurements replace the earlier source-scoped benchmark numbers for the current-master comparison.
This pull request and its description were written by Isaac.