Skip to content

[SPARK-60679][CONNECT] Cache successful SparkSession version lookups - #122

Draft
dbtsai wants to merge 3 commits into
apache:masterfrom
dbtsai:codex/spark-60679-cache-session-version
Draft

dbtsai wants to merge 3 commits into
apache:masterfrom
dbtsai:codex/spark-60679-cache-session-version

Conversation

@dbtsai

@dbtsai dbtsai commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

What changes

Repeated SparkSession.version access sends an AnalyzePlan RPC every time. Cache the first successful server version per session, matching Python Connect's cached property. Handles for the same session share the cache; newSession() and cloneSession() have independent caches. RPC failures and missing-result responses remain retryable, including after a stopped session's first lookup. Successful values, including the empty string, remain cached after stop(). The cache serializes concurrent first access, and the existing Python getter releases the GIL during the lookup.

Validated merge commit 961d68ed5a2d2ac62486ae6f2ee13b0cf80b522c incorporates Apache master 617fa1481b3e6fc7b825928130f280506328c9b1. The version-cache implementation is unchanged from the published PR. The test conflict is resolved by retaining master's complete offline suite and appending the existing 184-line version-cache block. Master's schema-cache, Arrow, mock-server, Cargo and workflow changes are preserved.

Validation

  • Fresh locked release wheel built from the merged tree. Current CI-selected scope, tests/test_dropin_offline.py tests/test_arrow_handoff.py, run from python/ with the native extension in the source package: 353 passed, zero skipped. The same test union with the master native baseline gives 345 passed and eight expected version-cache failures. Stop-before-first-lookup intentionally also passes on master.
  • All seven version test definitions / nine parametrized cases are retained: success and empty values, retry after RPC/missing-result failure, distinct sessions, shared active-session handles, concurrency, and stop before/after lookup. The Rust test checks cache identity across handles and distinct sessions. The version concurrency test observes one RPC across 24 reads; it does not deliberately stall the first RPC or independently test Python thread progress.
  • Rust release library tests: 399 client + 95 transport passed. All release integration tests compile, both normally and with wasm-udf. Formatting and diff checks pass.
  • Advisory Clippy reports 10 approximate-constant errors at existing test sites. Every error-containing file is byte-identical to master; a separate baseline Clippy run was not performed. The toolchain also warned that some debug information could not be stripped because bundled rust-objcopy could not load libLLVM.dylib; builds and tests completed.

Focused current-master benchmark

Unchanged frozen harness and validators; 120/120 samples validated: eight cases × three clients × five fresh processes. Separate baseline/candidate environments, wheel contents, module origins, source digests, dependency freezes and all 17 harness files were verified before and after. Measurements compare master 617fa1481b3e with the merged candidate on an arm64 macOS localhost gRPC fixture.

Case Master ms/op Candidate ms/op Candidate time change
version-first 0.519417 0.440667 -15.16%
version-repeat 0.18870084 0.000185 about 1,020× faster
schema-first 0.680125 0.621584 -8.61%
schema-repeat 0.000425 0.00044334 +4.32%
columns 0.00030084 0.00037250 +23.82%
withColumn 0.000367295 0.000398960 +8.62%
select-width512 0.00845625 0.00844417 -0.14%
missing-column 0.425725 0.3745918 -12.01%

These are medians of five process samples. First means the first version property/RPC after session setup: zero warmups, one timed read, exactly one version/AnalyzePlan RPC for every client. First-access ranges overlap (master 0.401834–1.709125 ms; candidate 0.403458–0.543083 ms), so this does not establish a first-access speedup. Repeat uses one warmup and 50 timed reads: every master sample makes 50 version/AnalyzePlan RPCs and sends 2,100 serialized request bytes; every candidate/reference sample makes zero. Repeat ranges are 0.14680750–1.16029666 ms on master and 0.000160–0.00023084 ms on candidate. Python reference repeat median is 0.00007334 ms/op.

All six guard cases preserve the full master/candidate RPC maps in every sample, including missing-column errors. Guard timings are exploratory: the adverse columns median is +23.82%, with overlapping ranges (master 0.000275–0.00035832 ms; candidate 0.00030416–0.00069166 ms). withColumn and schema-repeat ranges also overlap; these samples do not establish causality or general performance neutrality.

Client CPU medians, master → candidate, are 0.222 → 0.197 ms/op for first access and 0.0749 → 0.00026 ms/op for repeat. Median baseline/held/sampled-peak RSS in MiB is 100.906/101.156/101.156 → 100.859/101.062/101.062 for first access, and 101.031/101.062/101.062 → 101.234/101.234/101.234 for repeat. There is no lower-memory-footprint claim: tiny cases have only 1–3 RSS observations and roughly 3 ms observed polling gaps, so short peaks can be missed. CPU excludes server/observer work, lifetime RSS peaks include untimed setup, and serialized RPC payload bytes are not total network traffic.

This fixture measures client work; no JVM/live Spark suite was run. These measurements replace the earlier source-scoped benchmark numbers for the current-master comparison.

This pull request and its description were written by Isaac.

dbtsai and others added 3 commits October 28, 2026 22:05
Share the successful server version across handles for one session, isolate new sessions, and retry failures. Retain the cached value after stop, matching Python Connect.

Co-authored-by: Isaac <no-reply@databricks.com>
Move the nine version cache regression cases into the CI-selected offline suite with uniquely prefixed helpers and fixtures. Runtime code and workflows are unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant