[james-willis]

What changes were proposed in this pull request?

Declare #[pymodule(gil_used = false)] on the _pyspark module in crates/pyspark-rs/src/lib.rs, opting the pyspark-client-rust extension into free-threaded (no-GIL) CPython.

This is the first of the two steps identified in SPARK-59432. The second — adding a free-threaded (cp313t/cp314t) wheel build leg — is a separate build/packaging change (abi3 and the free-threaded ABI are mutually exclusive) and is left as a follow-up.

Why are the changes needed?

The original justificationfor this PR is struck out below but as Hyukjin points out in his comment, pyo3 flipped the default of this variable to false so this is effectively a no-op! So this PR only serves to make it explicit in the code that no-gil is supported.

On a free-threaded interpreter (CPython 3.13t/3.14t, PEP 703), CPython re-enables the GIL process-wide when it imports a PyO3 extension module that has not declared gil_used = false. A pure-Rust (tonic) Spark Connect client is well positioned to run with the GIL disabled — it pulls in no grpcio, which otherwise re-enables the GIL on import — so this declaration is a prerequisite for the extension to run GIL-off.

The declaration is truthful: the extension is already thread-safe. details in Jira ticket.

Does this PR introduce any user-facing change?

No. The attribute is a no-op on GIL-enabled interpreters. It has no runtime effect until a free-threaded wheel is published — the current wheels are cp39-abi3, which are not selectable on a free-threaded interpreter — and that wheel build leg is the tracked follow-up.

How was this patch tested?

Existing CI (cargo fmt --all --check and the workspace build); the attribute is inert on GIL-enabled builds and compiles under the current pyo3 0.28. End-to-end verification that the GIL is actually disabled (sys._is_gil_enabled() is False) will be added together with the free-threaded wheel build leg in the follow-up.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Anthropic)

[james-willis]

@HyukjinKwon This is a follow up from our linked in discussion

[HyukjinKwon]

Reviewed by building and inspecting against the pinned toolchain (pyo3 0.28.3). cargo fmt --all --check and cargo check -p pyspark-rs both pass, and the change is safe. One substantive concern about the rationale:

With pyo3 0.28 this is a functional no-op. In pyo3 0.23–0.27 free-threading was opt-in, so gil_used = false was indeed a prerequisite. But this repo pins pyo3 0.28 (Cargo.toml, lockfile 0.28.3), where the default flipped. From the 0.28 migration guide ("Default to supporting free-threaded Python"):

Modules now automatically allow use on free-threaded Python, unless they directly state they require the GIL with #[pymodule(gil_used = true)].

I confirmed this in the macro codegen: pyo3-macros-backend-0.28.3/src/module.rs computes gil_used = options.gil_used.is_some_and(|op| op.value.value), which is false when the attribute is absent, and with_gil_used(false) emits Py_MOD_GIL_NOT_USED. So:

• The plain #[pymodule] already on master emits Py_MOD_GIL_NOT_USED on a free-threaded build — this change produces byte-identical codegen.
• On the shipped abi3-py39 wheels, Py_mod_gil is gated behind #[cfg(Py_3_13)] (not set under the 3.9 limited API), so neither form emits the slot at all. (The "no-op on GIL-enabled builds" claim is correct.)

Net: across the entire pinned range (>=0.28.0, <0.29), this alters nothing.

Suggestion: the change is harmless and defensible purely as explicit documentation of intent (and a guard against a future pyo3 default change), which is a fine reason to keep a one-liner. But the "Why are the changes needed?" section should be corrected — under pyo3 0.28 this is not a prerequisite for the GIL to stay off; the module already opts in by default. Worth rewording so it doesn't mislead future readers, and so the "step 1 of 2" framing vs. #101 is accurate (the two are actually independent under 0.28).

Good-news corollary: because master already asserts free-threading-safety by default, the substance that matters is whether the extension is genuinely thread-safe. I audited the crate — no static mut, unsendable, RefCell/Cell, Rc, or thread_local; globals use OnceLock<Mutex<…>> / std::sync::Once / AtomicU64; transport uses Arc<Notify> / Arc<AtomicBool> — so it looks structurally sound. The authoritative compile-time enforcement of that (#[pyclass]: Sync) is exactly what #101 adds.

[HyukjinKwon]

Evidence / pointers for the "no-op under pyo3 0.28" finding above.

Version pin (this repo):

• Cargo.toml:87 — pyo3 = { version = "0.28", features = ["extension-module", "abi3-py39"] }
• Cargo.lock → pyo3 0.28.3. Semver "0.28" = >=0.28.0, <0.29, and the opt-out default holds for all of 0.28.x.

The change (this PR): crates/pyspark-rs/src/lib.rs:91, #[pymodule] fn _pyspark(...) — the function-style #[pymodule], so the codegen path is pymodule_function_impl, not the mod-style one.

pyo3 0.28.3 codegen — the attribute defaults to false when omitted:

• pyo3-macros-backend-0.28.3/src/module.rs:452 (inside pymodule_function_impl, defined at module.rs:437):

  let gil_used = options.gil_used.is_some_and(|op| op.value.value);

  Option::is_some_and returns false when the option is None (attribute absent) → gil_used = false. (The mod-style path at module.rs:392 is identical.)
• pyo3-0.28.3/src/impl_/pymodule.rs:209 with_gil_used:

  pub const fn with_gil_used(self, gil_used: bool) -> Self {
      #[cfg(Py_3_13)]                       // line 210
      { self.push(ffi::Py_mod_gil,
          if gil_used { ffi::Py_MOD_GIL_USED }      // 215
          else { ffi::Py_MOD_GIL_NOT_USED }) }      // 217
      #[cfg(not(Py_3_13))]                  // line 222
      { let _ = gil_used; self }            // no slot emitted at all
  }

  ⇒ absent attribute ≡ gil_used = false ⇒ Py_MOD_GIL_NOT_USED on 3.13+ builds — identical to what this PR writes explicitly.

Why the default flipped (authoritative):

• CHANGELOG pyo3-0.28.3/CHANGELOG.md:49, under [0.28.0]: "Support for free-threaded Python is now opt-out rather than opt-in. #5564"
• Migration guide pyo3-0.28.3/guide/src/migration.md:8-16 ("Default to supporting free-threaded Python") — rendered: https://pyo3.rs/v0.28.3/migration#default-to-supporting-free-threaded-python :

  "Modules now automatically allow use on free-threaded Python, unless they directly state they require the GIL with #[pymodule(gil_used = true)]."

Why it's fully inert on the shipped wheels: the wheels are cp39-abi3 (pyproject.toml:73, features = ["pyo3/abi3-py39", ...]). Under the 3.9 limited API the Py_3_13 cfg is unset, so with_gil_used takes the #[cfg(not(Py_3_13))] branch at pymodule.rs:222 and emits no Py_mod_gil slot regardless of the attribute value.

pyo3 source links (v0.28.3 tag):

• module.rs: https://github.com/PyO3/pyo3/blob/v0.28.3/pyo3-macros-backend/src/module.rs
• pymodule.rs: https://github.com/PyO3/pyo3/blob/v0.28.3/src/impl_/pymodule.rs
• migration.md: https://github.com/PyO3/pyo3/blob/v0.28.3/guide/src/migration.md

Reproduce locally: cargo check -p pyspark-rs (passes; the crate builds cp39-abi3 per pyproject.toml). The codegen facts above are read directly from the vendored crate under ~/.cargo/registry/src/*/pyo3-macros-backend-0.28.3/ and .../pyo3-0.28.3/.

Thread-safety audit backing the "structurally sound" note: no matches for static mut, unsendable, RefCell/Cell<, Rc<, or thread_local under crates/pyspark-rs/src/; process-global state is session.rs:201-204 (OnceLock<Mutex<Option<SparkSession>>>), session.rs:213 (std::sync::Once), observation.rs:8 (AtomicU64); transport uses Arc<Notify> / Arc<AtomicBool> (transport.rs:70-71,150-151).

[HyukjinKwon]

Short note on what to fix in the PR description (the code is fine; the rationale is what's stale):

• "Why are the changes needed?" — drop the framing that this is a prerequisite for GIL-off. Under the pinned pyo3 0.28 (Cargo.toml:87), free-threading is opt-out, so the plain #[pymodule] already emits Py_MOD_GIL_NOT_USED. Reframe as: making the free-threading declaration explicit rather than relying on pyo3 0.28's opt-out default (guards against a future default change / downgrade).
• "the extension to run GIL-off" claim — the sentence "CPython re-enables the GIL … when it imports a PyO3 extension module that has not declared gil_used = false" was true for pyo3 0.23–0.27 (opt-in); it no longer holds on 0.28. Either scope it to those versions or remove it.
• "first of the two steps … prerequisite" framing vs [SPARK-59432][CONNECT][INFRA] Build free-threaded (cp3xxt) wheels #101 — under 0.28 the two PRs are independent (the free-threaded wheel in [SPARK-59432][CONNECT][INFRA] Build free-threaded (cp3xxt) wheels #101 runs GIL-off with or without this change), so "step 1 of 2" overstates the dependency. Soften to "companion, explicitness change."

The "Does this PR introduce any user-facing change? No … no-op on GIL-enabled interpreters" part is accurate and can stay.

[james-willis]

Thank you, updated the description. Up to you if we merge or not

[HyukjinKwon]

Merged to master.