Repository navigation
[SPARK-59432][CONNECT][INFRA] Build free-threaded (cp3xxt) wheels - #101
james-willis wants to merge 8 commits into
Conversation
abi3 and the free-threaded (PEP 703) ABI are mutually exclusive, so a free-threaded interpreter cannot load the current cp39-abi3 wheels. Add a Linux cp313t/cp314t wheel build leg to the release workflow (alongside the abi3 leg), and add a PR-time workflow that compiles pyspark-rs against a free-threaded interpreter -- which enforces that every #[pyclass] is Sync -- and imports the built extension. The GIL is actually disabled once the module declares gil_used = false (the companion change); this leg makes a loadable free-threaded wheel exist.
|
@HyukjinKwon I thought this was a logical way to structure these CI steps but lmk if there is a different approach youd prefer. |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
cc @gaogaotiantian do you mind taking a look whenever you find some time? |
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
Reviewed against the pinned pyo3 0.28.3, building and inspecting locally (cargo fmt --all --check clean; cargo check -p pyspark-rs green on the companion #100 branch). This is the real substance of SPARK-59432 and should proceed — freethreaded.yml is the only CI job that compiles pyspark-rs against Py_GIL_DISABLED, which is what actually enforces #[pyclass]: Sync. Two things to address before merge, plus one minor and one doc fix — details inline:
- Blocker to verify: the
release.ymlfree-threaded--zigleg may not findziglangafter the interpreter switch — please attach a green release dry-run, or reorder the install (inline onrelease.yml). - Doc/accuracy: the "GIL stays on until
gil_used = false" premise is outdated under pyo3 0.28 (opt-out default) — affects both the description and thefreethreaded.ymlcomment (inline). - Minor: confirm the
python3.13t/python3.14texecutable names resolve in the dry-run (inline).
Evidence for the pyo3 0.28 opt-out default (module.rs:452 ⇒ default false ⇒ Py_MOD_GIL_NOT_USED; CHANGELOG #5564; migration guide https://pyo3.rs/v0.28.3/migration#default-to-supporting-free-threaded-python) is in the companion review on #100.
| # report whether the GIL is disabled. The GIL only actually stays off once | ||
| # the module declares gil_used = false (SPARK-59432); until then CPython | ||
| # re-enables it on import, which is expected and not a failure here. |
There was a problem hiding this comment.
This comment is inaccurate under the pinned pyo3 0.28. Free-threading is opt-out there, so the plain #[pymodule] already emits Py_MOD_GIL_NOT_USED — the GIL does not get re-enabled on import, with or without #100. So line 54 will print gil_enabled: False on master too, not True. The step doesn't assert on it, so no failure, but the comment (and the "until #100" framing) is misleading. Suggest rewording to: "reports whether the GIL is disabled; under pyo3 0.28 modules are free-threading by default, so this is expected to be False."
Ref: pyo3-macros-backend-0.28.3/src/module.rs:452, pyo3-0.28.3/CHANGELOG.md:49 (#5564), migration guide https://pyo3.rs/v0.28.3/migration#default-to-supporting-free-threaded-python
|
Is the current client use |
Yes it does. That's part of my motivation: the rust-based client is grpcio-free so it can safely be run with GIL off. |
…rect pyo3 0.28 free-threading note Install maturin and ziglang into the free-threaded interpreter that setup-python makes the default before building, so maturin --zig can locate zig via python -m ziglang (the abi3 step installs them into 3.11, which is no longer the default). Also correct the freethreaded.yml comment: under the pinned pyo3 0.28 a plain #[pymodule] is free-threading by default (Py_MOD_GIL_NOT_USED), so the GIL stays off without any gil_used declaration.
|
If we want to ship free-threaded wheel, we need to test it. We can't just do a release without any testing. |
Under pyo3 0.28 the abi3 and free-threaded ABIs are mutually exclusive: building the extension with pyo3/abi3-py39 against a free-threaded interpreter fails to compile (no fastcall pymethods). Move abi3-py39 out of the pyproject [tool.maturin] features and pass it only on the stable-ABI leg's maturin --features, so the free-threaded cp3xxt leg builds without it.
… for now The aarch64 free-threaded leg cross-compiles with the host x86_64 3.13t interpreter and pyo3 0.28 mis-detects the target's free-threaded ABI, failing to compile (no fastcall pymethods). Build the free-threaded (cp313t/cp314t) wheel on x86_64 Linux only; the aarch64 free-threaded cross-compile is a follow-up.
Fair. let me move this to draft for a while. My use case to be able to use this is still being spun up. Do you have any specific ideas about what you want tested in a free threaded environment? @gaogaotiantian |
|
In https://github.com/apache/spark-connect-rust/blob/master/.github/workflows/build_python_connect.yml there are tests to run the client against the server. If we want to build free threaded wheels, we need to run the same tests under 3.14t. |
…readed Python 3.14t Add a 3.14t leg to the connect parity job so the official connect suite runs through the Rust client on a free-threaded interpreter. The suite drives the reference client, which imports grpcio; grpcio does not declare free-threading support and would re-enable the GIL on import, so the leg forces the GIL off with PYTHON_GIL=0 and asserts it stays off.
…uite Scope PYTHON_GIL=0 to the parity-suite and Connect-server steps instead of the whole job, so the free-threaded check runs without it and verifies that the extension and test dependencies leave the GIL disabled on their own.
…fest On free-threaded CPython pip installs protobuf's pure-Python backend (protobuf ships upb only as an abi3 wheel), which overflows the recursion limit serializing the 400-deep plan in test_recursion_handling_for_plan_logging. The reference client hits the same overflow, so list it in a manifest applied only on the free-threaded leg; run_official_tests.py now accepts --manifest more than once.
…est default Extra --manifest values append to the default list, so the free-threaded leg only passes its own manifest.
This branch has not been deployed
3 participants
What changes were proposed in this pull request?
Add a free-threaded (no-GIL, PEP 703) wheel build for
pyspark-client-rust.release.yml: add a Linuxcp313t/cp314twheel build leg alongside the existing abi3 leg. abi3 and the free-threaded ABI are mutually exclusive, so these are separate, version-specific wheels; both land in the samedist/and are uploaded together, and pip selects the right one per interpreter. macOS/Windows free-threaded legs can follow.freethreaded.yml(runs on push / pull_request): compilespyspark-rsagainst a free-threaded interpreter and imports the built extension.Why are the changes needed?
The published wheels are
cp39-abi3. abi3 is incompatible with the free-threaded ABI, so no wheel is selectable on a3.13t/3.14tinterpreter — there is currently no way topip install pyspark-client-ruston free-threaded CPython at all. This PR builds a loadable free-threaded wheel.Under the pinned pyo3 0.28, a plain
#[pymodule]is free-threading-compatible by default (it emitsPy_MOD_GIL_NOT_USED), so no source change is needed for the module to keep the GIL disabled — the only missing piece is the wheel.freethreaded.ymlalso fills a real CI gap:pyspark-rsis excluded from the workspacedefault-members, so no existing job compiles it against a free-threaded interpreter — and compiling againstPy_GIL_DISABLEDis exactly what enforces that every#[pyclass]isSync. A green build in this workflow is that proof.Does this PR introduce any user-facing change?
Yes, additive: free-threaded (
cp313t/cp314t) Linux wheels become available on release, so users on free-threaded CPython can installpyspark-client-rust. No change for existing (abi3) users.How was this patch tested?
build_python_connect.yml: the connect parity job now also runs on free-threaded Python 3.14t, alongside 3.11. That leg runs the full official connect suite through the Rust client against a Spark 4.2.0 Connect server, the same suite and pass criteria as the 3.11 leg.PYTHON_GIL=0to keep the GIL off.scripts/parity_known_failures_freethreaded.txt. protobuf ships its C backend only as an abi3 wheel, which free-threaded CPython can't load, so pip installs the pure-Python backend. That backend overflows the recursion limit serializing the 400-deep plan intest_recursion_handling_for_plan_logging. The reference client serializes requests the same way, so it hits this too.freethreaded.ymlbuildspyspark-rsagainst a free-threaded 3.13t interpreter (proving theSyncrequirement holds) and imports the resulting extension.cp313t/cp314twheel builds across the Linux targets run inrelease.yml's dry-run path.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic)