Skip to content

[SPARK-59305][CONNECT] Track Apache Spark 4.3.0: retarget CI to the RC, re-vendor upstream, and close 4.3.0 client gaps #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HyukjinKwon wants to merge 4 commits into
base: master
Choose a base branch
from
Open

[SPARK-59305][CONNECT] Track Apache Spark 4.3.0: retarget CI to the RC, re-vendor upstream, and close 4.3.0 client gaps #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0fc3fba
[SPARK-59305][CONNECT] Track Apache Spark 4.3.0: retarget CI to the R…
HyukjinKwon Sep 8, 2026
89c8704
Fix function signature parity: typed wrappers + instr/regexp_replace …
HyukjinKwon Sep 8, 2026
edb52af
[SPARK-59305][CONNECT] Fix variant-mutation dispatch: keep string pat…
HyukjinKwon Sep 10, 2026
f056466
[SPARK-59305][CONNECT] Retarget Apache Spark 4.3.0 from RC1 to RC3
HyukjinKwon Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 31 additions & 33 deletions .github/workflows/build_python_connect.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,16 @@
# under the License.
#

name: Python Connect Parity (Spark 4.2.0)
name: Python Connect Parity (Spark 4.3.0-rc3)

# Cancel an in-progress run when a newer commit supersedes it (per workflow + ref).
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

# Mirrors Apache Spark's build_python_connect.yml approach, adapted for the
# Rust-backed client: download the Spark 4.2.0 Connect server, start it, pull the
# official v4.2.0 connect test files, and run them through our client - the gate is
# Rust-backed client: download the Spark 4.3.0-rc3 Connect server, start it, pull the
# official v4.3.0-rc3 connect test files, and run them through our client - the gate is
# "every test passes except the checked-in known-environmental-failures manifest".
# A separate job enforces Rust test coverage across all crates.

Expand All @@ -38,7 +38,7 @@ on:
workflow_dispatch:

env:
SPARK_VERSION: "4.2.0"
SPARK_VERSION: "4.3.0"
RUST_PARITY_LOG_DIR: /tmp/parity-test-logs
SPARK_CONNECT_TESTING_REMOTE: sc://localhost:15002
SPARK_TESTING: "1"
Expand All @@ -47,7 +47,7 @@ env:
jobs:
# ---------------------------------------------------------------------------
# Parity gate: the official connect suite through our client, against a real
# Spark 4.2.0 Connect server (see scripts/run_official_tests.py).
# Spark 4.3.0-rc3 Connect server (see scripts/run_official_tests.py).
# ---------------------------------------------------------------------------
connect-parity:
name: Connect parity (official suite via our client, Python ${{ matrix.python-version }})
Expand Down Expand Up @@ -118,12 +118,12 @@ jobs:
path: ~/spark-dist
key: spark-dist-${{ env.SPARK_VERSION }}-hadoop3

- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server)
- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server, RC)
id: spark-download
run: |
PKG="spark-${SPARK_VERSION}-bin-hadoop3"
if [ ! -d "${HOME}/spark-dist/${PKG}" ]; then
URL="https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${PKG}.tgz"
PKG="spark-${SPARK_VERSION}-bin-hadoop3-connect"
if [ ! -d "${HOME}/spark-dist/spark-4.3.0-bin-hadoop3"* ]; then
URL="https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc3-bin/${PKG}.tgz"
mkdir -p "${HOME}/spark-dist"
echo "Downloading ${URL}"
wget -q -O "${HOME}/spark-dist/${PKG}.tgz" "${URL}"
Expand All @@ -132,19 +132,20 @@ jobs:
else
echo "Using cached Spark distribution."
fi
echo "SPARK_HOME=${HOME}/spark-dist/${PKG}" >> "$GITHUB_OUTPUT"
SPARK_HOME=$(find "${HOME}/spark-dist" -maxdepth 1 -type d -name 'spark-4.3.0-bin-hadoop3*' | head -1)
echo "SPARK_HOME=${SPARK_HOME}" >> "$GITHUB_OUTPUT"

- name: Cache Spark ${{ env.SPARK_VERSION }} source (official test files)
uses: actions/cache@v4
with:
path: ~/spark-source
key: spark-source-${{ env.SPARK_VERSION }}

- name: Clone Apache Spark v${{ env.SPARK_VERSION }} (official test files + reference client)
- name: Clone Apache Spark v4.3.0-rc3 (official test files + reference client)
id: spark-source
run: |
if [ ! -d "${HOME}/spark-source/python" ]; then
git clone --depth 1 --branch "v${SPARK_VERSION}" \
git clone --depth 1 --branch "v4.3.0-rc3" \
https://github.com/apache/spark.git "${HOME}/spark-source"
else
echo "Using cached Spark source."
Expand All @@ -164,7 +165,6 @@ jobs:
# this one server, and a 1g default degrades late in the run (GC pressure),
# which is what made the streaming listener/observation files flake.
"${SPARK_HOME}/sbin/start-connect-server.sh" \
--packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
--driver-memory 3g \
--conf spark.sql.session.localRelationSizeLimit=1073741824 \
> /tmp/connect-server.log 2>&1 &
Expand Down Expand Up @@ -232,11 +232,9 @@ jobs:
sleep 1
done
echo "Starting Connect server in local-cluster mode..."
# Keep --packages: local-cluster launches real executor JVMs that each need the
# spark-connect jar on their classpath, so it is not redundant here.
# Connect jar is bundled in RC distribution, no --packages needed.
"${SPARK_HOME}/sbin/start-connect-server.sh" \
--master "local-cluster[2, 4, 1024]" \
--packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
--driver-memory 3g \
--conf spark.sql.session.localRelationSizeLimit=1073741824 \
> /tmp/connect-server-localcluster.log 2>&1 &
Expand Down Expand Up @@ -282,7 +280,7 @@ jobs:
if-no-files-found: warn

# ---------------------------------------------------------------------------
# Behavioral e2e tests (Rust drop-in against a live server), in their OWN job
# Behavioral e2e tests (Rust drop-in against a live 4.3.0-rc3 server), in their OWN job
# with a separate Connect server so they never share server state with the
# parity gate. The parity gate then starts against a server nothing else has
# touched, which avoids the SPARK-59032 arrow-UDF hang (a UDF that crashes a
Expand Down Expand Up @@ -338,12 +336,12 @@ jobs:
path: ~/spark-dist
key: spark-dist-${{ env.SPARK_VERSION }}-hadoop3

- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server)
- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server, RC)
id: spark-download
run: |
PKG="spark-${SPARK_VERSION}-bin-hadoop3"
if [ ! -d "${HOME}/spark-dist/${PKG}" ]; then
URL="https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${PKG}.tgz"
PKG="spark-${SPARK_VERSION}-bin-hadoop3-connect"
if [ ! -d "${HOME}/spark-dist/spark-4.3.0-bin-hadoop3"* ]; then
URL="https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc3-bin/${PKG}.tgz"
mkdir -p "${HOME}/spark-dist"
echo "Downloading ${URL}"
wget -q -O "${HOME}/spark-dist/${PKG}.tgz" "${URL}"
Expand All @@ -352,7 +350,8 @@ jobs:
else
echo "Using cached Spark distribution."
fi
echo "SPARK_HOME=${HOME}/spark-dist/${PKG}" >> "$GITHUB_OUTPUT"
SPARK_HOME=$(find "${HOME}/spark-dist" -maxdepth 1 -type d -name 'spark-4.3.0-bin-hadoop3*' | head -1)
echo "SPARK_HOME=${SPARK_HOME}" >> "$GITHUB_OUTPUT"

- name: Start Spark Connect server
env:
Expand All @@ -361,7 +360,6 @@ jobs:
run: |
SPARK_HOME="${{ steps.spark-download.outputs.SPARK_HOME }}"
"${SPARK_HOME}/sbin/start-connect-server.sh" \
--packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
--driver-memory 3g \
--conf spark.sql.session.localRelationSizeLimit=1073741824 \
> /tmp/connect-server.log 2>&1 &
Expand All @@ -383,7 +381,7 @@ jobs:
-- --test-threads=1

# ---------------------------------------------------------------------------
# E2E performance benchmark: our client vs reference, same 4.2.0 server.
# E2E performance benchmark: our client vs reference, same 4.3.0-rc3 server.
# ---------------------------------------------------------------------------
connect-benchmark:
name: E2E benchmark vs reference
Expand Down Expand Up @@ -415,7 +413,7 @@ jobs:
run: |
sudo apt-get update && sudo apt-get install -y protobuf-compiler
python -m pip install --upgrade pip
python -m pip install 'pandas>=2.2,<3.0' pyarrow numpy grpcio grpcio-status googleapis-common-protos cloudpickle zstandard py4j "pyspark==${SPARK_VERSION}"
python -m pip install 'pandas>=2.2,<3.0' pyarrow numpy grpcio grpcio-status googleapis-common-protos cloudpickle zstandard py4j "https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc3-bin/pyspark-4.3.0.tar.gz"
- name: Build the Rust extension into the skin
# Build the extension into ./python/pyspark so our client is usable via
# PYTHONPATH. We do NOT install a wheel: our package and the reference
Expand All @@ -433,24 +431,24 @@ jobs:
with:
path: ~/spark-source
key: spark-source-${{ env.SPARK_VERSION }}
- name: Download & start Spark ${{ env.SPARK_VERSION }} server
- name: Download & start Spark ${{ env.SPARK_VERSION }} server (RC)
run: |
PKG="spark-${SPARK_VERSION}-bin-hadoop3"
if [ ! -d "${HOME}/spark-dist/${PKG}" ]; then
PKG="spark-${SPARK_VERSION}-bin-hadoop3-connect"
if [ ! -d "${HOME}/spark-dist/spark-4.3.0-bin-hadoop3"* ]; then
mkdir -p "${HOME}/spark-dist"
wget -q -O "/tmp/${PKG}.tgz" "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${PKG}.tgz"
wget -q -O "/tmp/${PKG}.tgz" "https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc3-bin/${PKG}.tgz"
tar -xzf "/tmp/${PKG}.tgz" -C "${HOME}/spark-dist"
rm -f "/tmp/${PKG}.tgz"
fi
if [ ! -d "${HOME}/spark-source/python" ]; then
git clone --depth 1 --branch "v${SPARK_VERSION}" https://github.com/apache/spark.git "${HOME}/spark-source"
git clone --depth 1 --branch "v4.3.0-rc3" https://github.com/apache/spark.git "${HOME}/spark-source"
fi
"${HOME}/spark-dist/${PKG}/sbin/start-connect-server.sh" \
--packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" > /tmp/server.log 2>&1 &
SPARK_HOME=$(find "${HOME}/spark-dist" -maxdepth 1 -type d -name 'spark-4.3.0-bin-hadoop3*' | head -1)
"${SPARK_HOME}/sbin/start-connect-server.sh" > /tmp/server.log 2>&1 &
for i in $(seq 1 90); do (echo > /dev/tcp/localhost/15002) 2>/dev/null && break; sleep 1; done
- name: Benchmark reference client
run: |
# Reference uses the pip-installed pyspark 4.2.0 (no ./python on the path).
# Reference uses the pip-installed pyspark 4.3.0-rc3 (no ./python on the path).
python scripts/benchmark_e2e.py --isolate --label reference --out /tmp/bench_reference.json
- name: Benchmark our client
run: |
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ on:

env:
CARGO_TERM_COLOR: always
SPARK_VERSION: "4.2.0"
SPARK_VERSION: "4.3.0"

jobs:
coverage:
Expand Down Expand Up @@ -98,12 +98,12 @@ jobs:
path: ~/spark-dist
key: spark-dist-${{ env.SPARK_VERSION }}-hadoop3

- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server)
- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server, RC)
id: spark-download
run: |
PKG="spark-${SPARK_VERSION}-bin-hadoop3"
if [ ! -d "${HOME}/spark-dist/${PKG}" ]; then
URL="https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${PKG}.tgz"
PKG="spark-${SPARK_VERSION}-bin-hadoop3-connect"
if [ ! -d "${HOME}/spark-dist/spark-4.3.0-bin-hadoop3"* ]; then
URL="https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc3-bin/${PKG}.tgz"
mkdir -p "${HOME}/spark-dist"
echo "Downloading ${URL}"
wget -q -O "${HOME}/spark-dist/${PKG}.tgz" "${URL}"
Expand All @@ -112,7 +112,8 @@ jobs:
else
echo "Using cached Spark distribution."
fi
echo "SPARK_HOME=${HOME}/spark-dist/${PKG}" >> "$GITHUB_OUTPUT"
SPARK_HOME=$(find "${HOME}/spark-dist" -maxdepth 1 -type d -name 'spark-4.3.0-bin-hadoop3*' | head -1)
echo "SPARK_HOME=${SPARK_HOME}" >> "$GITHUB_OUTPUT"

- name: Start Spark Connect server
env:
Expand All @@ -124,7 +125,6 @@ jobs:
SPARK_HOME="${{ steps.spark-download.outputs.SPARK_HOME }}"
echo "Starting Connect server on sc://localhost:15002"
"${SPARK_HOME}/sbin/start-connect-server.sh" \
--packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
--driver-memory 3g \
--conf spark.sql.session.localRelationSizeLimit=1073741824 \
> /tmp/connect-server.log 2>&1 &
Expand Down
22 changes: 11 additions & 11 deletions .github/workflows/pandas-parity.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
# full visibility into pass/fail status. If this becomes a critical blocker, add
# continue-on-error: true back in; currently it provides honest signal on pandas parity.

name: Pandas Connect Parity (Spark 4.2.0)
name: Pandas Connect Parity (Spark 4.3.0-rc3)

# Cancel an in-progress run when a newer commit supersedes it (per workflow + ref).
concurrency:
Expand All @@ -38,15 +38,15 @@ on:
workflow_dispatch:

env:
SPARK_VERSION: "4.2.0"
SPARK_VERSION: "4.3.0"
SPARK_CONNECT_TESTING_REMOTE: sc://localhost:15002
SPARK_TESTING: "1"
SPARK_SKIP_CONNECT_COMPAT_TESTS: "1" # force remote-only (pure connect, no local JVM)

jobs:
# ---------------------------------------------------------------------------
# Pandas-on-Spark Connect parity: run the official pandas tests through our
# Rust client against a real Spark 4.2.0 Connect server. This is a separate,
# Rust client against a real Spark 4.3.0-rc3 Connect server. This is a separate,
# long-running job (30-60 min) to avoid blocking the main parity gate.
# ---------------------------------------------------------------------------
pandas-parity:
Expand Down Expand Up @@ -122,12 +122,12 @@ jobs:
path: ~/spark-dist
key: spark-dist-${{ env.SPARK_VERSION }}-hadoop3

- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server)
- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server, RC)
id: spark-download
run: |
PKG="spark-${SPARK_VERSION}-bin-hadoop3"
if [ ! -d "${HOME}/spark-dist/${PKG}" ]; then
URL="https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${PKG}.tgz"
PKG="spark-${SPARK_VERSION}-bin-hadoop3-connect"
if [ ! -d "${HOME}/spark-dist/spark-4.3.0-bin-hadoop3"* ]; then
URL="https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc3-bin/${PKG}.tgz"
mkdir -p "${HOME}/spark-dist"
echo "Downloading ${URL}"
wget -q -O "${HOME}/spark-dist/${PKG}.tgz" "${URL}"
Expand All @@ -136,19 +136,20 @@ jobs:
else
echo "Using cached Spark distribution."
fi
echo "SPARK_HOME=${HOME}/spark-dist/${PKG}" >> "$GITHUB_OUTPUT"
SPARK_HOME=$(find "${HOME}/spark-dist" -maxdepth 1 -type d -name 'spark-4.3.0-bin-hadoop3*' | head -1)
echo "SPARK_HOME=${SPARK_HOME}" >> "$GITHUB_OUTPUT"

- name: Cache Spark ${{ env.SPARK_VERSION }} source (official test files)
uses: actions/cache@v4
with:
path: ~/spark-source
key: spark-source-${{ env.SPARK_VERSION }}

- name: Clone Apache Spark v${{ env.SPARK_VERSION }} (official test files + reference client)
- name: Clone Apache Spark v4.3.0-rc3 (official test files + reference client)
id: spark-source
run: |
if [ ! -d "${HOME}/spark-source/python" ]; then
git clone --depth 1 --branch "v${SPARK_VERSION}" \
git clone --depth 1 --branch "v4.3.0-rc3" \
https://github.com/apache/spark.git "${HOME}/spark-source"
else
echo "Using cached Spark source."
Expand Down Expand Up @@ -177,7 +178,6 @@ jobs:
# how Apache's PYTHON connect tests run (the runner's UTC); the pom.xml America/Los_Angeles
# is for the JVM/Scala maven-surefire tests, not the separately-launched python connect server.
"${SPARK_HOME}/sbin/start-connect-server.sh" \
--packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
--driver-memory 3g \
--conf spark.sql.session.localRelationSizeLimit=1073741824 \
--conf spark.sql.session.timeZone=UTC \
Expand Down
22 changes: 11 additions & 11 deletions .github/workflows/pipelines-parity.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
# 2. The full official upstream pipeline connect suite (incl. server-backed tests like
# test_spark_connect.py) run through OUR Rust client via the transport plugin.

name: Pipelines Connect Parity (Spark 4.2.0)
name: Pipelines Connect Parity (Spark 4.3.0-rc3)

# Cancel an in-progress run when a newer commit supersedes it (per workflow + ref).
concurrency:
Expand All @@ -39,15 +39,15 @@ on:
workflow_dispatch:

env:
SPARK_VERSION: "4.2.0"
SPARK_VERSION: "4.3.0"
SPARK_CONNECT_TESTING_REMOTE: sc://localhost:15002
SPARK_TESTING: "1"
SPARK_SKIP_CONNECT_COMPAT_TESTS: "1" # force remote-only (pure connect, no local JVM)

jobs:
# ---------------------------------------------------------------------------
# Pipelines Connect parity: run the official pipeline connect tests through our
# Rust client against a real Spark 4.2.0 Connect server. Two blocking groups: pure-Python
# Rust client against a real Spark 4.3.0-rc3 Connect server. Two blocking groups: pure-Python
# pipeline tests and the full server-backed official suite.
# ---------------------------------------------------------------------------
pipelines-parity:
Expand Down Expand Up @@ -117,12 +117,12 @@ jobs:
path: ~/spark-dist
key: spark-dist-${{ env.SPARK_VERSION }}-hadoop3

- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server)
- name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server, RC)
id: spark-download
run: |
PKG="spark-${SPARK_VERSION}-bin-hadoop3"
if [ ! -d "${HOME}/spark-dist/${PKG}" ]; then
URL="https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${PKG}.tgz"
PKG="spark-${SPARK_VERSION}-bin-hadoop3-connect"
if [ ! -d "${HOME}/spark-dist/spark-4.3.0-bin-hadoop3"* ]; then
URL="https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc3-bin/${PKG}.tgz"
mkdir -p "${HOME}/spark-dist"
echo "Downloading ${URL}"
wget -q -O "${HOME}/spark-dist/${PKG}.tgz" "${URL}"
Expand All @@ -131,19 +131,20 @@ jobs:
else
echo "Using cached Spark distribution."
fi
echo "SPARK_HOME=${HOME}/spark-dist/${PKG}" >> "$GITHUB_OUTPUT"
SPARK_HOME=$(find "${HOME}/spark-dist" -maxdepth 1 -type d -name 'spark-4.3.0-bin-hadoop3*' | head -1)
echo "SPARK_HOME=${SPARK_HOME}" >> "$GITHUB_OUTPUT"

- name: Cache Spark ${{ env.SPARK_VERSION }} source (official test files)
uses: actions/cache@v4
with:
path: ~/spark-source
key: spark-source-${{ env.SPARK_VERSION }}

- name: Clone Apache Spark v${{ env.SPARK_VERSION }} (official test files + reference client)
- name: Clone Apache Spark v4.3.0-rc3 (official test files + reference client)
id: spark-source
run: |
if [ ! -d "${HOME}/spark-source/python" ]; then
git clone --depth 1 --branch "v${SPARK_VERSION}" \
git clone --depth 1 --branch "v4.3.0-rc3" \
https://github.com/apache/spark.git "${HOME}/spark-source"
else
echo "Using cached Spark source."
Expand Down Expand Up @@ -178,7 +179,6 @@ jobs:
echo "Starting Connect server on ${SPARK_CONNECT_TESTING_REMOTE}"
# Give the driver extra heap for the pipeline tests which may be I/O heavy.
"${SPARK_HOME}/sbin/start-connect-server.sh" \
--packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
--driver-memory 3g \
--conf spark.sql.session.localRelationSizeLimit=1073741824 \
> /tmp/connect-server.log 2>&1 &
Expand Down
Loading
Loading