Repository navigation
[SPARK-59305][CONNECT] Track Apache Spark 4.3.0: retarget CI to the RC, re-vendor upstream, and close 4.3.0 client gaps - #99
Open
HyukjinKwon wants to merge 3 commits into
Conversation
…C, re-vendor upstream, and close the API + ML surface
Make master track Apache Spark 4.3.0 (currently RC1).
Retargeting:
- Version strings -> 4.3.0 (channel.rs SPARK_VERSION, version.py, PROTO_VERSION).
- CI (5 workflows) -> the v4.3.0-rc1 staging: connect-bundled tarball (no --packages),
official tests at the v4.3.0-rc1 tag, reference pyspark from the staging sdist.
- Re-vendor all changed upstream Python to v4.3.0-rc1 + the new pyspark.messages package;
bump the drift SPARK_TAG.
- Docs/examples -> 4.3.0.
API surface (verified by a runtime dir() + __all__ sweep vs pyspark-client 4.3.0-rc1,
recorded in dev/parity/4.3.0-introspection-diff.md; now 0 gaps across every module):
- Functions: collect_union, to_base32, from_base32, variant_strip_nulls, variant_delete;
re-export PandasUDFType, ArrowUDFType, UserDefinedFunction/TableFunction, UDTF helpers.
- Types: Geometry/GeometryType/Geography/GeographyType/VariantVal in __all__, plus the
datatype class constants (DayTimeIntervalType.*, YearMonthIntervalType.*,
DateType.EPOCH_ORDINAL, Geometry/Geography/SpatialType CRS/SRID/ALG) as PyO3 #[classattr]s
AND native spark_connect::{DayTimeIntervalType, GeometryType, ...} constants.
- DataFrame.zip (PyO3), SparkSession.Hook + Builder._registerHook, MergeIntoWriter nested
classes, SparkSession.Builder keyword-name fixes.
- ML: vendor upstream ml/connect/base.py + ml/functions.py; export the reference
ml.connect surface (Estimator/Transformer/Evaluator/Model/feature/evaluation/Pipeline/
PipelineModel/tuning). Add __version__ to pyspark.__all__.
Tests/coverage: un-skip the arrow-UDF parity test (passes on 4.3.0); omit the vendored
pyspark.messages package from .coveragerc; add offline tests for the new API. Python
coverage 76% -> 91%, Rust 90.3%. Build (-D warnings), fmt, all tests compile; the official
4.3.0 connect/pandas/pipelines suites pass on CI against the RC.
Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
force-pushed
the
spark-4.3.0-support
branch
from
September 8, 2026 03:50
bb30d56 to
09dd20a
Compare
…params Rigorous signature diff vs pyspark-client 4.3.0-rc1 (896 signatures) found 18 new functions exposed as generic *args (keyword calls broken) and 2 with missing params. - Regenerate functions_generated.py with the 4.3.0 reference (gen_fn_dispatch.py --pyspark-src) so all wrappers carry typed parameter names/order (jaro_winkler_similarity, hmac, vector_*, timestamp_nanos, variant_*, ...). Pure signature change - the runtime path is unchanged (_dispatch -> pyfunc_call_function forwards all args). - instr (start/occurrence) and regexp_replace (position): forward all args through call_function in the call_builtin path instead of truncating to the first 2/3. Signature sweep now: 0 *args wrappers, 0 missing-params. The remaining 132 optional-arg defaults use the =_UNSET sentinel (forward-only-if-supplied) rather than =None - a deliberate, functionally-equivalent convention (=None would append unsupplied args to the dispatch). Golden + 282 offline tests pass. Co-authored-by: Isaac <no-reply@databricks.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…h args literal
variant_delete and its sibling variant-mutation wrappers (variant_insert,
variant_set, variant_array_append, and their try_ forms) were absent from
_FN_COL_ARGS, so _dispatch ran every positional arg through _to_col. A
bare-string path like "$.b" became an unresolved attribute col("$.b") instead
of a string literal, which fails analysis on the server.
Add the correct column-position mappings, matching how variant_get already lits
string paths: only the value column(s) are coerced, the path stays a literal
(and create_if_missing stays a bool literal for variant_set/try_variant_set).
Add an offline regression test that captures the exact positional args handed to
the core and asserts literal paths/flags stay raw while value columns are coerced
-- the prior test only checked that a Column was built, which could not catch this.
json_tuple has the same generated-wrapper shape but is already correct via a
hand-written override that lits its fields, so it is left unchanged.
Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Make
mastertrack Apache Spark 4.3.0 (currently at RC1). One PR:channel.rsSPARK_VERSION,version.py,PROTO_VERSION.txt).archive.apache.org/ PyPI, so CI downloads the connect-bundledspark-4.3.0-bin-hadoop3-connect.tgzfromdist/dev/spark/v4.3.0-rc1-bin/and starts the server without--packages; clones official tests atv4.3.0-rc1; installs the reference pyspark from the staging sdist;SPARK_HOMEresolved by glob.v4.3.0-rc1(pandas, sql/worker, errors/, logger, pipelines/, ml/, mllib/, sql/pandas/*, profiler.py, …) — byte-identical, drift check exits 0 — plus the newpyspark.messagespackage;SPARK_TAGbumped.collect_union,to_base32,from_base32,variant_strip_nulls(dedicated bool-arg binding).DataFrame.zipexposed in PyO3.SparkSession.Hook(new 4.3.0 nested class) +Builder._registerHookexposed; Builder keyword names fixed (appName(name)/master(master)/remote(location='sc://localhost')).Why are the changes needed?
Spark 4.3.0 adds client APIs and refactors vendored modules; master carried the 4.3.0 proto but still targeted 4.2.0 everywhere. This retargets the whole surface and closes the API gaps.
Does this PR introduce any user-facing change?
Yes — targets Spark 4.3.0; adds
collect_union/to_base32/from_base32/variant_strip_nulls,DataFrame.zip,SparkSession.Hook; ships vendoredpyspark.messages.How was this patch tested?
pyspark-client 4.3.0-rc1(runtimedir(), method + signature) → 0 method-name gaps across all 12 connect classes (DataFrame, Column, SparkSession(+Builder), Reader/Writer/WriterV2, GroupedData, Catalog, Window, WindowSpec, Observation); datatype classes match too. Recorded indev/parity/4.3.0-introspection-diff.md. Justified non-gaps:Column.to_plan,SparkSession.client(native transport).cargo fmt --all --checkclean;cargo test --no-runforapache-spark-connect+pyspark-rscompiles clean; vendored re-syncdiff -rbyte-identical tov4.3.0-rc1.This pull request and its description were written by Isaac.