[python] Support plan by paimon-rust#8523
Open
XiaoHongbo-Hope wants to merge 13 commits into
Open
Conversation
XiaoHongbo-Hope
marked this pull request as ready for review
July 9, 2026 11:26
XiaoHongbo-Hope
marked this pull request as draft
July 9, 2026 11:31
XiaoHongbo-Hope
marked this pull request as ready for review
July 9, 2026 11:32
JingsongLi
reviewed
Jul 9, 2026
| return [by_name[name] for name in schema.partition_keys] | ||
|
|
||
|
|
||
| def rust_plan(table, table_identifier: str, catalog_options: dict, *, |
Contributor
There was a problem hiding this comment.
This is not called, can this PR not be introduced
XiaoHongbo-Hope
force-pushed
the
rust_plan
branch
from
July 9, 2026 13:16
df093f8 to
022aeac
Compare
XiaoHongbo-Hope
marked this pull request as draft
July 9, 2026 13:48
Contributor
XiaoHongbo-Hope
changed the title
XiaoHongbo-Hope
force-pushed
the
rust_plan
branch
3 times, most recently
from
July 18, 2026 09:04
53fe967 to
0360a46
Compare
XiaoHongbo-Hope
marked this pull request as ready for review
July 18, 2026 09:09
XiaoHongbo-Hope
force-pushed
the
rust_plan
branch
5 times, most recently
from
July 18, 2026 10:34
88ab6be to
9327cfa
Compare
pypaimon's pure-Python, driver-side scan planning is the bottleneck for distributed reads and training. This routes plan() through pypaimon_rust (native, much faster) behind scan.native-plan.enabled -- default OFF, behavior unchanged. Splits are decoded back to pypaimon DataSplits (plan in Rust, read in pypaimon); the predicate is applied pypaimon-side so results match. Works for append and primary-key tables. pypaimon_rust is an optional dependency (extra sql).
XiaoHongbo-Hope
force-pushed
the
rust_plan
branch
from
July 18, 2026 10:43
9327cfa to
ececc92
Compare
Use current_branch() (not the option) so catalog branch tables fall back; fall back to the Python scanner when native returns zero splits (atomic snapshot id under concurrent commits); fall back when the catalog loader has no context().
…et, and old rust Extend the native-plan capability gate so scans the Rust planner cannot carry parity with the Python scanner fall back instead of returning different data: - deletion vectors (Python drops primary-key level-0 files) - data evolution (dedicated split generator with row ranges) - postpone bucket (only_read_real_buckets drops synthetic buckets) Also probe the installed pypaimon-rust for the split-planning API (get_table); when it is missing or too old, fall back to the Python scanner instead of raising. Reuse the scanner's own flags as the source of truth and add routing tests for each case.
native_runtime_available checked only PaimonCatalog.get_table, but native_plan also calls Split.serialize on the planned splits. An intermediate pypaimon-rust with get_table but no Split.serialize would pass the gate and then raise AttributeError mid-plan instead of falling back. Probe both entry points and cover each missing case.
…rs, assert native routed Three fixes for the native-plan path: - A primary-key table whose trimmed PK is empty (PK equals the partition key) is now excluded from native planning. Native marks such splits raw-convertible and the reader skips merge, which would return duplicate/stale rows (the Python path rejects the config outright). - _try_native_plan now falls back to the Python scanner on any native construction/planning failure (e.g. a storage scheme the pinned Rust build does not support, such as viewfs://) instead of failing the scan. - Integration tests now assert native_planned matches expectations so a silent fallback can no longer pass as Python-vs-Python, and the capability probe checks Split.serialize as well as get_table.
…amic bucket; lowercase bools Fixes for further native-plan divergences from the Python scanner: - Rust reloads the on-disk schema, so FileStoreTable.copy overrides that native does not forward (e.g. a removed scan.snapshot-id) made Rust plan a different config than the effective table. copy() now records the cumulative overrides and the gate falls back unless every override is one native forwards. - Dynamic-bucket and cross-partition PK tables (unconfirmed Rust parity) are excluded from native planning. - Boolean catalog options are normalized to lowercase 'true'/'false' so Rust's case-sensitive parser accepts them (str(True) -> 'True' before). - Rename the local native_plan result var so it no longer shadows the module function.
… and scan.version Three more correctness gates: - Any partitioned table now falls back: the Rust bucket_path encodes partition values differently from the pypaimon writer's unescaped str(value) (BOOL True/true, DATE 2024-01-01/19723, TIMESTAMP formats, STRING a/b vs a%2Fb), so native file paths can miss on disk with no read-time fallback. - Fall back when the loaded table schema is stale (Rust reloads the latest on-disk schema; an object altered elsewhere would diverge). - Treat scan.version like the other point-in-time scan keys. Update the partition integration test to assert the fallback instead of silently comparing Python-vs-Python.
_native_plan_supported() runs a remote schema_manager.latest() read; if that (or any other probe step) raised, plan() failed instead of falling back to the Python scanner. Wrap the whole probe so any exception is logged and treated as unsupported -> Python fallback.
… errors Two Bugbot findings: - A scan with a row limit no longer uses native planning. Native has no plan-time limit pushdown (the Python scanner trims splits before read), so under the same limit the returned rows could diverge. - Partition pruning (partition_key_predicate.test) runs inside the fallback try now, so a predicate error falls back to the Python scanner instead of failing the scan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
pypaimon's pure-Python, driver-side scan planning is the bottleneck for
distributed reads and training. This routes
plan()through pypaimon_rust(native, much faster) behind
scan.native-plan.enabled- default OFF, behaviorunchanged. Currently supports plain, unpartitioned, latest-snapshot append and
primary-key tables; unsupported scans fall back to Python planning.
This PR only introduces Rust-based split planning. Filters are still enforced by
the Python reader, while scans with a limit currently fall back to Python
planning. Native predicate and limit pushdown will be added in follow-up PRs.
As a side effect, native splits carry no value stats, so min/max file skipping is
lost and will also be added in a follow-up.
Tests
Unit tests and a native-backend round-trip integration test.
The integration test requires the split-planning API added in pypaimon_rust
0.3.0, which is not published yet. Python 3.11 CI installs a pinned
pypaimon_rust revision and runs the integration test; lanes without that API
skip it.