[core][spark] Support managed Format Table partition operations#8713
Closed
sundapeng wants to merge 1 commit into
Closed
[core][spark] Support managed Format Table partition operations#8713sundapeng wants to merge 1 commit into
sundapeng wants to merge 1 commit into
Conversation
sundapeng
changed the title
sundapeng
force-pushed
the
format-table-partition-operations
branch
from
July 18, 2026 06:24
f2a9b50 to
df367a3
Compare
sundapeng
changed the title
sundapeng
changed the title
This was referenced Jul 19, 2026
Member
Author
|
To make this easier to review, I have split it into three stacked PRs with identical content (the tip of the stack is byte-identical to this branch):
Converting this PR to draft and keeping it open as the full-feature view; review can start with #8728. I will close this one once the split PRs are merged. |
sundapeng
marked this pull request as draft
July 19, 2026 07:46
sundapeng
force-pushed
the
format-table-partition-operations
branch
5 times, most recently
from
July 20, 2026 15:51
f84a6d4 to
16e10a2
Compare
A partitioned Format Table finds its partitions by walking the table directory. That is how Hive works and it stays the default, but it means a directory appearing under the table is a partition, whether or not anyone meant it to be, and that listing a large table costs a full traversal of object storage. With format-table.partition-source=rest on an internal Format Table in a REST catalog, the catalog becomes the source of truth instead: a scan reads the partitions it has registered, and a batch write registers the ones it wrote. A directory nobody registered is not part of the table, and an empty catalog means an empty table — reads never quietly fall back to the filesystem, since that would defeat the point. A registered partition whose directory is missing reads as empty, matching Hive, where ADD PARTITION before the first insert is ordinary. Spark gains the partition DDL this makes possible: ADD and DROP PARTITION, and MSCK REPAIR TABLE to reconcile a table whose directories and catalog have drifted apart — after a backfill by an older writer, say. DROP unregisters before deleting, so a failed deletion leaves nothing readable behind, and a repair never deletes data at all. SHOW PARTITIONS needs no special support: it already asks the scan. The option names one source and one only, so asking for it where it cannot be served — outside a REST catalog, on an external table, under the engine implementation — fails rather than handing back a table reading its partitions from somewhere it did not ask for. Whether a table's partitions come from the catalog is then answered in one place, by whether loading it produced a partition manager, so the engines cannot disagree about it.
sundapeng
force-pushed
the
format-table-partition-operations
branch
from
July 20, 2026 16:55
16e10a2 to
3ae0818
Compare
This was referenced Jul 20, 2026
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes a REST catalog the source of truth for an internal Format Table's
partitions, opt in with
TBLPROPERTIES ('metastore.partitioned-table' = 'true').A scan reads the partitions registered in the catalog and a batch write
registers the ones it wrote; an unregistered directory is not part of the table,
and reads never fall back to the filesystem. Spark gets
ADD/DROP PARTITIONand
MSCK REPAIR TABLEon top.This is the full view. It is split for review into:
Supersedes #8728, #8729, #8730, which reviewed an earlier shape.
Notes on the final shape
metastore.partitioned-table; a REST catalog already gates thepartition API on it. A table is catalog-managed only when loading it produced
a partition manager — an internal Format Table, in a REST catalog, under the
paimon implementation — so nothing a Paimon table or a catalog default sets
changes a Format Table by accident. Requesting it elsewhere fails the load
rather than degrading silently.
MSCK REPAIR TABLEis the reconcile entry point.Testing
ALTER TABLE ... ADD PARTITIONend to end and MSCK with fault injection