Skip to content

Core: Merge concurrently added DVs for RowDelta pure delete commits - #17754

Open
raunaqmorarka wants to merge 1 commit into
apache:mainfrom
raunaqmorarka:dv-conflict-merge
Open

raunaqmorarka wants to merge 1 commit into
apache:mainfrom
raunaqmorarka:dv-conflict-merge

Conversation

@raunaqmorarka

@raunaqmorarka raunaqmorarka commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Concurrent DELETEs on V3 tables fail with Found concurrently added DV when they add DVs for the same data file, even though position deletes are commutative.

This change merges the concurrently added DV into the operation's DV instead of failing, when all of the following hold:

  • the RowDelta does not add or remove data files
  • validateNoConflictingDeleteFiles is not enabled
  • the concurrent DV was added by a delete operation (a DV added by an overwrite marks rows that live on in rewritten data files, so merging it would drop this operation's deletes of those rows)

The newest concurrent DV for a data file contains the content of all older ones, since the V3 spec requires a DV to include all positions of the DV it replaces. That DV is removed from the table and its content is merged into the new DV, and pending removals of delete files for that data file are dropped. Merged puffin files from invalidated commit attempts are deleted on retry and on commit failure.

Comment thread core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java Outdated
@sezruby

sezruby commented Sep 11, 2026

Copy link
Copy Markdown

Reviewed this end to end. The concurrent-DV merge is correct — the union preserves all deleted positions given the one-DV-per-file invariant, and restricting merges to DELETE-sourced snapshots (mergeableSnapshotIds) correctly excludes overwrite/replace-added DVs. Test coverage is comprehensive: retry idempotency, a concurrent DV replaced between attempts (with the prior merged file's deletion asserted), cleanup on commit failure, and the fail-closed cases for adds/removes-data-files and overwrite-sourced DVs.

@uros-b the FileIO-resolved cleanup-location comment looks addressed — would you mind another look? This is a clean base to extend row-level concurrency to UPDATE/MERGE.

@sezruby

sezruby commented Sep 14, 2026

Copy link
Copy Markdown

cc @huaxingao — you pointed to this PR on the dev@ thread on row-level concurrency for deletion vectors. I've reviewed it end to end and it looks ready to land: the one open review comment (tracking the FileIO-resolved cleanup location) is addressed, and the test coverage is thorough. The position-granular extension to UPDATE/MERGE I'm working on builds directly on this, so it'd be great if you could do a review pass to help it move forward.

Comment thread core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java Outdated
@raunaqmorarka
raunaqmorarka force-pushed the dv-conflict-merge branch 2 times, most recently from 4d95c47 to 99592bf Compare September 17, 2026 09:11
Comment thread core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
@huaxingao

Copy link
Copy Markdown
Contributor

@amogh-jahagirdar @nastra could you please take a look? This relaxes the "Found concurrently added DV" failure for pure RowDelta deletes and merges the concurrent DV instead. I've reviewed it and it looks correct to me, but a second pair of eyes on the commit-path semantics would be good. Thanks!

Concurrent DELETE commits producing DVs for the same data file always
fail validation even though position deletes are commutative.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants