Core: Partition prune delete manifests in ManifestFilterManager by grantatspothero · Pull Request #18154 · apache/iceberg

grantatspothero · 2026-09-17T14:54:28Z

This PR: #13222 added support for tracking removed data files during MergingSnapshotProducer so that orphan DV could be removed.

The purpose of this PR is to increase performance of delete manifest filtering/rewriting through partition pruning delete manifests. See below commit breakdown:

3 commits in this PR:

Fixes bugs in the existing canContainDroppedFiles implementation for dangling DV detection.
Adds more parameters to the existing RewriteDataFilesBenchmark benchmark. I tried to make the benchmark more realistic (injecting object-storage like latency, using a partitioned table, etc).
Adds partition pruning into the delete manifest path. This mirrors the data manifest path.

JMH results:
commit (2)

Benchmark                                   (manifestTargetSizeBytes)  (metadataFileLatencyMs)  (numFiles)  (numPartitions)  (percentDataFilesRewritten)  Mode  Cnt   Score    Error  Units
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            1    ss    5   0.240 ±  0.038   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            5    ss    5   0.385 ±  0.192   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                           50    ss    5   1.071 ±  0.743   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                          100    ss    5   2.047 ±  1.011   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            1    ss    5   0.260 ±  0.092   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            5    ss    5   0.364 ±  0.103   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                           50    ss    5   1.106 ±  0.568   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                          100    ss    5   1.956 ±  0.658   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            1    ss    5   0.298 ±  0.143   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            5    ss    5   0.514 ±  0.184   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                           50    ss    5   1.933 ±  0.873   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                          100    ss    5   3.821 ±  3.269   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            1    ss    5   0.302 ±  0.143   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            5    ss    5   0.540 ±  0.276   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                           50    ss    5   1.905 ±  0.947   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                          100    ss    5   3.650 ±  2.995   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            1    ss    5   0.559 ±  0.203   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            5    ss    5   1.326 ±  1.889   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                           50    ss    5  10.509 ±  5.856   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                          100    ss    5  27.937 ± 33.988   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            1    ss    5   0.729 ±  0.422   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            5    ss    5   1.607 ±  0.531   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                           50    ss    5  12.706 ± 20.017   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                          100    ss    5  29.234 ± 33.748   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            1    ss    5   0.787 ±  0.130   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            5    ss    5   1.119 ±  0.119   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                           50    ss    5   2.921 ±  0.206   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                          100    ss    5   4.449 ±  0.259   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            1    ss    5   0.762 ±  0.064   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            5    ss    5   1.114 ±  0.274   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                           50    ss    5   2.922 ±  0.181   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                          100    ss    5   4.546 ±  0.578   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            1    ss    5   0.996 ±  0.079   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            5    ss    5   1.575 ±  0.107   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                           50    ss    5   4.382 ±  0.248   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                          100    ss    5   7.181 ±  1.516   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            1    ss    5   1.000 ±  0.085   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            5    ss    5   1.584 ±  0.097   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                           50    ss    5   4.396 ±  0.251   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                          100    ss    5   7.647 ±  2.419   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            1    ss    5   2.791 ±  0.072   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            5    ss    5   4.353 ±  0.516   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                           50    ss    5  18.207 ±  3.437   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                          100    ss    5  37.054 ± 21.876   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   2.869 ±  0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   4.403 ±  0.355   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                           50    ss    5  22.089 ±  9.031   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                          100    ss    5  40.231 ± 45.683   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            1    ss    5   0.560 ±  0.263   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            5    ss    5   0.524 ±  0.158   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                           50    ss    5   1.979 ±  0.469   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                          100    ss    5   2.146 ±  0.093   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            1    ss    5   0.475 ±  0.059   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            5    ss    5   0.511 ±  0.033   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                           50    ss    5   1.950 ±  0.083   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                          100    ss    5   2.269 ±  0.300   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            1    ss    5   0.546 ±  0.345   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            5    ss    5   0.553 ±  0.193   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                           50    ss    5   2.216 ±  0.354   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                          100    ss    5   2.561 ±  0.239   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            1    ss    5   0.478 ±  0.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            5    ss    5   0.637 ±  0.911   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                           50    ss    5   2.456 ±  1.072   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                          100    ss    5   2.931 ±  1.298   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            1    ss    5   1.139 ±  0.323   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            5    ss    5   1.237 ±  0.494   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                           50    ss    5   5.043 ±  1.615   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                          100    ss    5   7.641 ±  4.469   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            1    ss    5   1.243 ±  0.533   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            5    ss    5   1.537 ±  0.607   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                           50    ss    5   6.671 ±  2.818   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                          100    ss    5   6.675 ±  1.322   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            1    ss    5   0.968 ±  0.101   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            5    ss    5   1.226 ±  0.536   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                           50    ss    5   2.720 ±  0.199   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                          100    ss    5   2.920 ±  0.263   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            1    ss    5   0.941 ±  0.036   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            5    ss    5   0.983 ±  0.125   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                           50    ss    5   2.734 ±  0.118   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                          100    ss    5   3.022 ±  0.440   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            1    ss    5   1.091 ±  0.341   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            5    ss    5   1.032 ±  0.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                           50    ss    5   3.110 ±  0.353   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                          100    ss    5   3.635 ±  0.315   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            1    ss    5   0.993 ±  0.096   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            5    ss    5   1.034 ±  0.036   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                           50    ss    5   3.090 ±  0.289   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                          100    ss    5   3.830 ±  0.522   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            1    ss    5   1.624 ±  0.088   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            5    ss    5   1.868 ±  0.790   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                           50    ss    5   6.796 ±  2.851   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                          100    ss    5   8.207 ±  2.527   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            1    ss    5   1.708 ±  0.576   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            5    ss    5   2.426 ±  1.983   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                           50    ss    5   7.736 ±  2.159   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                          100    ss    5   7.131 ±  0.931   s/op

commit (3):

RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            1    ss    5   0.247 ± 0.070   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            5    ss    5   0.379 ± 0.181   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                           50    ss    5   1.026 ± 0.132   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                          100    ss    5   1.924 ± 0.259   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            1    ss    5   0.220 ± 0.047   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            5    ss    5   0.343 ± 0.149   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                           50    ss    5   1.023 ± 0.168   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                          100    ss    5   1.862 ± 0.259   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            1    ss    5   0.282 ± 0.144   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            5    ss    5   0.473 ± 0.224   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                           50    ss    5   1.787 ± 0.109   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                          100    ss    5   3.267 ± 0.231   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            1    ss    5   0.279 ± 0.126   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            5    ss    5   0.511 ± 0.335   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                           50    ss    5   1.830 ± 0.364   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                          100    ss    5   3.235 ± 0.116   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            1    ss    5   0.613 ± 0.162   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            5    ss    5   1.217 ± 0.557   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                           50    ss    5   9.368 ± 1.900   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                          100    ss    5  24.045 ± 7.809   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            1    ss    5   0.861 ± 0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            5    ss    5   1.388 ± 2.192   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                           50    ss    5  12.689 ± 1.326   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                          100    ss    5  24.632 ± 4.927   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            1    ss    5   0.799 ± 0.072   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            5    ss    5   1.080 ± 0.102   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                           50    ss    5   3.106 ± 0.506   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                          100    ss    5   4.482 ± 1.094   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            1    ss    5   0.754 ± 0.064   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            5    ss    5   1.026 ± 0.037   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                           50    ss    5   2.921 ± 0.174   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                          100    ss    5   4.527 ± 1.071   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            1    ss    5   0.993 ± 0.020   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            5    ss    5   1.597 ± 0.084   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                           50    ss    5   4.435 ± 0.549   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                          100    ss    5   6.963 ± 1.439   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            1    ss    5   0.833 ± 0.088   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            5    ss    5   1.464 ± 0.224   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                           50    ss    5   4.440 ± 0.700   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                          100    ss    5   7.265 ± 1.029   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            1    ss    5   2.913 ± 0.574   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            5    ss    5   4.400 ± 0.444   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                           50    ss    5  17.025 ± 2.188   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                          100    ss    5  34.733 ± 2.459   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   1.658 ± 0.345   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   3.183 ± 0.533   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                           50    ss    5  18.244 ± 1.344   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                          100    ss    5  36.146 ± 4.005   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            1    ss    5   0.487 ± 0.062   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            5    ss    5   0.508 ± 0.060   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                           50    ss    5   2.003 ± 0.051   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                          100    ss    5   2.584 ± 0.444   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            1    ss    5   0.476 ± 0.043   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            5    ss    5   0.509 ± 0.088   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                           50    ss    5   1.939 ± 0.110   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                          100    ss    5   2.210 ± 0.191   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            1    ss    5   0.504 ± 0.228   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            5    ss    5   0.568 ± 0.275   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                           50    ss    5   2.271 ± 0.532   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                          100    ss    5   2.718 ± 0.533   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            1    ss    5   0.591 ± 0.462   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            5    ss    5   0.508 ± 0.047   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                           50    ss    5   2.390 ± 0.840   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                          100    ss    5   2.796 ± 0.898   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            1    ss    5   1.046 ± 0.094   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            5    ss    5   1.246 ± 0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                           50    ss    5   4.596 ± 1.478   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                          100    ss    5   7.689 ± 2.578   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            1    ss    5   1.337 ± 1.510   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            5    ss    5   1.424 ± 0.758   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                           50    ss    5   5.610 ± 2.835   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                          100    ss    5  10.338 ± 3.360   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            1    ss    5   1.176 ± 1.446   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            5    ss    5   1.102 ± 0.626   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                           50    ss    5   2.982 ± 0.515   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                          100    ss    5   2.986 ± 0.499   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            1    ss    5   1.039 ± 0.501   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            5    ss    5   0.964 ± 0.059   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                           50    ss    5   2.725 ± 0.139   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                          100    ss    5   2.971 ± 0.389   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            1    ss    5   0.992 ± 0.228   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            5    ss    5   0.977 ± 0.033   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                           50    ss    5   3.017 ± 0.137   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                          100    ss    5   3.706 ± 1.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            1    ss    5   0.973 ± 0.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            5    ss    5   1.033 ± 0.051   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                           50    ss    5   3.118 ± 0.291   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                          100    ss    5   3.638 ± 0.398   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            1    ss    5   1.562 ± 0.116   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            5    ss    5   1.854 ± 1.245   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                           50    ss    5   5.304 ± 1.084   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                          100    ss    5   9.529 ± 3.073   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            1    ss    5   1.694 ± 0.341   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            5    ss    5   2.154 ± 1.048   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                           50    ss    5   6.544 ± 3.243   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                          100    ss    5  11.614 ± 8.374   s/op

Most of the benchmark shows no change as expected, the notable improvements are for the "large partitioned table with realistic object storage latency" cases:

(2)

RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   2.869 ±  0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   4.403 ±  0.355   s/op

(3):

RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   1.658 ± 0.345   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   3.183 ± 0.533   s/op

canContainDroppedFiles only checked minSequenceNumber inside the removedDataFilePaths branch, so a commit that calls dropDeleteFilesOlderThan without also removing data files (e.g. a plain append) never admitted delete manifests for that check, and obsolete delete files were silently never dropped in that case.

…ataFilesBenchmark The benchmark builds an unpartitioned table on the local filesystem, so it cannot show the cost of reading or writing manifests. A local open takes well under a millisecond while an object store charges tens of milliseconds per round trip, and manifest work against remote storage is dominated by those round trips rather than by decoding.

Removing data files makes every DV that references them dangling, so ManifestFilterManager scans delete manifests to find them. Previously canContainDroppedFiles admitted every delete manifest unconditionally, so a commit that replaces data files opened every delete manifest. Now uses partition pruning to determine which delete manifests to open. This mirrors partition pruning done with data manifests.

grantatspothero · 2026-09-18T17:41:25Z

cc: @nastra since you built the referencing PR.

github-actions Bot added the core label Sep 17, 2026

grantatspothero force-pushed the gn/pruneDeleteManifestScanForDanglingDVs branch 2 times, most recently from e80b3f8 to ffb5a5a Compare September 17, 2026 15:21

grantatspothero changed the title ~~Partition prune delete manifests in ManifestFilterManager~~ Core: Partition prune delete manifests in ManifestFilterManager Sep 17, 2026

grantatspothero force-pushed the gn/pruneDeleteManifestScanForDanglingDVs branch 2 times, most recently from b76776c to 634fca2 Compare September 17, 2026 15:44

grantatspothero added 3 commits September 18, 2026 12:32

grantatspothero force-pushed the gn/pruneDeleteManifestScanForDanglingDVs branch from 634fca2 to 6dca9e2 Compare September 18, 2026 17:34

Navigation Menu

Core: Partition prune delete manifests in ManifestFilterManager - #18154

Core: Partition prune delete manifests in ManifestFilterManager#18154
grantatspothero wants to merge 3 commits into
apache:mainfrom
grantatspothero:gn/pruneDeleteManifestScanForDanglingDVs

grantatspothero commented Sep 17, 2026 •

edited

Loading

Uh oh!

grantatspothero commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

grantatspothero commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grantatspothero commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

grantatspothero commented Sep 17, 2026 •

edited

Loading