[xingbowang]

Summary:

Why

Some useful compaction filters only make sense on merged results. For example, a filter that removes counters below a threshold must see the counter's total, not each +N operand. With the current order, the compaction filter runs before the merge operator, so it sees (and can remove) individual operands and never sees the merge result.

Important limit: compaction can only merge what is in its own inputs. If a key's base value or older operands are in files outside the compaction, the filter sees a partial merge result (or single operands), not the logical value of the key.

Behavior

• Default (unchanged): CompactionFilter::MergeOrder::kFilterBeforeMerge. Operands are filtered one by one before merging.
• New opt-in: a filter or factory returns kMergeBeforeFilter from the new GetMergeOrder(). During compaction the merge operator runs first, then the filter sees:
  • the full merge result as kValue / kWideColumnEntity, if the inputs include the base (Put or tombstone), or the compaction is bottommost with no older entries;
  • otherwise the operands as kMergeOperand: one partial result from PartialMergeMulti, or each operand by itself if partial merge declines.
• A filter result is not merged again in the same compaction. Filters that only implement FilterV2/V3 also work.
• A merge result is offered only if no snapshot can see any of its operands. Plain values are filtered as before, regardless of snapshots.
• kPurge on a value is treated as kRemove (ordinary Delete), because SingleDelete does not mix with Merge history. kPurge and kChangeWideColumnEntity on an operand fail the compaction with NotSupported.
• kRemoveAndSkipUntil on any operand of an uncombined batch drops the whole batch, the same result as the default order.
• COMPACTION_KEY_DROP_USER keeps the same meaning in both orders (skipped ranges are not counted).
• The new order applies only to compaction. Flush, recovery, MemPurge, and repair keep the default order.

Example

Counter merge operator (add) with a filter that removes counters below 10. The compaction input for one key is Merge(+3), Merge(+5), Put(4). Before: the filter sees the raw operands +3 and +5, removes both, and the output is Put(4); it never sees the merged value. After: the merge runs first (4 + 5 + 3 = 12), the filter sees kValue "12", and keeps it. If the base is not in the compaction, partial merge gives +8 and the filter sees it as kMergeOperand.

Implementation

• MergeHelper carries the merge order and records each merge outcome (full, partial, uncombined) and whether it can be filtered under snapshots.
• CompactionIterator reads the order from MergeHelper and filters after the merge. A full result changed to Delete goes through the normal ProcessDeletion() path. Uncombined operand batches are filtered as a whole before any of them is output, so a skip decision can drop the full batch.
• Order checks (GetCompactionFilterMergeOrder()) run outside the DB mutex. A factory's non-null filters must declare the same order; otherwise the compaction fails with InvalidArgument. A factory that returns a null filter means no filtering, which is valid with either order.
• Remote compaction: CompactionService::SupportsMergeBeforeFilter() (default false) lets a service receive new-order jobs; otherwise they run locally. The request carries merge_before_filter, and the worker acknowledges the order it actually used. A missing or wrong acknowledgment (for example, from an older worker) falls back to local compaction before any output is imported. Real compaction errors stay errors. A worker with no filter accepts either order.
• Early rejection: DB open and CF creation reject a new-order declaration with user-defined timestamps, WritePrepared/WriteUnprepared transactions, stacked BlobDB, or a factory that selects non-compaction table file creation. This avoids a background error that would stop writes at the first compaction.
• ShouldFilterTableFileCreation() checks use an exhaustive switch over TableFileCreationReason, so a new reason must be classified.
• Bug fix included: MemPurge() returned with the DB mutex unlocked when the flush filter failed a check (for example, IgnoreSnapshots() == false). The new order check adds another early return on the same path, so the re-lock is fixed here, with a bug_fixes release note.
• Stress test: with merge_before_compaction_filter, the db_stress filter now makes real decisions that never change what a reader sees: kChangeValue with the same bytes for about 1 in N records (merge_before_filter_change_one_in); with merge_before_filter_remove_deleted, kRemove for keys deleted in the expected state, and kRemoveAndSkipUntil over just that key in full compactions (where every version of the key is an input). It also aborts if a value it sees is not a well-formed stress value for its key, or if the new-order factory is asked to filter a non-compaction file. It prints per-type decision counts at the end of a run. The crash test picks these modes randomly and uses merges in about 3/4 of new-order runs; the remove/skip mode uses the same no-snapshot settings as the default-order filter.
• Repairer only marks BuildTable()'s IOStatus as checked, because BuildTable() can now return early before reading it. Repair behavior does not change.

Safety and limitations

• The default order and existing filters behave as before. There is no SST format change, no new CF option, and no C/Java binding.
• Not supported in the new order: user-defined timestamps, WritePrepared/WriteUnprepared transactions (SnapshotChecker), and stacked BlobDB. Integrated BlobDB is supported.
• Do not use SingleDelete on keys handled by a new-order filter, even after switching back to the default order.
• How operands are grouped into partial results can differ between compactions, and SST files keep no record of earlier groupings. Filter decisions must be correct for any grouping.
• Stress test limits: the stress merge operators keep only the newest operand, and every stress filter decision is chosen so that reads do not change. So both orders give the same visible result, and the oracle checks robustness of the new paths (wrong output key or sequence, lost or returning data, buffer and pinning errors, crash recovery, remote fallback), not whether the filter saw the right input. A same-bytes rewrite also cannot detect a change that RocksDB ignores. Skips only run in full compactions, which are rare in leveled runs. The decision counters print only when db_stress exits normally (not in blackbox kills).
• Unit tests cover the rest: which input the filter sees (full, partial, uncombined), the snapshot rule, changed values in the output (MergeBeforeFilterFullReplacementAndFailure, MergeBeforeFilterPartialAndDecline, remote full/partial tests), skip and removal results, kPurge handling, unsupported decisions, early rejection, remote routing and fallback, and BlobDB.
• Validation: the full suites of the touched test binaries pass, and local db_stress and crash-test runs with the new modes pass. See the Test Plan for details and the benchmark results.

Differential Revision: D128313536

[meta-codesync]

@xingbowang has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128313536.

[github-actions]

⚠️ clang-tidy: 1 warning(s) on changed lines

Completed in 1059.6s.

Summary by check

Check	Count
cppcoreguidelines-special-member-functions	1
Total	1

Details

db/flush_job_test.cc (1 warning(s))

db/flush_job_test.cc:190:9: warning: class 'Filter' defines a non-default destructor but does not define a copy constructor, a copy assignment operator, a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]