Skip to content

tantivy: backfill today's converged files - #431

Merged
tonyalaribe merged 1 commit into
masterfrom
perf/consolidate-today-indexes
Oct 25, 2026
Merged

tonyalaribe merged 1 commit into
masterfrom
perf/consolidate-today-indexes

Conversation

@tonyalaribe

Copy link
Copy Markdown
Contributor

After #427 a warm 7-day errors list on 6297304f still searched ~291 indexes per query, most of them carried-forward entries on today's packed files (up to 539 covering one file). The census already counts such a file as uncovered (carried entries surrender ordinals), but the backfill dropped all of today's partition to avoid racing the hot-tail packer — and the packer never repacks a file at its target size.

drop_hot_partition now keeps today's files at or above timefusion_light_optimize_target_size, so the backfill rebuilds them as one index within its existing per-pass file/byte budget. Smaller (still-packing) files of today stay skipped.

Test: the hot-partition case table gains a converged file; red when the predicate is ignored.

Local signoff: make ci-signoff fmt/test/pg-smoke/e2e/clippy all pass (attested).

🤖 Generated with Claude Code

The backfill skipped today's partition to avoid racing the hot-tail packer, but
the packer never repacks a file at its target size. Such a file whose coverage is
only carried-forward entries (up to 539 on prod) was searched as hundreds of
indexes until the date rolled over; it is now rebuilt as one.
@tonyalaribe
tonyalaribe merged commit 25e1ad7 into master Oct 25, 2026
13 checks passed
@claude

claude Bot commented Oct 25, 2026

Copy link
Copy Markdown

Review: looks good, small and well-scoped.

  • Correctness: drop_hot_partition keeps today's files whose size is at or above the target, and sizes was already in scope. The .max(1) guard means a zero or negative config can't make every file count as converged. A file with no size entry (is_some_and) stays skipped, which is the safe default.
  • Race risk, worth verifying: the PR relies on the packer treating files at or above timefusion_light_optimize_target_size as never repacked. Please confirm the packer's own 'full' predicate uses the same threshold (>= vs >, and no slack factor). If the packer can still rewrite a file just over the target, such as a dedup or repair rewrite of today's partition, the backfilled index is voided. That wastes one pass's budget but isn't a correctness bug.
  • Style: the closure is computed inside the per-pid loop. target could be hoisted above the loop, which is minor.
  • Tests: the case table covers the new branch, including the prefix-marker and disabled cases. It only exercises the predicate in isolation, so there's no test that the backfill actually queues a converged today file. That's acceptable given the existing structure.

I haven't run the build or tests. LGTM, pending the threshold check above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant