Skip to content

bazel: sweep only generations that changed - #42

Merged
Uzaaft merged 2 commits into
masterfrom
perf/bazel-sweep
Nov 18, 2026
Merged

Uzaaft merged 2 commits into
masterfrom
perf/bazel-sweep

Conversation

@Uzaaft

@Uzaaft Uzaaft commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

No description provided.

The collector swept every generation each minute: full listings of
its index, objects, receipts, sessions, and markers, a read of each
entry, and a usage upsert, then a listing of every Bazel object.
Almost every pass recomputed what the last one had found, and with
forty generations that was the largest periodic cost in the service.

Each store operation that writes a generation's objects now marks it
as the operation ends, and wakes the collector. A pass sweeps only
the marked generations, no sooner than a minute after the last pass,
so a build's writes share sweeps as before while a lone write is
observed at once. Management rereads the targets after it creates a
Bazel cache or changes a capacity or an organization's limit, and
only generations left above their target are swept. With nothing
marked, the collector issues no storage request and no statement.

A pass over everything remains, one minute after start and then
hourly. It covers what this process never saw: another instance's
writes, manual changes, and targets moved by other products' usage.
It is also the timer for the two-hour graces and for retired
generations, which are minimum ages rather than deadlines. Access
markers mark nothing: a swept generation is within its target, so a
marker alone evicts nothing and changes no usage.
The collector read retention targets and recorded usage through runtime
query functions, so a schema change could only fail them when a sweep
ran. They now go through the query macros and are described in the
offline metadata with the rest of the server's statements. The
retention view's columns are marked non-null by hand because Postgres
reports every view column as nullable.
@Uzaaft
Uzaaft merged commit cdc2e3c into master Nov 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant