Skip to content

rfc(storage): tiered storage - #4404

Open
numinnex wants to merge 9 commits into
masterfrom
rfc_tiered_storage_w
Open

numinnex wants to merge 9 commits into
masterfrom
rfc_tiered_storage_w

Conversation

@numinnex

@numinnex numinnex commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@numinnex
numinnex added this pull request to stack #4405 October 5, 2026 09:45
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Oct 5, 2026
@numinnex
numinnex force-pushed the rfc_tiered_storage_w branch from e611efb to ca0214d Compare October 5, 2026 09:45
Comment thread rfcs/rfc-0-tiered-storage.md Outdated
Comment thread rfcs/rfc-0-tiered-storage.md Outdated
Comment thread rfcs/rfc-0-tiered-storage.md Outdated
@hubcio

hubcio commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

we're dropping purge, removal pr goes up today. so purge_generation can come out of the key layout and the marker. same for the purge invariants and the "purge during an upload" row. i think created_revision is enough on its own since offsets can't restart anymore, wdyt?

@hubcio
hubcio force-pushed the rfc_tiered_storage_w branch 4 times, most recently from 05d74d2 to 82a6d1d Compare October 10, 2026 12:21
Base automatically changed from rfc_template_w to master October 10, 2026 13:20
@hubcio
hubcio force-pushed the rfc_tiered_storage_w branch from 82a6d1d to 2aac350 Compare October 10, 2026 13:20
@djanand

djanand commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

@hubcio @numinnex thanks, read through this against master. Agree with the direction: keep the replicated local log as the write path, copy sealed segments out after, one replica uploads and tells the others an offset, stubs keep offset lookups off the network. A couple of questions/observations that I had

  1. Phases. The plan reads as a chain, but it looks to me like phases 1 and 3 don't depend on each other and only phase 2 is blocked on the partition state-machine RFC, with 4 and 5 behind it. Is that right? Is the phase 2 RFC on partition state-machine drafted anywhere ?
  2. The age roll in phase 5: There's a version that I think needs no new consensus operation seal when the arriving batch's timestamp is more than segment_max_age past the segment's first timestamp. Both are assigned by the
    primary and replicated unchanged, so every replica seals at the same message and segments stay identical. It covers slow-but-steady and long-gap traffic, leaving only a fully idle partition for the replicated roll. Is segment_max_age gated on tiered_storage being on? It's listed with the tiered options. If it works on any topic it also fixes message_expiry on a topic that doesn't fill a segment.

@numinnex
numinnex force-pushed the rfc_tiered_storage_w branch from d3b5560 to ee99522 Compare October 26, 2026 07:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accepted rfc S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants