Repository navigation
fix(bottomless): reap finished S3 uploads as they complete - #2299
Draft
pavanmanishd wants to merge 1 commit into
Draft
pavanmanishd wants to merge 1 commit into
pavanmanishd wants to merge 1 commit into
Conversation
The upload loop spawns one task per WAL batch into a JoinSet and drains that set only after the receive loop ends, which happens at shutdown. A JoinSet keeps every finished task until it is joined, so each uploaded batch stayed in memory for the life of the replicator. Reap the finished uploads inside the receive loop. The retained set is then bounded by s3_max_parallelism instead of growing with the WAL. Fixes tursodatabase#2298.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2298.
The bottomless upload loop spawns one task per WAL batch into a
JoinSetand drains that set only after the receive loop ends, which happens at shutdown. AJoinSetkeeps every finished task until it is joined, so each uploaded batch stayed in memory for the life of the replicator. The issue reportssqldgrowing by about 10 KB per batch until it is restarted or OOM-killed.This change reaps the finished uploads as they complete, inside the receive loop. The retained set is then bounded by
s3_max_parallelismplus the one task spawned in the current iteration, instead of growing with the WAL.The shutdown drain is unchanged. It already discarded task results, and the upload task still returns
().No test is added. S3-backed tests need a live endpoint and are not part of PR CI. The
nemesisS3 tests live in the separatetursodatabase/tursotestrepository and run only on manual dispatch.Checked locally with
cargo check -p bottomlessandcargo fmt -p bottomless -- --check;cargo clippy -p bottomlessreports no new findings.