Repository navigation
Keep tier1 streams going when merger is behind - #1067
Merged
Merged
Conversation
The live hub keeps final blocks until they are merged, and the linear handoff stops at the last merged block so tier2 jobs do not wait for merged-blocks files that are not written yet.
The merged-blocks store alone decides what the live hub keeps, instead of that plus a fixed 200 blocks below LIB.
Tier2 needs the first block of the next merged-blocks file to know a segment is over, so a segment ending where merged blocks end waited for the merger.
Also note the memory the hub can hold while the merger is behind, and fix two comments on the handoff cap.
…locks-until-merged # Conflicts: # docs/release-notes/change-log.md # go.mod # go.sum # go.work.sum
sduchesneau
marked this pull request as ready for review
November 12, 2026 21:27
| // when the merger is behind: 20 merged-blocks files worth. A hub holds up to two files | ||
| // worth when the merger keeps up. | ||
| func HubMaxKeptFinalBlocks(mergedBlocksBundleSize uint64) int { | ||
| return int(20 * bstream.SanitizeBundleSize(mergedBlocksBundleSize)) |
Contributor
There was a problem hiding this comment.
magic number here! Style choice but I dont like this pattern, move it to a constant
also, is the 20 merged-blocks safe ? if a merger is behind, a tier1 can hold 2000 blocks in memory instead of 200
Contributor
Author
There was a problem hiding this comment.
it is a default value that is influenced by the bundle size. It can be overwritten in firecore with the flag. A constant wouldn't help -- this is the default value "20 x the bundle size".
Contributor
Author
There was a problem hiding this comment.
Also,
- the alternative is worst (dropping those files but then some requests get stuck!
- we can set a specific value per-chain on each deployment.
So I wouldn't take this as too much of a problem, since it's just the choice of default.
UlysseCorbeil
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Needs streamingfast/bstream#92 (
go.modpoints at that branch for now and must be bumped again once it is merged).app.HubKeepFinalBlocksis removed,LiveHubConfig.MergedBlocksStoreis required (NewLiveHubnow returns an error and refuses a config without it), the hub is givenMergedBlocksBundleSizeas the size of the files to look for, andHubMaxKeptFinalBlockslimits the blocks held in memory (default 20 merged-blocks files worth).GetRecentFinalBlockis capped at the start of the last merged-blocks file the hub found. With a merger behind, the linear handoff was placed above it and tier2 jobs waited for files that were not written yet. It is the start of the last file and not its end because tier2 needs the first block of the next file to know a segment is over. The two checks that disconnect a stream too far behind the last final block use the same value. Before the first merged-blocks file exists, the handoff is the first streamable block.LiveHubConfigandTier1Modulestake an optional metric for the final blocks not merged yet.NewLiveHubreturns an error, which it does whenMergedBlocksStoreis missing.With a merger keeping up, the handoff only sits one file lower for the few seconds the hub takes to find each new file.
Tested
substreams run common -o clock --production-modeagainst a local dummy chain (segment size 100), fixed build againstdevelop:developit stops at block 799.These runs predate the later commits on this branch (file size passed to the hub,
NewLiveHuberror, bstream bumps); the handoff is also covered by a test against a real hub.Limits: