Skip to content

Keep tier1 streams going when merger is behind - #1067

Merged
sduchesneau merged 12 commits into
developfrom
feature/hub-keep-blocks-until-merged
Nov 13, 2026
Merged

sduchesneau merged 12 commits into
developfrom
feature/hub-keep-blocks-until-merged

Conversation

@sduchesneau

@sduchesneau sduchesneau commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Needs streamingfast/bstream#92 (go.mod points at that branch for now and must be bumped again once it is merged).

  • The tier1 live hub keeps its final blocks from the start of the last merged-blocks file, instead of a fixed 200 blocks below LIB. Before, a merger more than 200 blocks behind stalled every stream that read through merged-blocks files. app.HubKeepFinalBlocks is removed, LiveHubConfig.MergedBlocksStore is required (NewLiveHub now returns an error and refuses a config without it), the hub is given MergedBlocksBundleSize as the size of the files to look for, and HubMaxKeptFinalBlocks limits the blocks held in memory (default 20 merged-blocks files worth).
  • GetRecentFinalBlock is capped at the start of the last merged-blocks file the hub found. With a merger behind, the linear handoff was placed above it and tier2 jobs waited for files that were not written yet. It is the start of the last file and not its end because tier2 needs the first block of the next file to know a segment is over. The two checks that disconnect a stream too far behind the last final block use the same value. Before the first merged-blocks file exists, the handoff is the first streamable block.
  • LiveHubConfig and Tier1Modules take an optional metric for the final blocks not merged yet.
  • NewLiveHub returns an error, which it does when MergedBlocksStore is missing.

With a merger keeping up, the handoff only sits one file lower for the few seconds the hub takes to find each new file.

Tested

substreams run common -o clock --production-mode against a local dummy chain (segment size 100), fixed build against develop:

  • merger stopped at file 800, LIB at 1500: a request for 650 to 1100 completes with the handoff at 800. On develop it stops at block 799.
  • no merger and no merged-blocks file: a request from block 0 completes with the handoff at 0.

These runs predate the later commits on this branch (file size passed to the hub, NewLiveHub error, bstream bumps); the handoff is also covered by a test against a real hub.

Limits:

  • A hub that boots further above the last merged-blocks file than its limit does not look for files below that limit, so it reports merged blocks as ending at its own depth. The handoff is then placed there and tier2 waits for the merger, as it did before this change.
  • The live backfiller was not looked at and may still wait on merged-blocks files during an outage.

The live hub keeps final blocks until they are merged, and the linear
handoff stops at the last merged block so tier2 jobs do not wait for
merged-blocks files that are not written yet.
The merged-blocks store alone decides what the live hub keeps, instead
of that plus a fixed 200 blocks below LIB.
Tier2 needs the first block of the next merged-blocks file to know a
segment is over, so a segment ending where merged blocks end waited
for the merger.
Also note the memory the hub can hold while the merger is behind, and fix two
comments on the handoff cap.
…locks-until-merged

# Conflicts:
#	docs/release-notes/change-log.md
#	go.mod
#	go.sum
#	go.work.sum
@dfuse-bot

dfuse-bot commented Nov 12, 2026 •

Copy link
Copy Markdown

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:a8b1123

📦 Image Reference ghcr.io/streamingfast/substreams:a8b1123
digestsha256:ce517eb54ca5a631605fc4f76520084dc40bc6a64fa7f41432dcfe5d16749d42
vulnerabilitiescritical: 0 high: 0 medium: 0 low: 0
platformlinux/amd64
size98 MB
packages387
📦 Base Image ubuntu:24.04
also known as
  • noble
  • noble-20261110
digestsha256:77eb101c607d6e3d4f5ad0c8a02c2136c689b98aba1f359e6502ba517e5c57ec
vulnerabilitiescritical: 0 high: 0 medium: 13 low: 2

@sduchesneau
sduchesneau marked this pull request as ready for review November 12, 2026 21:27
Comment thread app/live_hub.go
// when the merger is behind: 20 merged-blocks files worth. A hub holds up to two files
// worth when the merger keeps up.
func HubMaxKeptFinalBlocks(mergedBlocksBundleSize uint64) int {
return int(20 * bstream.SanitizeBundleSize(mergedBlocksBundleSize))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

magic number here! Style choice but I dont like this pattern, move it to a constant
also, is the 20 merged-blocks safe ? if a merger is behind, a tier1 can hold 2000 blocks in memory instead of 200

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is a default value that is influenced by the bundle size. It can be overwritten in firecore with the flag. A constant wouldn't help -- this is the default value "20 x the bundle size".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also,

  1. the alternative is worst (dropping those files but then some requests get stuck!
  2. we can set a specific value per-chain on each deployment.

So I wouldn't take this as too much of a problem, since it's just the choice of default.

@sduchesneau
sduchesneau merged commit 1661a96 into develop Nov 13, 2026
10 checks passed
@sduchesneau
sduchesneau deleted the feature/hub-keep-blocks-until-merged branch November 13, 2026 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants