Skip to content

tools check merged-blocks: add --parallel N - #356

Open
matthewdarwin wants to merge 3 commits into
streamingfast:developfrom
pinax-network:feat/check-merged-blocks-parallel
Open

matthewdarwin wants to merge 3 commits into
streamingfast:developfrom
pinax-network:feat/check-merged-blocks-parallel

Conversation

@matthewdarwin

@matthewdarwin matthewdarwin commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What

tools check merged-blocks --parallel N (default 8) reads up to N merged-blocks files at the same time with --validate-blocks. A full-chain --validate-blocks run over a remote store was limited by reading one file at a time. With this, downloading and decoding run concurrently, and the output stays exactly what a sequential run prints.

Per review, this also removes --print-stats (-s) and --print-full (-f) from check merged-blocks. Printing blocks is what tools print merged-blocks is for.

This picks up streamingfast/sf-tools#1 (2023), which stalled. The design follows the review there:

  • Ordered, interactive output. A dhammer.Nailer (already a dependency, used by the block poller) runs readBlockSegment on up to N files concurrently and hands the results back in file order. A single goroutine applies each one as it arrives: it links the blocks into the forkable.ForkDB and prints the file's lines. So holes, segments and every other line come out file by file, in order, as they arrive rather than at the end.
  • Same checking logic. validateBlockSegment is split in two:
    • readBlockSegment opens and decodes the file and runs the bundle-boundary validator. It touches no shared state.
    • segmentContent.apply holds the exact linking code that was in validateBlockSegment, and runs only on the consumer goroutine. There's no shared state between workers and nothing to lock.
  • Small memory use per file. The nailer keeps at most N files in flight. A file waiting to be applied keeps only Id, Number, ParentId, ParentNum, LibNum and Timestamp per block, with no payload. That's why parallel is now the default: the cost of a higher N is mostly load on the store.
  • One code path. With --parallel 1, each file is read and applied inline through the same two functions.

The hole lines (❌ Range … (Missing …)) and the every-10000-files progress lines printed by the walk are carried on the same item and printed before or after the file, so they keep their place in the output too.

API changes in the check package

  • Added: MergedBlocksCheckOptions.Parallel. Zero or one reads one file at a time.
  • Removed: PrintDetails and its constants, MergedBlocksCheckOptions.PrintDetails, and the deprecated CheckMergedBlocks. Their only callers were in this repository. firehose-aptos uses the old sftools package and isn't affected.
  • All of this is listed in the CHANGELOG under Added and Removed.

Tests

  • TestCheckMergedBlocks_ParallelOutputMatchesSequential: a 60-file store with one missing file, one file lacking blocks (everything after it unlinkable), one file holding a block beyond its bundle, and one short file. With Parallel 2, 4, 8 and 16, the output is byte for byte the sequential output.
  • Checks: go test -race ./cmd/tools/check/, go vet, gofmt and go build ./... pass. firecore tools check merged-blocks --help shows --parallel with (default 8) and no print flags.
  • Real data: on real merged-blocks files (Solana devnet, around a chain restart), firecore built from this branch printed the same output with --parallel 1 and --parallel 8 for:
    • a healthy range;
    • a range with a missing file;
    • a range with a broken parent link.

🤖 Generated with Claude Code

Reads up to N merged-blocks files at the same time when their content is
read (--validate-blocks / --print-stats). Each file is downloaded,
decoded and checked against its bundle boundaries in a dhammer.Nailer
worker (readBlockSegment), which touches no shared state and keeps only
what linking needs from each block. The nailer hands results back in
file order to a single goroutine that links them into the fork database
and prints (segmentContent.apply), with the hole and progress lines of
the walk carried on the same item, so the output is identical to a
sequential run and at most N files are in memory.

--parallel 1 (default) reads and applies each file inline: one code path.
--print-full refuses a value above 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@maoueh maoueh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove the print block capabilities of it.

Also, does parallel still shows a sequentially all the holes/segments still?

Comment thread cmd/tools/check/blocks.go Outdated
Comment on lines +59 to +61
if parallel > 1 && options.PrintDetails == PrintFull {
return fmt.Errorf("printing full blocks cannot run in parallel, use a parallelism of 1")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove altogether the ability to print blocks from this command. We have other tools now that make more sense anyway.

This will simplify everything.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d8c254c. --print-stats and --print-full are gone, along with PrintDetails, MergedBlocksCheckOptions.PrintDetails and the deprecated CheckMergedBlocks; their only callers were here. That also removed the per-block text kept in memory and the --print-full exception for parallel, so the change is now +18/-136. The CHANGELOG lists it under Removed and points to tools print merged-blocks.

Comment thread CHANGELOG.md Outdated

### Added

- `tools check merged-blocks --parallel N`: reads up to N merged-blocks files at the same time when their content is read (`--validate-blocks` or `--print-stats`). Downloading, decoding and the bundle-boundary checks run concurrently; linking the blocks into the chain and printing still happen in file order on a single goroutine, so the output is the same as without the flag and at most N files are held in memory. The default `1` keeps the sequential behavior; `--print-full` refuses a value above 1. A full-chain `--validate-blocks` run over a remote store was limited by reading one file at a time.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parallel should be the default if there is no real downside to it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: --parallel now defaults to 8. A file waiting to be applied keeps only the link fields of its blocks, with no payload, so a higher value costs little memory and mostly adds load on the store. 8 seemed safe for remote stores; happy to pick another number. --parallel 1 still reads one file at a time.

…l defaults to 8

Printing blocks is what 'tools print merged-blocks' does; check keeps
validating. Removes PrintDetails, MergedBlocksCheckOptions.PrintDetails and the
deprecated CheckMergedBlocks, and with them the per-block details kept for
printing and the --print-full parallel exception. Only what linking needs is
kept of each block, so --parallel defaults to 8; the output is unchanged from
--parallel 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matthewdarwin

matthewdarwin commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Also, does parallel still shows a sequentially all the holes/segments still?

Yes. Files are read concurrently, but one goroutine applies the results in file order: linking, hole lines, segment lines and progress lines. The output is byte for byte what --parallel 1 prints. TestCheckMergedBlocks_ParallelOutputMatchesSequential checks that at 2, 4, 8 and 16 on a store with a missing file, unlinkable blocks, a bloated file and a short file.

The print removal and the default are answered inline (d8c254c).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants