Repository navigation
tools check merged-blocks: add --parallel N - #356
matthewdarwin wants to merge 3 commits into
Conversation
Reads up to N merged-blocks files at the same time when their content is read (--validate-blocks / --print-stats). Each file is downloaded, decoded and checked against its bundle boundaries in a dhammer.Nailer worker (readBlockSegment), which touches no shared state and keeps only what linking needs from each block. The nailer hands results back in file order to a single goroutine that links them into the fork database and prints (segmentContent.apply), with the hole and progress lines of the walk carried on the same item, so the output is identical to a sequential run and at most N files are in memory. --parallel 1 (default) reads and applies each file inline: one code path. --print-full refuses a value above 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
maoueh
left a comment
There was a problem hiding this comment.
Let's remove the print block capabilities of it.
Also, does parallel still shows a sequentially all the holes/segments still?
| if parallel > 1 && options.PrintDetails == PrintFull { | ||
| return fmt.Errorf("printing full blocks cannot run in parallel, use a parallelism of 1") | ||
| } |
There was a problem hiding this comment.
Let's remove altogether the ability to print blocks from this command. We have other tools now that make more sense anyway.
This will simplify everything.
There was a problem hiding this comment.
Done in d8c254c. --print-stats and --print-full are gone, along with PrintDetails, MergedBlocksCheckOptions.PrintDetails and the deprecated CheckMergedBlocks; their only callers were here. That also removed the per-block text kept in memory and the --print-full exception for parallel, so the change is now +18/-136. The CHANGELOG lists it under Removed and points to tools print merged-blocks.
|
|
||
| ### Added | ||
|
|
||
| - `tools check merged-blocks --parallel N`: reads up to N merged-blocks files at the same time when their content is read (`--validate-blocks` or `--print-stats`). Downloading, decoding and the bundle-boundary checks run concurrently; linking the blocks into the chain and printing still happen in file order on a single goroutine, so the output is the same as without the flag and at most N files are held in memory. The default `1` keeps the sequential behavior; `--print-full` refuses a value above 1. A full-chain `--validate-blocks` run over a remote store was limited by reading one file at a time. |
There was a problem hiding this comment.
Parallel should be the default if there is no real downside to it.
There was a problem hiding this comment.
Done: --parallel now defaults to 8. A file waiting to be applied keeps only the link fields of its blocks, with no payload, so a higher value costs little memory and mostly adds load on the store. 8 seemed safe for remote stores; happy to pick another number. --parallel 1 still reads one file at a time.
…l defaults to 8 Printing blocks is what 'tools print merged-blocks' does; check keeps validating. Removes PrintDetails, MergedBlocksCheckOptions.PrintDetails and the deprecated CheckMergedBlocks, and with them the per-block details kept for printing and the --print-full parallel exception. Only what linking needs is kept of each block, so --parallel defaults to 8; the output is unchanged from --parallel 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Yes. Files are read concurrently, but one goroutine applies the results in file order: linking, hole lines, segment lines and progress lines. The output is byte for byte what The print removal and the default are answered inline (d8c254c). |
What
tools check merged-blocks --parallel N(default 8) reads up to N merged-blocks files at the same time with--validate-blocks. A full-chain--validate-blocksrun over a remote store was limited by reading one file at a time. With this, downloading and decoding run concurrently, and the output stays exactly what a sequential run prints.Per review, this also removes
--print-stats(-s) and--print-full(-f) fromcheck merged-blocks. Printing blocks is whattools print merged-blocksis for.This picks up streamingfast/sf-tools#1 (2023), which stalled. The design follows the review there:
dhammer.Nailer(already a dependency, used by the block poller) runsreadBlockSegmenton up to N files concurrently and hands the results back in file order. A single goroutine applies each one as it arrives: it links the blocks into theforkable.ForkDBand prints the file's lines. So holes, segments and every other line come out file by file, in order, as they arrive rather than at the end.validateBlockSegmentis split in two:readBlockSegmentopens and decodes the file and runs the bundle-boundary validator. It touches no shared state.segmentContent.applyholds the exact linking code that was invalidateBlockSegment, and runs only on the consumer goroutine. There's no shared state between workers and nothing to lock.Id,Number,ParentId,ParentNum,LibNumandTimestampper block, with no payload. That's why parallel is now the default: the cost of a higher N is mostly load on the store.--parallel 1, each file is read and applied inline through the same two functions.The hole lines (
❌ Range … (Missing …)) and the every-10000-files progress lines printed by the walk are carried on the same item and printed before or after the file, so they keep their place in the output too.API changes in the
checkpackageMergedBlocksCheckOptions.Parallel. Zero or one reads one file at a time.PrintDetailsand its constants,MergedBlocksCheckOptions.PrintDetails, and the deprecatedCheckMergedBlocks. Their only callers were in this repository. firehose-aptos uses the oldsftoolspackage and isn't affected.Tests
TestCheckMergedBlocks_ParallelOutputMatchesSequential: a 60-file store with one missing file, one file lacking blocks (everything after it unlinkable), one file holding a block beyond its bundle, and one short file. WithParallel2, 4, 8 and 16, the output is byte for byte the sequential output.go test -race ./cmd/tools/check/,go vet,gofmtandgo build ./...pass.firecore tools check merged-blocks --helpshows--parallelwith(default 8)and no print flags.firecorebuilt from this branch printed the same output with--parallel 1and--parallel 8for:🤖 Generated with Claude Code