Conversation
verify-cold decodes every ledger of each frozen chunk into Go structs and checks the ledgers pack as a source: the pack's content hash, the header's slot sequence and stored hash, the previous-hash chain including the link to the previous chunk's last ledger, the transaction set hash and the result set hash the header commits to, and optionally one header per chunk against a history archive. It then compares the chunk's events and tx-hash artifacts with what the SDK's decode path derives from those structs: every payload in cursor order against events.pack, the per-ledger offsets, every expected term's bitmap and the term count against the index, and every transaction hash, plus the inner hash of a fee bump, against the frozen tx-hash index covering the chunk and against the chunk's .bin while that is still frozen. A frozen index coverage whose chunks were all checked also gets its key count compared. The two tx-hash checks are keyed on their own catalog facts, because a terminal index commit demotes the per-chunk .bin keys: a chunk in a finalized window has only its index to check against, and a chunk still waiting on its index build has only its .bin. The oracle shares no extraction code with the view-based writers, only the cursor-sentinel policy and the term-key constructors that define the format. The tests hold it against both the v1 SQLite path and the v2 view path, run the source checks on a real pubnet ledger, and verify synthetic trees built through backfill.RunBackfill with injected faults. The catalog is opened read-only, so a run can sit beside a live daemon. Production packages gain read accessors and two small shared helpers: catalog.OpenReadOnly and the per-chunk secret derivations Catalog.TxHashIndexSecret and EventsIndexSecret, which backfill now uses too; PackReader.Verify and ledger.ColdReader.Verify; event.ColdReader.TermCount and event.TopicTermKey; txhash.ColdReader.KeyCount, txhash.ReadColdBin and txhash.SortColdEntries, which the cold ingester and the test fixtures now share with the verifier. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The chain authenticates backwards: each header commits to the one before it, so an authentic last header makes every header of the chunk the network's, while an authentic first header vouches for nothing after it. The archive anchor now compares the last header. The link to the previous chunk is recomputed from that chunk's decoded last header rather than read from its stored hash field, so the verifier reads nothing through the view accessors. The tx-hash indexes a run needs are resolved and opened up front from the target chunks instead of lazily under a mutex. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A successful InvokeHostFunction result carries sha256 over the return value and the contract events the invocation emitted, and the ledger header commits to every result. verify-cold now recomputes that hash for every successful invocation of a successful transaction from the events the oracle emits and compares it with the result's, so a chunk whose stored meta lost or altered an invocation's events fails even when every artifact agrees with the ledgers it was built from. The hash is taken over the oracle's events rather than the meta's own arrays, so a divergence shared by the decode path and the writers still fails here; the payload comparison holds the artifact byte-equal to those events. Three export shapes are handled. A native export hashes the operation's events exactly. At protocol 23 core may prepend asset-contract reconciliation events without hashing them, so a leading run of mint or burn events may be skipped. Below protocol 23 a V4 meta marks an export that backfilled asset-contract events: core rewrote such events in the operation meta after hashing them and kept the originals among the diagnostic events, so those are hashed when the operation's events no longer match, and an export without diagnostics is reported as not checkable rather than as a mismatch. Verified on three real pubnet chunks: 1,381,285 invocations under protocols 24 and later hash over their operation events exactly, and 655,157 invocations under protocol 22 from the backfilled SDF lake all check, none through the not-checkable path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds verify-cold to validate frozen rpcv2 artifacts against decoded source ledgers and network commitments.
Changes:
- Adds CLI orchestration, reporting, parallel chunk verification, and read-only catalog access.
- Validates ledger chains, events, transaction hashes, indexes, and invocation hashes.
- Adds shared cold-store helpers and comprehensive fixtures/tests.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
cmd/stellar-rpc/rpcv2/main.go |
Registers verify-cold. |
verify/run.go |
Coordinates parallel verification. |
verify/run_test.go |
Tests end-to-end verification scenarios. |
verify/report.go |
Defines results and summaries. |
verify/oracle.go |
Builds decode-path expectations. |
verify/oracle_test.go |
Tests oracle parity and commitments. |
verify/ledgers.go |
Validates ledger commitments. |
verify/fixtures_test.go |
Provides verification fixtures. |
verify/command.go |
Implements the CLI command. |
verify/chunk.go |
Verifies individual chunks. |
verify/artifacts.go |
Compares event and tx-hash artifacts. |
stores/txhash/cold_reader.go |
Exposes index key counts. |
stores/txhash/cold_index.go |
Reuses binary-header decoding. |
stores/txhash/cold_bin.go |
Adds binary reading and sorting helpers. |
stores/txhash/cold_bin_test.go |
Tests the shared binary reader. |
stores/packreader.go |
Exposes pack hash verification. |
stores/ledger/cold_reader.go |
Exposes ledger-pack verification. |
stores/event/index.go |
Exposes topic-term construction. |
stores/event/cold_reader.go |
Exposes event term counts. |
rpcv2test/rpcv2test.go |
Reuses production tx-hash helpers. |
ingest/txhash.go |
Reuses shared entry sorting. |
catalog/secret.go |
Centralizes derived index secrets. |
catalog/catalog.go |
Adds read-only catalog opening. |
backfill/process.go |
Uses catalog secret accessors. |
Suppressed comments (1)
cmd/stellar-rpc/internal/rpcv2/verify/chunk.go:197
- Artifact checks start as soon as each ledger passes, before the remaining ledgers and archive anchor have been validated. If an early artifact is bad and a later ledger fails its source checks, the report retains both findings, despite the documented guarantee that a bad source is reported alone; an artifact error can also prevent the anchor from running. Complete the full source/anchor pass before opening or comparing derived artifacts (or otherwise discard all derived findings when source validation later fails).
exp, err := expectLedger(r.d.opts.Passphrase, &lcm)
if err != nil {
r.rec.add(Mismatch{Ledger: seq, Artifact: "ledgers", Field: "decode_path", Actual: err.Error()})
r.sourceBad = true
return nil //nolint:nilerr // see above
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+80
to
+85
| targets, err := frozenChunks(cat, opts) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| indexes, err := openIndexes(cat, targets) | ||
| if err != nil { |
Resolve a chunk's tx-hash artifacts when it opens rather than when the run lists its targets: the backfill sweeps demoted .bin files right after a terminal index commit, so a run beside a live daemon could reach a chunk whose .bin was gone. A .bin missing since listing re-reads the catalog through a fresh read-only open, the run's own handle being a snapshot as of open, and checks the chunk through the index that now covers it; a key still frozen means the file really is missing. Index readers open on first use per coverage. Validate the chunk bounds and refuse a range that selects no frozen chunk, so a typo cannot produce a green run that verified nothing. Record only a content-hash mismatch or a corrupt file as a verdict on the ledgers pack; a missing or unreadable one is the chunk's error. Bind a reconciliation event to the contract of the asset it names before accepting it as an unhashed prefix at protocol 23. Name the transaction hash in mismatch log lines, and fix the two doc strings that still said the first header is anchored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
Previously missed (3) — in code that hasn't changed since the last review.
cmd/stellar-rpc/internal/rpcv2/verify/artifacts.go:162
- Returning when the recorder reaches its cap stops checking all remaining term postings, so
Droppeddoes not count those mismatches even though the recorder and CLI describe the overflow as fully counted. Keep scanning the keys and letrecorder.addsuppress storage while incrementingdropped.
This issue also appears on line 239 of the same file.
cmd/stellar-rpc/internal/rpcv2/verify/chunk.go:245
- After a decode failure,
prevHashstill refers to the ledger before the undecodable one. The next valid ledger is therefore compared against a stale hash and incorrectly gets an additionalprevious_ledger_hashmismatch; clear it so that unavailable link is skipped and checking can resume from the next decoded header.
cmd/stellar-rpc/internal/rpcv2/verify/chunk.go:251 - Advancing the chain with the stored
entry.Hashmakes a single corrupted stored hash also produce a falseprevious_ledger_hashmismatch on the following ledger. The chain commits to the hash computed from the previous header; use that computed value here, aspreviousChunkHashalready does.
cmd/stellar-rpc/internal/rpcv2/verify/artifacts.go:239
- Stopping this comparison when the recorder is full means later differing
.binentries are neither checked nor included inDropped, making the reported “more mismatches not shown” count inaccurate. Iterate the complete common range and rely onrecorder.addto enforce the output cap.
for i := 0; i < len(t.bin) && i < len(t.want) && !t.rec.full(); i++ {
Comment on lines
+72
to
+76
| for entry, err := range lr.IterateLedgers(r.c.FirstLedger(), r.c.LastLedger()) { | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if err := r.ledger(entry.Seq, entry.Bytes); err != nil { |
The archive anchor ran after the artifact checks, so an artifact that failed to open skipped it, and a ledger failing a source check mid-chunk left the chunk unanchored. It now runs on the last header before the walk, so its verdict stands whatever the walk finds, and a broken link found later is known to lie before an authentic header. The doc states the actual stage rule: derived artifacts are compared for every ledger up to the first that fails a source check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Suppressed comments (4)
Previously missed (2) — in code that hasn't changed since the last review.
cmd/stellar-rpc/internal/rpcv2/verify/artifacts.go:161
- Stopping here means later term postings are neither verified nor passed to
recorder.add, soDroppedundercounts mismatches and the--max-mismatchespromise that remaining mismatches are counted is false. Continue checking keys after the output cap;recorder.addalready avoids retaining them while incrementingdropped.
This issue also appears on line 239 of the same file.
cmd/stellar-rpc/internal/rpcv2/verify/report.go:38
- This documentation says
Invokesexcludes uncheckable invocations, butchunkRun.ledgerincrements it by allexp.invokes, including theInvokesUncheckedsubset, andSummarysubtracts that subset to derive the checked count. DocumentInvokesas the total andInvokesUncheckedas its subset so report consumers do not double-count or mislabel it.
cmd/stellar-rpc/internal/rpcv2/verify/chunk.go:327
- An archive-anchor mismatch is a failed source check, but this only records it and leaves
sourceBadfalse. The verifier therefore proceeds to compare all derived artifacts against ledgers it has just established are not the network's chain, contrary to the source-stage stop policy. Mark the source bad here so the ledger walk continues source checks without running artifact comparisons.
if entry.Hash != stored {
r.rec.add(Mismatch{
Ledger: seq, Artifact: "ledgers", Field: "archive_anchor",
Expected: hexHash(entry.Hash), Actual: hexHash(stored),
})
}
cmd/stellar-rpc/internal/rpcv2/verify/artifacts.go:239
- The
!t.rec.full()condition stops comparing.binentries once any earlier check fills the shared recorder. Remaining differing entries are therefore not added toDropped, even though the CLI says mismatches beyond the cap are counted. Iterate through all paired entries and letrecorder.addcount overflow without storing it.
for i := 0; i < len(t.bin) && i < len(t.want) && !t.rec.full(); i++ {
…hunk One rule for every file the verifier reads: a missing or unreadable file, or a canceled run, is the run's error; anything wrong with the bytes that are there is a verdict, recorded as a mismatch. Before this, every store reader error was treated as the run's error, so a malformed events segment, .bin, or index was reported as infrastructure and aborted the chunk, skipping the source checks of every later ledger and the other artifacts. An artifact that fails to open or read is now set aside on its own and the rest of the chunk is still checked; the chunk reports the environment failures it could not get past alongside its verdicts. Source checks: a ledger with more envelopes than results, which the decode path would index past, is a verdict rather than a crash; the chain check carries the hash computed over each header rather than the stored copy, so a corrupt stored hash blames one ledger; a ledger that does not decode leaves the next one's link unchecked rather than blamed; the pack's span is checked at both ends; a pack written without a content hash is a verdict, not a silent pass. The archive anchor runs first, a reachable archive that disagrees is a verdict on the last ledger, and an unreachable one is the chunk's error with the walk still run. An index file swept by a live daemon's rebuild since the run listed its coverage is re-resolved the way a swept .bin already was. A run whose range holds no frozen ledgers pack is an error rather than a green result. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment on lines
+458
to
+463
| reader, err := txhash.OpenColdReader(ic.layout.TxHashIndexFilePath(cov)) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| ic.readers[cov.Key] = &indexEntry{cov: cov, reader: reader} | ||
| return reader, nil |
Comment on lines
+182
to
+187
| if err != nil { | ||
| if isInfrastructure(err) { | ||
| r.infra = errors.Join(r.infra, fmt.Errorf("previous chunk %s: %w", prev, err)) | ||
| } | ||
| return nil | ||
| } |
Comment on lines
+326
to
+330
| r.invokes += uint64(len(exp.invokes)) | ||
| for _, c := range exp.invokes { | ||
| if c.skipped != "" { | ||
| r.invokesUnchecked++ | ||
| continue |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A
verify-coldsubcommand of the v2 binary that checks a cold tree's frozen chunks against the ledgers they were built from, so a bug that puts invalid data into a cold artifact shows up as a mismatch rather than as a wrong query answer months later.Per frozen chunk it runs two stages and stops at the first that fails.
The ledgers pack as a source. The pack's content hash is recomputed. Every ledger is then decoded into Go structs and checked: the header names its slot, hashes to the hash stored beside it, chains to the previous ledger (including the link to the previous chunk's last ledger), and commits to the stored envelopes and results through the transaction set hash and the result set hash. With
--history-archive-url, one header per chunk is also compared with the network's history archive.The derived artifacts against the decode path. From the same decoded ledgers, the SDK's
LedgerTransactionReaderandGetTransactionEventsproduce what the artifacts must hold, and the verifier compares:events.packposition by position, and every per-ledger range against the offsets;.binwhile that is still frozen;Invocation events against the network's own commitment. A successful InvokeHostFunction result carries a hash over the return value and the contract events the invocation emitted, and the header commits to every result. The verifier recomputes that hash from the events the oracle emits and compares it, so a chunk whose stored meta lost or altered an invocation's events fails even when every artifact agrees with the ledgers it was built from. This is the one check that does not depend on the decode oracle being right: a bug shared by the SDK's event accessor and the writers still fails it. It covers the contract events of successful invocations; nothing in the ledger commits to classic, fee or diagnostic events, so those remain covered by the oracle comparison alone.
Three export shapes are handled: a native export hashes the operation's events exactly; at protocol 23 core may prepend unhashed asset-contract reconciliation events, which are skipped; below protocol 23 a V4 meta marks an export that backfilled asset-contract events, where core rewrote those events after hashing them and kept the originals among the diagnostic events, so the originals are hashed instead and an export without diagnostics is reported as not checkable.
The two tx-hash checks are keyed on their own catalog facts. A terminal index commit demotes the per-chunk
.binkeys, so a chunk in a finalized window has only its index to check against, and a chunk still waiting on its index build has only its.bin.The oracle shares no extraction code with the view-based writers. The only shared functions define the format rather than read the ledger: the cursor-sentinel policy and the term-key constructors.
The archive anchor is taken on the chunk's last header, before the walk: the chain authenticates backwards, so an authentic last header makes every header of the chunk, and everything they commit to, the network's, and a broken link found later lies before it. Derived artifacts are compared for every ledger up to the first that fails a source check; from there on only the source checks continue.
The catalog is opened read-only, so a run can sit beside a live daemon. Output goes to the log, one line per mismatch up to
--max-mismatchesper chunk, and the exit status is non-zero on any mismatch.Why
stellar/go-stellar-sdk#5997 found the view-based transaction read path disagreeing with the decode path on a real transaction shape. The same class of bug in the writers would produce chunks that are internally consistent and wrong, and only reading the artifacts back can tell. Comparing artifacts against a decode oracle covers the extractors, the payload shaping, the index build and the encoders in one check, and also covers chunks written by an older binary and files damaged after the fact, which a code-path differential cannot.
Changes outside the new package
Production packages gain read accessors and two small shared helpers, with no change to what is written:
catalog.OpenReadOnly, andCatalog.TxHashIndexSecret/EventsIndexSecret, whichbackfillnow uses instead of deriving the secrets itself.stores.PackReader.Verifyandledger.ColdReader.Verifyover the existing content hash.event.ColdReader.TermCountandevent.TopicTermKey.txhash.ColdReader.KeyCount,txhash.ReadColdBinandtxhash.SortColdEntries, which the cold ingester, the.bincodec tests andrpcv2testnow share with the verifier.Tests
SorobanMeta, a fee bump, and a pre-Soroban V2 meta. All three agree on every event, cursor, byte and term.backfill.RunBackfill: a clean tree, an eventless chunk, artifacts diverging from replaced ledgers, a bad source stopping the artifact checks, a broken link between chunks, a damaged pack, a chunk range with an unfrozen ledgers artifact, the finalized-index shape with the.binkey demoted, and an index missing hashes.Three real pubnet chunks built from the lake and verified through the CLI with archive anchoring:
A dense chunk takes about two minutes on a laptop; chunks verify in parallel.
Not in this PR
verify-extractmode that runs the same oracle against a lake stream with no built tree, to certify a binary before a full backfill.🤖 Generated with Claude Code