Conversation
…dd the checksum counter family (format_version 5) A checksum pass was dark to an observer: the tracker could report a running step but nothing about how far the comparison or the repair had got. The verifier is now the tracker's WorkSource for the lifetime of Verify or Check (Options.Tracker), as the copier is for the copy, and reports a third counter family read from the pass's own memory: chunks_compared and rows_hashed (two-sided digests that committed and the source rows they covered; a repaired chunk's reread counts again), chunks_mismatched (chunks the comparison found differing), and chunks_repaired (chunks whose one recopy transaction committed). The counters reset when a pass starts, and Check registers once so the repair phase is covered without a second registration. Adding fields and the `checksum` operation to the snapshot bumps format_version to 5. Tests: unit tests for the counters, the reset at report, registration and release; integration tests polling the tracker after every statement of a repair pass (each chunk counted with the source's row count the moment it commits, all three mismatches counted before the repair opens, repaired moving 0 -> 3 at the one commit, the rereads counted again), of a clean pass and a second pass on the same verifier (first poll sees every counter reset), and of an abort pass (mismatches counted, nothing repaired); the progress JSON shape tests pin the version and the zero counters, and a new one pins a checksum step. Docs: progress-report.md gains the version-5 note, the checksum counter family, the checksum operation row, and a checksum-step example; the copy-and-swap package map, architecture.md and SAFETY.md name the verifier as a work source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the checksum verifier the progress tracker's
WorkSourcefor the lifetime ofVerifyorCheck, and adds the counter family a poller needs to see a pass move: chunks compared, rows hashed, chunks mismatched, chunks repaired. The snapshot contract moves toformat_version5.Why
The #136 review found progress dark during
Check: the tracker could say a checksum step was running, but nothing about how far the comparison had got, whether it had found anything, or whether the repair had committed. The copier already fills the tracker'sWorkSourceseam (#132); the verifier is the second engine-measured step, and an orchestrator wiringCheckneeds its counters before it does.What
pkg/progress:FormatVersion4 → 5.OperationChecksum = "checksum".Workgainschunks_compared,rows_hashed,chunks_mismatched,chunks_repaired, placed after the copy counters and before the concurrent-index ones. A copy step reports them as zero, as a checksum step reports the copy counters.pkg/checksum:Options.Tracker *progress.Tracker. When set,VerifyandCheckregister the verifier withSetWorkSourcebefore the first chunk andStopWorkSourcejust before returning;Checkregisters once, so the repair phase is covered without a second registration.Verifier.Work(ctx)reads the pass's counters from memory under one mutex — no catalog read, so a poll never waits on the database and the CO-9 read-site rule has nothing to cover.digestChunkcounts a compared chunk and its source rows after its commit (a repaired chunk's reread counts again);passcounts a mismatch as it records one;recopycounts every repaired chunk at the one commit. Counters reset when a pass starts, so the only state a verifier keeps between passes is the pass in progress.docs/progress-report.md(version-5 note, the checksum counter family and its semantics table, thechecksumoperation row, a checksum-step JSON example);docs/copy-and-swap-design.mdpackage map,docs/architecture.mdandSAFETY.mdpkg/checksum/pkg/progressrows.Counter semantics
chunks_comparedrows_hashedchunks_comparedchunks_mismatchedabort, where nothing is repairedchunks_repaired0tochunks_mismatchedat once; a chunk still differing at reread stops the pass but stays counted, since its recopy did commitTests
Unit: counters accumulate what the pass reports;
report()zeroes the previous pass and registers, a poll during the pass carries the verifier's work and none after stop; no tracker configured costs nothing.Integration (real PostgreSQL, tracker polled after every statement through a pool tracer): a repair pass over a shadow differing in every chunk is seen stage by stage — the first chunk counted with the source's 1000 rows (not the shadow's 999), the repair's first statement polled at compared 3 / hashed 2500 / mismatched 3 / repaired 0, the first reread polled with repaired 3, and the verifier reporting 6 / 5000 / 3 / 3 after return with the tracker released; a clean pass counts 3 / 2500 and a second pass on the same verifier is first polled with every counter zero; an abort pass counts 3 mismatches and no repair and still releases the tracker. Progress JSON shape tests pin version 5 and the zero counters on copy and native steps, and a new one pins a checksum step.
Decisions to veto
chunks_mismatchedis one counter beyond the three the review asked for. Without it anabortpass reports nothing about divergence until it returns; it is the one signal that distinguishes "comparing" from "found something".Workreads memory only. The copier measurespg_table_sizeat each poll; a checksum pass has no comparable size to measure, so no connection is held for progress.format_versionbumps to 5 for the added fields and the new operation value, per the contract's rule that adding a field bumps the version. A consumer pinned to 4 must move.Verification
go test -race ./pkg/checksum/ ./pkg/progress/ ./pkg/copier/,SKIP_INTEGRATION=1 go test ./...,make lint(0 issues), andscripts/test-flaky.sh×5 on the three new integration tests, all green locally on PG16.