Skip to content

progress, checksum: make the verifier the tracker's work source and add the checksum counter family (format_version 5) - #142

Draft
Kiran01bm wants to merge 1 commit into
mainfrom
kiran01bm/cs6-checksum-progress
Draft

Kiran01bm wants to merge 1 commit into
mainfrom
kiran01bm/cs6-checksum-progress

Conversation

@Kiran01bm

Copy link
Copy Markdown
Collaborator

Makes the checksum verifier the progress tracker's WorkSource for the lifetime of Verify or Check, and adds the counter family a poller needs to see a pass move: chunks compared, rows hashed, chunks mismatched, chunks repaired. The snapshot contract moves to format_version 5.

Why

The #136 review found progress dark during Check: the tracker could say a checksum step was running, but nothing about how far the comparison had got, whether it had found anything, or whether the repair had committed. The copier already fills the tracker's WorkSource seam (#132); the verifier is the second engine-measured step, and an orchestrator wiring Check needs its counters before it does.

What

  • pkg/progress:
    • FormatVersion 4 → 5. OperationChecksum = "checksum".
    • Work gains chunks_compared, rows_hashed, chunks_mismatched, chunks_repaired, placed after the copy counters and before the concurrent-index ones. A copy step reports them as zero, as a checksum step reports the copy counters.
  • pkg/checksum:
    • Options.Tracker *progress.Tracker. When set, Verify and Check register the verifier with SetWorkSource before the first chunk and StopWorkSource just before returning; Check registers once, so the repair phase is covered without a second registration.
    • Verifier.Work(ctx) reads the pass's counters from memory under one mutex — no catalog read, so a poll never waits on the database and the CO-9 read-site rule has nothing to cover.
    • Counter sites: digestChunk counts a compared chunk and its source rows after its commit (a repaired chunk's reread counts again); pass counts a mismatch as it records one; recopy counts every repaired chunk at the one commit. Counters reset when a pass starts, so the only state a verifier keeps between passes is the pass in progress.
  • Docs in the same PR: docs/progress-report.md (version-5 note, the checksum counter family and its semantics table, the checksum operation row, a checksum-step JSON example); docs/copy-and-swap-design.md package map, docs/architecture.md and SAFETY.md pkg/checksum / pkg/progress rows.

Counter semantics

Counter Moves when Note
chunks_compared a two-sided digest commits every chunk of the comparison once, every repaired chunk once more at its reread — so it passes the comparison's chunk count during the repair phase
rows_hashed with chunks_compared the source rows the digest covered; a shadow missing a row still counts the source's full chunk
chunks_mismatched the comparison records a differing chunk the only divergence signal under abort, where nothing is repaired
chunks_repaired the repair transaction commits one transaction recopies every differing chunk, so it moves from 0 to chunks_mismatched at once; a chunk still differing at reread stops the pass but stays counted, since its recopy did commit

Tests

Unit: counters accumulate what the pass reports; report() zeroes the previous pass and registers, a poll during the pass carries the verifier's work and none after stop; no tracker configured costs nothing.

Integration (real PostgreSQL, tracker polled after every statement through a pool tracer): a repair pass over a shadow differing in every chunk is seen stage by stage — the first chunk counted with the source's 1000 rows (not the shadow's 999), the repair's first statement polled at compared 3 / hashed 2500 / mismatched 3 / repaired 0, the first reread polled with repaired 3, and the verifier reporting 6 / 5000 / 3 / 3 after return with the tracker released; a clean pass counts 3 / 2500 and a second pass on the same verifier is first polled with every counter zero; an abort pass counts 3 mismatches and no repair and still releases the tracker. Progress JSON shape tests pin version 5 and the zero counters on copy and native steps, and a new one pins a checksum step.

Decisions to veto

  • chunks_mismatched is one counter beyond the three the review asked for. Without it an abort pass reports nothing about divergence until it returns; it is the one signal that distinguishes "comparing" from "found something".
  • The verifier counts, the tracker observes: Work reads memory only. The copier measures pg_table_size at each poll; a checksum pass has no comparable size to measure, so no connection is held for progress.
  • format_version bumps to 5 for the added fields and the new operation value, per the contract's rule that adding a field bumps the version. A consumer pinned to 4 must move.

Verification

go test -race ./pkg/checksum/ ./pkg/progress/ ./pkg/copier/, SKIP_INTEGRATION=1 go test ./..., make lint (0 issues), and scripts/test-flaky.sh ×5 on the three new integration tests, all green locally on PG16.

…dd the checksum counter family (format_version 5)

A checksum pass was dark to an observer: the tracker could report a
running step but nothing about how far the comparison or the repair had
got. The verifier is now the tracker's WorkSource for the lifetime of
Verify or Check (Options.Tracker), as the copier is for the copy, and
reports a third counter family read from the pass's own memory:
chunks_compared and rows_hashed (two-sided digests that committed and
the source rows they covered; a repaired chunk's reread counts again),
chunks_mismatched (chunks the comparison found differing), and
chunks_repaired (chunks whose one recopy transaction committed). The
counters reset when a pass starts, and Check registers once so the
repair phase is covered without a second registration.

Adding fields and the `checksum` operation to the snapshot bumps
format_version to 5.

Tests: unit tests for the counters, the reset at report, registration
and release; integration tests polling the tracker after every
statement of a repair pass (each chunk counted with the source's row
count the moment it commits, all three mismatches counted before the
repair opens, repaired moving 0 -> 3 at the one commit, the rereads
counted again), of a clean pass and a second pass on the same verifier
(first poll sees every counter reset), and of an abort pass (mismatches
counted, nothing repaired); the progress JSON shape tests pin the
version and the zero counters, and a new one pins a checksum step.

Docs: progress-report.md gains the version-5 note, the checksum counter
family, the checksum operation row, and a checksum-step example; the
copy-and-swap package map, architecture.md and SAFETY.md name the
verifier as a work source.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant