Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTENT_BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The Module column uses `1A` (definition quality) and `1B` (data integrity) for t
| M1-05 | 1B | Agreement measurement for analytics QA | pattern | v0.2.0 | drafted | Twenty-nine teams on one dataset returned odds ratios from 0.89 to 2.93; ISO 5725's repeatability and reproducibility as the lens; LinkedIn found 60 percent of its questions had more than one right answer | 1 session |
| M1-06 | 1B | Data quality SLOs and error budgets | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-07 | 1B | Schema drift detection and data contracts | pattern | v0.1.0 | drafted | Avro resolution as the mechanism behind the quiet change; GoCardless's six-month report on what a producer-side programme costs; the author's addition of the assistant's memory as a drift of its own | 1 session |
| M1-08 | 1B | The reconciliation protocol | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-08 | 1B | The reconciliation protocol | pattern | v0.2.0 | drafted | The bank reconciliation's reconciling-items schedule as the lineage, from the 2025 Green Book and a state auditor's guide; Airbnb, Uber and Zoom as the two-numbers stories; the author's three causes, exclusions, calendars and as-of time | 1 session |
| M1-09 | 1B | Measuring trust, not just accuracy | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-10 | 1B | Data incident root cause analysis | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-11 | 1A | State-based retention measurement | pattern | v0.2.0 | drafted | Duolingo growth model (Gustafson, Mazal) | 1 session |
Expand Down
8 changes: 8 additions & 0 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

One entry per judgment call: decision, alternatives, why.

## 2026-09-18 — M1-08 argues with escalation to the senior person, imports the reconciling-items schedule as the ladder, and defaults the reference to the blessed computation

**Decision:** five calls made while drafting the reconciliation pattern. (a) The Position argues with the practice of handing a disagreement to the most senior person in the room, or to whoever has time, and moving on. It is taken at its strongest: the senior person may still choose the reference, and the page only forbids choosing before the conditions are listed and the ladder closes. The critique is sourced from peer-reviewed group-decision work (motivated numeracy, confidence matching, the hierarchy meta-analysis, the many-teams finance study) and from the experimentation literature's highest-paid-person's-opinion framing, and Sources & Stories says the transfer from laboratory tasks to a metrics meeting is the framework's own. (b) The difference ladder is the reconciling-items schedule from a bank reconciliation, imported from the 2025 Green Book and a state auditor's guide and said so; the rule that it closes to zero or the number is on hold is the accounting rule carried over unchanged. The lead on reconciliation testing in data migration produced no peer-reviewed source inside the ten-year window, so the audit sources carry that lineage and the page says so in Sources & Stories rather than in the body. (c) The reference defaults to the blessed computation named by M1-01, corrected if the ladder shows it holds the defect, and the independent total keeps its job as the M1-03 control total rather than becoming the reference; where no blessed computation exists, the reconciliation is where one gets named. (d) Zoom's 2020 participants-versus-users correction is used as the definition-word example beside Airbnb and Uber, cited from the company's own progress report plus the TechCrunch report that reproduces its statement, because it is the cleanest public case of one figure travelling under two words in one week. (e) The HiPPO sentence and Twyman's law are quoted from the experimentation book's preface, read from the publisher-hosted sample chapter, which upgrades that entry from chapter-title level; the revision is noted for REFERENCES.md.

**Alternatives:** argue with averaging or with taking the friendlier number, which no competent practitioner defends in print, so the Position would contest a strawman; make the independent total the reference, which would put the reference outside the warehouse and contradict M1-01; cite a vendor's reconciliation feature as the lineage, which would name a product and misplace the origin; note the missing migration-testing source in the body with a caveat, which the reader-not-literature-review rule forbids.

**Why:** a Position should contest the practice the reader actually follows, and at ten people a disagreement is settled by whoever is senior or free. Importing the accounting schedule whole gives the reader a procedure their accountant already recognizes. Defaulting to the blessed computation keeps the three integrity pages consistent about which number stands and which number checks it.

## 2026-09-16 — M1-05 argues with the single-owner key its sibling page installed, scores values by raw agreement and verdicts by kappa, and refuses to score disputed questions

**Decision:** four calls made while drafting the agreement-measurement pattern. (a) The Position argues with the single owner writing the reference answers, the practice M1-04 itself installs. The page keeps that owner and treats the owner's answers as a gauge with an unmeasured error: a second person answers the same twenty questions blind before the assistant is scored, the human-to-human rate is the ceiling the assistant is held to, and an assistant inside that rate's band gets no model work. (b) Values within tolerance are scored by raw percent agreement, because two numbers landing within a tenth of a percent by chance is not a real risk; kappa is reserved for pass-or-fail verdicts and is reported beside the raw rate, never alone. The fixed kappa bands the quality manuals grade against are not printed, because no source inside the ten-year window could be verified for them, and the question is left for the author. (c) The band is plus and minus the square root of p(1−p)/n, about two questions at twenty, and the floor of twelve agreements in twenty below which the questions rather than the people are the problem is likewise the framework's own; both are marked as such on the page and in the template. (d) A question the two people disagree on is not scored for or against the assistant. It routes to one line in the context file and a dated new row in the key, so the assistant is scored only on settled questions, and its misses there are the only errors that are its own. (e) Three calls from the author's review, folded in on 2026-09-17: the quality manuals' kappa bands are not used, because a percentage difference can be a delay, an exclusion, a calendar, or two different metrics landing near each other, and no band separates those; the logic is checked before any statistic, through a three-bullet conditions summary under every answer (what was excluded, which calendar, when the data was pulled) that an assistant can write and a person can verify; and the per-metric tolerance is set by a person as the reporting difference the operation can live with, before the study. Nobody is the better judge; the study is run for alignment, and its working signal is that important calls spend their time on actions rather than on whose number is right.
Expand Down
Loading
Loading