Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTENT_BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The Module column uses `1A` (definition quality) and `1B` (data integrity) for t
| M1-03 | 1B | Detecting plausible-but-wrong outputs | pattern | v0.1.0 | drafted | Twyman's law via Kohavi; Shen's taxonomy, where only syntax and schema errors stop a query; control totals from the audit manual; the Twitter and Facebook restatements | 1 session |
| M1-04 | 1B | Golden question sets for AI analytics validation | pattern | v0.1.0 | drafted | Half the gold answers in public text-to-SQL benchmarks are wrong; LinkedIn and Uber built hand-verified sets from their own logs; 80 answers from 1,000 identical runs | 1 session |
| M1-05 | 1B | Agreement measurement for analytics QA | pattern | v0.2.0 | drafted | Twenty-nine teams on one dataset returned odds ratios from 0.89 to 2.93; ISO 5725's repeatability and reproducibility as the lens; LinkedIn found 60 percent of its questions had more than one right answer | 1 session |
| M1-06 | 1B | Data quality SLOs and error budgets | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-06 | 1B | Data quality SLOs and error budgets | pattern | v0.2.0 | drafted | The reliability books say not to pick a target from current performance, and data teams import the mechanics without that rule; process capability as the comparison of the reader's requirement against the pipeline's limits; Airbnb's owner-committed landing times, Uber's tier defaults and Shopify's eight-hour default as the practice it argues with | 1 session |
| M1-07 | 1B | Schema drift detection and data contracts | pattern | v0.1.0 | drafted | Avro resolution as the mechanism behind the quiet change; GoCardless's six-month report on what a producer-side programme costs; the author's addition of the assistant's memory as a drift of its own | 1 session |
| M1-08 | 1B | The reconciliation protocol | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-09 | 1B | Measuring trust, not just accuracy | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
Expand Down
8 changes: 8 additions & 0 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

One entry per judgment call: decision, alternatives, why.

## 2026-09-18 — M1-06 argues with the imported site-reliability objective using the reliability books' own rule, counts bad days instead of nines, and imports process capability as the comparison

**Decision:** four calls made while drafting the data-quality-SLO pattern. (a) The Position argues with the site-reliability objective as data teams import it, the target set by the engineer from what the system achieves and expressed in nines, and it does so with the reliability literature's own words: the first book's rule not to pick a target based on current performance, and the workbook's rule that the product manager has to agree the threshold is good enough for users. The platform accounts (Uber's tier defaults, Shopify's eight-hour default, Airbnb's owner-committed landing times and its score that credits "a more realistic SLA") are used as the competent practice, never as failures. (b) The budget is a count of bad periods per quarter agreed with the reader, and the percentage is derived from it. The nines are rejected on scale: a daily table has about sixty-five readings a quarter, a tenth of a percent of that is not a morning, and the one first-party nine found sits inside a freshness definition counting rows within a partition. (c) The comparison of requirement to pipeline is process capability, with the one-sided index (Cpk for a single limit) as the register's arithmetic, read first as a gap in the reader's units; the control limits stay in the SPC control plan and are copied into the register, never edited there. (d) The incident-cost figures were traced and not repeated: the Gartner average is a self-estimate by 154 vendor-nominated reference customers, the IBM trillion ends at an undated infographic, and the vendor survey has no published questionnaire; the page says plainly that no figure survived.

**Alternatives:** argue with doing nothing, which no competent practitioner defends; argue with the data-observability purchase, which this audience mostly has not made; express the objective as a percentage with the count in parentheses, which puts the number nobody agreed to first; teach the two-sided Cp and Cpk, which doubles the arithmetic for requirements that are almost always one-sided.

**Why:** a Position should contest the best advice the reader will actually hear, and for this unit that is the reliability literature, which is good and which the reader's engineer has read. Using its own rule against the import keeps the argument one a competent practitioner could still contest on scale grounds. The count keeps the requirement in the reader's words, and the capability index is the one piece of arithmetic that lets a team with one chart say whether the requirement is meetable at all.

## 2026-09-16 — M1-05 argues with the single-owner key its sibling page installed, scores values by raw agreement and verdicts by kappa, and refuses to score disputed questions

**Decision:** four calls made while drafting the agreement-measurement pattern. (a) The Position argues with the single owner writing the reference answers, the practice M1-04 itself installs. The page keeps that owner and treats the owner's answers as a gauge with an unmeasured error: a second person answers the same twenty questions blind before the assistant is scored, the human-to-human rate is the ceiling the assistant is held to, and an assistant inside that rate's band gets no model work. (b) Values within tolerance are scored by raw percent agreement, because two numbers landing within a tenth of a percent by chance is not a real risk; kappa is reserved for pass-or-fail verdicts and is reported beside the raw rate, never alone. The fixed kappa bands the quality manuals grade against are not printed, because no source inside the ten-year window could be verified for them, and the question is left for the author. (c) The band is plus and minus the square root of p(1−p)/n, about two questions at twenty, and the floor of twelve agreements in twenty below which the questions rather than the people are the problem is likewise the framework's own; both are marked as such on the page and in the template. (d) A question the two people disagree on is not scored for or against the assistant. It routes to one line in the context file and a dated new row in the key, so the assistant is scored only on settled questions, and its misses there are the only errors that are its own. (e) Three calls from the author's review, folded in on 2026-09-17: the quality manuals' kappa bands are not used, because a percentage difference can be a delay, an exclusion, a calendar, or two different metrics landing near each other, and no band separates those; the logic is checked before any statistic, through a three-bullet conditions summary under every answer (what was excluded, which calendar, when the data was pulled) that an assistant can write and a person can verify; and the per-metric tolerance is set by a person as the reporting difference the operation can live with, before the study. Nobody is the better judge; the study is run for alignment, and its working signal is that important calls spend their time on actions rather than on whose number is right.
Expand Down
Loading
Loading