Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTENT_BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The Module column uses `1A` (definition quality) and `1B` (data integrity) for t
| M1-02 | 1B | Statistical process control for data pipelines | pattern | v0.1.0 | drafted | Shewhart's limits-from-history; Wheeler's XmR arithmetic; the observability category re-derives the same move | 1 session |
| M1-03 | 1B | Detecting plausible-but-wrong outputs | pattern | v0.1.0 | drafted | Twyman's law via Kohavi; Shen's taxonomy, where only syntax and schema errors stop a query; control totals from the audit manual; the Twitter and Facebook restatements | 1 session |
| M1-04 | 1B | Golden question sets for AI analytics validation | pattern | v0.1.0 | drafted | Half the gold answers in public text-to-SQL benchmarks are wrong; LinkedIn and Uber built hand-verified sets from their own logs; 80 answers from 1,000 identical runs | 1 session |
| M1-05 | 1B | Agreement measurement for analytics QA | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-05 | 1B | Agreement measurement for analytics QA | pattern | v0.2.0 | drafted | Twenty-nine teams on one dataset returned odds ratios from 0.89 to 2.93; ISO 5725's repeatability and reproducibility as the lens; LinkedIn found 60 percent of its questions had more than one right answer | 1 session |
| M1-06 | 1B | Data quality SLOs and error budgets | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
| M1-07 | 1B | Schema drift detection and data contracts | pattern | v0.1.0 | drafted | Avro resolution as the mechanism behind the quiet change; GoCardless's six-month report on what a producer-side programme costs; the author's addition of the assistant's memory as a drift of its own | 1 session |
| M1-08 | 1B | The reconciliation protocol | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session |
Expand Down
8 changes: 8 additions & 0 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

One entry per judgment call: decision, alternatives, why.

## 2026-09-16 — M1-05 argues with the single-owner key its sibling page installed, scores values by raw agreement and verdicts by kappa, and refuses to score disputed questions

**Decision:** four calls made while drafting the agreement-measurement pattern. (a) The Position argues with the single owner writing the reference answers, the practice M1-04 itself installs. The page keeps that owner and treats the owner's answers as a gauge with an unmeasured error: a second person answers the same twenty questions blind before the assistant is scored, the human-to-human rate is the ceiling the assistant is held to, and an assistant inside that rate's band gets no model work. (b) Values within tolerance are scored by raw percent agreement, because two numbers landing within a tenth of a percent by chance is not a real risk; kappa is reserved for pass-or-fail verdicts and is reported beside the raw rate, never alone. The fixed kappa bands the quality manuals grade against are not printed, because no source inside the ten-year window could be verified for them, and the question is left for the author. (c) The band is plus and minus the square root of p(1−p)/n, about two questions at twenty, and the floor of twelve agreements in twenty below which the questions rather than the people are the problem is likewise the framework's own; both are marked as such on the page and in the template. (d) A question the two people disagree on is not scored for or against the assistant. It routes to one line in the context file and a dated new row in the key, so the assistant is scored only on settled questions, and its misses there are the only errors that are its own. (e) Three calls from the author's review, folded in on 2026-09-17: the quality manuals' kappa bands are not used, because a percentage difference can be a delay, an exclusion, a calendar, or two different metrics landing near each other, and no band separates those; the logic is checked before any statistic, through a three-bullet conditions summary under every answer (what was excluded, which calendar, when the data was pulled) that an assistant can write and a person can verify; and the per-metric tolerance is set by a person as the reporting difference the operation can live with, before the study. Nobody is the better judge; the study is run for alignment, and its working signal is that important calls spend their time on actions rather than on whose number is right.

**Alternatives:** argue with the model-as-judge, which this audience is not yet using and which the page handles as a failure mode; use kappa for every comparison, which collapses under the skewed pass rates a golden set produces; quote a formal confidence interval, which at twenty questions is four questions wide and tells the reader nothing; score the disputed questions against whichever reading the owner wrote, which is the practice the page exists to stop.

**Why:** a Position should contest the practice the reader actually follows, and a reader of this library has the single owner because the previous page told them to. The arithmetic choices keep the whole study inside one spreadsheet. Refusing to score disputed questions is what makes the routing to definitions rather than to the model mechanical instead of a judgment call.

## 2026-09-16 — The entry point is one raw markdown file, routed by the pages' own problem lists

**Decision:** three calls made while building the pasted-line entry. (a) The entry point is `llms.txt` at the repository root, fetched through the raw GitHub URL, with no server of the framework's own. GitHub already returns plain text to a fetcher and HTML to a browser, so the content negotiation a product page would need comes for free. (b) The routing table restates each page's "When this applies" list in the reader's words, and CI fails when a finished page has no row, so the entry file cannot drift from the library. (c) The instructions to the assistant are short and begin with a question to the person rather than with an execution contract. The reader may not be technical, and the first job is to find out which number they are trying to trust; the page does the rest.
Expand Down
Loading
Loading