From 9b3f28151dfb4d72468da0a6e4af53c2203498f8 Mon Sep 17 00:00:00 2001 From: Heqing Huang Date: Wed, 16 Sep 2026 17:05:43 +0800 Subject: [PATCH] docs: add the agreement measurement pattern, with its control-plan template An assistant scored 70 percent against one person's reference answers, and the team is about to spend a week on prompts. Nobody asked whether two of their own people would agree on those twenty questions. In the published studies where independent teams analysed the same data, they did not, and inside a company the cause is nearly always an exclusion, a calendar, or the time the data was pulled. This page has a second person answer the twenty questions blind with the conditions under each answer, checks the conditions before the values, holds the assistant to the rate at which the two people agree, and sends every disputed question to the definitions instead of the model. Nineteen references added. --- CONTENT_BACKLOG.md | 2 +- DECISIONS.md | 8 + REFERENCES.md | 25 ++- docs/modules/01-ai-data-quality/README.md | 2 +- .../agreement-measurement.md | 212 ++++++++++++++++++ llms.txt | 1 + .../agreement-measurement-control-plan.md | 179 +++++++++++++++ templates/README.md | 2 +- 8 files changed, 424 insertions(+), 7 deletions(-) create mode 100644 docs/modules/01-ai-data-quality/agreement-measurement.md create mode 100644 templates/01-ai-data-quality/agreement-measurement-control-plan.md diff --git a/CONTENT_BACKLOG.md b/CONTENT_BACKLOG.md index 1583d49..72f774d 100644 --- a/CONTENT_BACKLOG.md +++ b/CONTENT_BACKLOG.md @@ -17,7 +17,7 @@ The Module column uses `1A` (definition quality) and `1B` (data integrity) for t | M1-02 | 1B | Statistical process control for data pipelines | pattern | v0.1.0 | drafted | Shewhart's limits-from-history; Wheeler's XmR arithmetic; the observability category re-derives the same move | 1 session | | M1-03 | 1B | Detecting plausible-but-wrong outputs | pattern | v0.1.0 | drafted | Twyman's law via Kohavi; Shen's taxonomy, where only syntax and schema errors stop a query; control totals from the audit manual; the Twitter and Facebook restatements | 1 session | | M1-04 | 1B | Golden question sets for AI analytics validation | pattern | v0.1.0 | drafted | Half the gold answers in public text-to-SQL benchmarks are wrong; LinkedIn and Uber built hand-verified sets from their own logs; 80 answers from 1,000 identical runs | 1 session | -| M1-05 | 1B | Agreement measurement for analytics QA | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session | +| M1-05 | 1B | Agreement measurement for analytics QA | pattern | v0.2.0 | drafted | Twenty-nine teams on one dataset returned odds ratios from 0.89 to 2.93; ISO 5725's repeatability and reproducibility as the lens; LinkedIn found 60 percent of its questions had more than one right answer | 1 session | | M1-06 | 1B | Data quality SLOs and error budgets | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session | | M1-07 | 1B | Schema drift detection and data contracts | pattern | v0.1.0 | drafted | Avro resolution as the mechanism behind the quiet change; GoCardless's six-month report on what a producer-side programme costs; the author's addition of the assistant's memory as a drift of its own | 1 session | | M1-08 | 1B | The reconciliation protocol | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session | diff --git a/DECISIONS.md b/DECISIONS.md index 417b329..5572e0c 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -2,6 +2,14 @@ One entry per judgment call: decision, alternatives, why. +## 2026-09-16 — M1-05 argues with the single-owner key its sibling page installed, scores values by raw agreement and verdicts by kappa, and refuses to score disputed questions + +**Decision:** four calls made while drafting the agreement-measurement pattern. (a) The Position argues with the single owner writing the reference answers, the practice M1-04 itself installs. The page keeps that owner and treats the owner's answers as a gauge with an unmeasured error: a second person answers the same twenty questions blind before the assistant is scored, the human-to-human rate is the ceiling the assistant is held to, and an assistant inside that rate's band gets no model work. (b) Values within tolerance are scored by raw percent agreement, because two numbers landing within a tenth of a percent by chance is not a real risk; kappa is reserved for pass-or-fail verdicts and is reported beside the raw rate, never alone. The fixed kappa bands the quality manuals grade against are not printed, because no source inside the ten-year window could be verified for them, and the question is left for the author. (c) The band is plus and minus the square root of p(1−p)/n, about two questions at twenty, and the floor of twelve agreements in twenty below which the questions rather than the people are the problem is likewise the framework's own; both are marked as such on the page and in the template. (d) A question the two people disagree on is not scored for or against the assistant. It routes to one line in the context file and a dated new row in the key, so the assistant is scored only on settled questions, and its misses there are the only errors that are its own. (e) Three calls from the author's review, folded in on 2026-09-17: the quality manuals' kappa bands are not used, because a percentage difference can be a delay, an exclusion, a calendar, or two different metrics landing near each other, and no band separates those; the logic is checked before any statistic, through a three-bullet conditions summary under every answer (what was excluded, which calendar, when the data was pulled) that an assistant can write and a person can verify; and the per-metric tolerance is set by a person as the reporting difference the operation can live with, before the study. Nobody is the better judge; the study is run for alignment, and its working signal is that important calls spend their time on actions rather than on whose number is right. + +**Alternatives:** argue with the model-as-judge, which this audience is not yet using and which the page handles as a failure mode; use kappa for every comparison, which collapses under the skewed pass rates a golden set produces; quote a formal confidence interval, which at twenty questions is four questions wide and tells the reader nothing; score the disputed questions against whichever reading the owner wrote, which is the practice the page exists to stop. + +**Why:** a Position should contest the practice the reader actually follows, and a reader of this library has the single owner because the previous page told them to. The arithmetic choices keep the whole study inside one spreadsheet. Refusing to score disputed questions is what makes the routing to definitions rather than to the model mechanical instead of a judgment call. + ## 2026-09-16 — The entry point is one raw markdown file, routed by the pages' own problem lists **Decision:** three calls made while building the pasted-line entry. (a) The entry point is `llms.txt` at the repository root, fetched through the raw GitHub URL, with no server of the framework's own. GitHub already returns plain text to a fetcher and HTML to a browser, so the content negotiation a product page would need comes for free. (b) The routing table restates each page's "When this applies" list in the reader's words, and CI fails when a finished page has no row, so the entry file cannot drift from the library. (c) The instructions to the assistant are short and begin with a question to the person rather than with an execution contract. The reader may not be technical, and the first job is to find out which number they are trying to trust; the page does the rest. diff --git a/REFERENCES.md b/REFERENCES.md index 4274db7..885ae0b 100644 --- a/REFERENCES.md +++ b/REFERENCES.md @@ -12,11 +12,14 @@ Bibliography for the framework. Every citation key used anywhere in `docs/`, `re _The full Phase 0 seeding is pending; entries below were verified individually as units were synthesized._ +- **[ACZEL-2021]** — Aczel, Balazs; Szaszi, Barnabas; Nilsonne, Gustav; van den Akker, Olmo R.; et al. "Consensus-based guidance for conducting and reporting multi-analyst studies." _eLife_ 10, e72185, November 9, 2021. doi:10.7554/eLife.72185. Read in full from the publisher. A 54-author consensus paper produced by a reactive-Delphi panel of methodologists, defining a multi-analyst study as one that engages multiple analysts to independently analyze the same data. Verified verbatim: "Co-analysts should not communicate with each other about their analyses until after all initial reports have been submitted", and the recommendation to state the research question "without communicating the lead team's preferred analysis choices or expectations about the conclusions". Cited for the blinding rule; the guidance is written for research studies, and the framework carries it to a two-person study inside a company. - **[AGENTSMD]** — AGENTS.md. "A simple, open format for guiding coding agents." agents.md, fetched August 26, 2026. Open format stewarded by the Agentic AI Foundation under the Linux Foundation; the page describes it as "a README for agents: a dedicated, predictable place to provide the context and instructions" and confirms there are no required fields, just standard Markdown. The key carries no year because the page is an undated living document. The adoption figure ("used by over 60k open-source projects") is self-reported with no methodology; attribute it, never assert it. The format targets coding agents; carrying business context in it is this framework's extension, not documented practice. - **[ATIL-2024]** — Atil, Berk; Aykent, Sarp; Chittams, Alexa; Fu, Lisheng; Passonneau, Rebecca J.; et al. "Non-Determinism of 'Deterministic' LLM Settings." arXiv:2408.04667, submitted August 6, 2024, read at abstract level (v5, April 2, 2025). Five language models configured for deterministic output, eight tasks, ten runs each: "accuracy variations up to 15% across naturally occurring runs", a best-to-worst gap of up to 70 percent, and no model that "consistently delivers repeatable accuracy across all tasks, much less identical output strings". Cited for the size of run-to-run variation at settings meant to be deterministic; the models are 2024-era and the per-task figures are in the unread PDF. +- **[AUST-PONS-2022]** — Aust, Jonas; Pons, Dirk (University of Canterbury). "Assessment of Aircraft Engine Blade Inspection Performance Using Attribute Agreement Analysis." _Safety_ 8(2), article 23, March 29, 2022. doi:10.3390/safety8020023. Read at abstract level through the Crossref record, because the publisher refuses automated fetching. Verified: photographs of 26 blades were shown twice, in random order, to 50 industry practitioners of three skill levels, 2,600 observations, analysed with attribute agreement analysis and kappa; "operators were on average 82.5% consistent with their serviceability decision, while achieving an inspection accuracy of 67.7%"; and "the operators' reproducibility was 15.4%, as was the accuracy of all operators with the ground truth". Cited as a peer-reviewed application of attribute agreement analysis and for the within-person figure. The domain is visual inspection, and the 15.4 percent is all fifty operators agreeing at once, which is not the two-judge measure the framework uses. - **[AVRO-SPEC]** — The Apache Software Foundation. "Apache Avro Specification," version 1.12.0. avro.apache.org, undated page, fetched September 11, 2026. The key carries no year because the page is versioned rather than dated; the same rules were spot-checked against the 1.11.1 spec and the wording is unchanged. Cited for the Schema Resolution section, which makes the structural-versus-semantic asymmetry mechanical: fields are "matched by name" rather than position; "if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored"; "if the reader's record schema has a field that contains a default value, and writer's schema does not have a field with the same name, then the reader should use the default value from its field"; and "if the reader's record schema has a field with no default value, and writer's schema does not have a field with the same name, an error is signalled". Enum resolution behaves the same way: an unknown writer symbol falls back to the reader's default if one exists, and otherwise errors. - **[BALFOUR-2013]** — Balfour, Brian. "The Never Ending Road To Product Market Fit." brianbalfour.com, December 11, 2013. The flattening-curve reading of product-market fit, with his own caveat that flattening only shows fit for some market and that the next job is segmenting the curve to identify who retained. Distinct from [BALFOUR-2018]. - **[BALFOUR-2018]** — Balfour, Brian; Winters, Casey; Kwok, Kevin; Chen, Andrew. "Growth Loops are the New Funnels." Reforge, 2018. Existence verified via the authors' own announcements; the page blocked direct fetching at verification time (2026-08-04), so cite the thesis, and [TODO: verify] any direct quote against the live page. +- **[BAVARESCO-2025]** — Bavaresco, Anna; Bernardi, Raffaella; Bertolazzi, Leonardo; Elliott, Desmond; et al. "LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks." ACL 2025, main conference; arXiv:2406.18403, submitted June 26, 2024, read at abstract level (v3, June 2, 2025). The JUDGE-BENCH study: "20 NLP datasets with human annotations" and "11 current LLMs, covering both open-weight and proprietary models". Verified: "Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated", and "LLMs should be carefully validated against human judgments before being used as evaluators". Cited for the validation rule; the per-dataset correlations are in the unread PDF. Twenty authors in all. - **[BEREZOVSKY-2022]** — Berezovsky, Olga. "How to measure cohort retention." Lenny's Newsletter, August 30, 2022. N-day against unbounded retention, and the cohort-table layout; stresses that the two give completely different retention data. Possible partial paywall. - **[BEREZOVSKY-2024]** — Berezovsky, Olga. "Measuring Non-Cohorted Retention or Blended Churn." Data Analysis Journal #188, February 21, 2024. Framing citable; article body is paywalled — [TODO: verify] specifics against a subscription copy before quoting. - **[BEREZOVSKY-2025]** — Berezovsky, Olga. "How to Report on DAU/MAU Ratio." Data Analysis Journal #261, June 4, 2025. Free excerpt verified (scope: ratio misconceptions, reporting mistakes, benchmark caveats); full text paywalled — cite excerpt-level claims only, [TODO: verify] deeper claims against a subscription copy. @@ -27,10 +30,13 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[BETTER-2020]** — Better Mortgage. Closing-time claims disclosure page. First-party company marketing claim, not independent measurement: average closing time from January 2019 to May 2020 stated as 32 days against a 42-day industry average attributed to a third-party origination report. Use only as a labelled company claim. - **[BEYER-2016]** — Beyer, Betsy; Jones, Chris; Petoff, Jennifer; Murphy, Niall Richard (eds.). _Site Reliability Engineering: How Google Runs Production Systems_. O'Reilly Media, 2016. Cited for chapter 6, "Monitoring Distributed Systems," written by Rob Ewaschuk. Verbatim and load-bearing: "We avoid 'magic' systems that try to learn thresholds or automatically detect causality," alongside the rule that the alerting rules catching real incidents "should be as simple, predictable, and reliable as possible." The chapter is freely readable on Google's own sre.google site, whose page metadata shows 2017 in places; the print edition is 2016. Context caveat: the chapter argues from serving systems with load-testable capacities, not data pipelines; cite its position, never a claim about pipelines. - **[BHADAURIA-2026]** — Bhadauria, Divya (Hasso Plattner Institute, University of Potsdam); Harmouch, Hazar (University of Amsterdam); Naumann, Felix (HPI); Srivastava, Divesh (AT&T Chief Data Office); Ehrlinger, Lisa (HPI). "A Catalog of Data Errors." arXiv:2604.09277v1 [cs.DB], submitted April 10, 2026. Preprint with no stated venue; read from the PDF. Catalogs 35 error types across three non-overlapping categories, missing, incorrect and redundant. Cited for the disguised missing value, defined as a value where "𝑡.𝑎.𝑣 = 𝑀(⊥) ∧ 𝑡.𝑎.𝑣 ∈ 𝑑𝑜𝑚(𝑎)", in words a value that "falls within the domain and appears syntactically valid, but whose true real-world counterpart is a missing value". Load-bearing caveat in the paper's own words: "their detection can be challenging for automated approaches because they often fall within the defined domain or expected context of the attribute, thus appearing valid". Preprint status means the taxonomy may still change; cite the definition, not the count. +- **[BHASKAR-2023]** — Bhaskar, Adithya; Tomar, Tushar; Sathe, Ashutosh; Sarawagi, Sunita. "Benchmarking and Improving Text-to-SQL Generation under Ambiguity." EMNLP 2023, main conference; arXiv:2310.13659, submitted October 20, 2023, read at abstract level (v1). Verified: existing benchmarks assume "each text query corresponds to one correct SQL", and the AmbiQT benchmark holds over 3,000 examples whose questions can be read as two plausible SQL queries "due to lexical and/or structural ambiguity". Cited for the one-correct-answer assumption that benchmark scoring rests on. - **[BIGEYE-2021]** — Li, Henry (Bigeye). "Anomaly detection part 1: The key to effective data observability." Bigeye blog, October 20, 2021. Vendor material from a data-observability company selling adaptive thresholds; the institutional key flags the conflict. Verified claims: "Thresholds need to adapt to changes in the business behavior that appear as seasonality, trends, and pattern changes," plus the warning that alert overload makes the safety net stop working. Cite as the category's own description of adaptive thresholds, never as evidence they work. Not the Jinyang Li of [LI-2023]. - **[BITOL-ODCS]** — Bitol. "Open Data Contract Standard (ODCS)," version 3.2.0. bitol.io, site footer dated December 7, 2025. The key carries no year because the standard is a living versioned document. An incubation project of Linux Foundation AI & Data, Apache 2.0 licensed, and vendor-neutral by governance rather than by claim. Carries the definition that "A data contract is an agreement between a data producer and its consumers", covering the data, its expected behavior, data quality rules, service levels, stakeholders, roles and pricing; expressed in YAML so it "can, therefore, easily be versioned and governed". The standard separates a logical type from a physical type, which is the distinction this framework's semantic checks rest on. Grew out of PayPal's published data-contract template, a lineage asserted on PayPal's repository rather than on Bitol's own pages. - **[BLAKE-2015]** — Blake, Thomas; Nosko, Chris; Tadelis, Steven. "Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment." _Econometrica_ 83(1), 2015. Also NBER WP 20171. Bibliography verified across independent listings; full text paywalled — cite the findings via Tadelis's first-person account in [FREAKONOMICS-2020]. +- **[BOTVINIK-NEZER-2020]** — Botvinik-Nezer, Rotem; Holzmeister, Felix; Camerer, Colin F.; et al. "Variability in the analysis of a single neuroimaging dataset by many teams." _Nature_ 582, pp. 84–88, June 2020. doi:10.1038/s41586-020-2314-9. Abstract verified through Europe PMC because the publisher gates automated fetching. Verified: 70 independent teams analysed the same dataset against the same nine hypotheses; "no two teams chose identical workflows to analyse the data"; this produced "sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline"; and "prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset". Cited for the workflow finding and for the overestimation; the domain is brain imaging. - **[BRECK-2019]** — Breck, Eric; Polyzotis, Neoklis; Roy, Sudip; Whang, Steven Euijong; Zinkevich, Martin. "Data Validation for Machine Learning." Proceedings of the 2nd SysML Conference, Palo Alto, 2019. Google Research, with Whang at KAIST and the work done at Google. Peer-reviewed and first-party; read in full from the conference PDF. Load-bearing figures are Table 1, an analysis over the most recent thirty-day period across the >700 evaluation pipelines the paper considers: a new feature column present in the data but not in the schema fired on 10 percent of pipelines, out-of-domain values for categorical features on 6 percent, a missing feature column on 6 percent, and an incorrect data type for feature values on under 1 percent. The paper states that "it is very rare that the physical type of a feature column is wrong". Its motivating example "reflects an actual data-related production outage in Google", in which a seemingly innocuous code refactoring "pins the value of a specific int feature to -1 for some slice of the serving data". Also carries the schema-as-production-asset practice, "in our pipelines the schema is stored in the version-control system for source code", and the inversion this framework leans on, that "the schema can be inferred from and co-evolve with the data". Deployment scale: hundreds of product teams and several petabytes of training and serving data per day. Context caveat: these are machine-learning pipelines, not warehouse metric pipelines, so transfer arguments must be made rather than assumed. +- **[BREZNAU-2022]** — Breznau, Nate; Rinke, Eike Mark; Wuttke, Alexander; et al. "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty." _Proceedings of the National Academy of Sciences_ 119(44), e2203150119, November 1, 2022. doi:10.1073/pnas.2203150119. Abstract verified through Europe PMC because the publisher refuses automated fetching. Verified: "161 researchers in 73 research teams" independently tested the same hypothesis on the same data; "Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes"; and "More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow". Cited for the unexplained-variance finding; the setting is social-science survey data. - **[CHANG-2021]** — Chang, Robert (Airbnb). "How Airbnb achieved metric consistency at scale." The Airbnb Tech Blog on Medium, May 1, 2021. Part one of the Minerva series; the post's by-line credits twelve platform contributors with Chang as author of record. Read in full. Carries the trust story: when the CEO asked which city had the most bookings in the previous week, Data Science and Finance would sometimes provide diverging answers using slightly different tables, metric definitions, and business logic, and over time even data scientists second-guessed their own data. Minerva's stated vision is "define metrics once, use them everywhere"; definitions live in a centralized repository with code review, static validation, and test runs; definition changes are version-controlled and automatically backfilled; A/B tests source their base events from the same definitions. Scale figures: more than 12,000 metrics, 4,000 dimensions, and 200+ data producers. Caveats: a first-party success account with no failure numbers, and direct fetching is blocked (HTTP 403), so verification was against the rendered page. - **[CHANG-2021B]** — Chang, Robert (guest). "Ep 1: Robert Chang on Scaling Metrics @ Airbnb." The Analytics Engineering Podcast, show notes published July 2, 2021 by David Krevitt on dbt Labs' Roundup newsletter. The suffix distinguishes it from [CHANG-2021]. Quotes verified against the published show-note excerpts: a metrics store is "a single source of truth metric repository" where users "define metrics and dimensions once, and then they can use it everywhere"; Minerva began mid-2017 with one engineer not working on it full time and was recognized company-wide as the source of truth by 2020. Load-bearing small-company advice: tackle table standardization first, since "just having tables standardization actually takes you a long way," and expect metric standardization "as the company and as the organization becomes more and more mature"; he explicitly does not claim Airbnb's success transfers easily. Caveats: the host publication is a transformation-tool vendor with a commercial stake in the metrics-layer idea, and the quotes are the show notes' transcript excerpts rather than an independent transcript. - **[CHAPELLE-2014]** — Chapelle, Olivier. "Modeling delayed feedback in display advertising." KDD 2014, pp. 1097–1105. doi:10.1145/2623330.2623634. Bibliography verified across independent listings; no abstract was shown on the fetched record, so cite at paper level only. @@ -42,12 +48,14 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[COLDEWEY-2016]** — Coldewey, Devin. "Facebook miscalculation significantly inflated average video view times for years." TechCrunch, September 22, 2016. Secondary source, cited because the primary, a post to Facebook's advertiser help page, is no longer retrievable. Reproduces the company's statement verbatim: the metric was defined as "total time spent watching a video divided by the total number of people who have played the video" but "erroneously" calculated as "the total time spent watching a video divided by only the number of people who have viewed a video for three or more seconds." The 60 to 80 percent overstatement is a Publicis Media estimate reported by the Wall Street Journal, which first reported the error; it is not a Facebook figure. The error stood for about two years. - **[CONFLUENT-COMPAT]** — Confluent. "Schema Evolution and Compatibility for Schema Registry on Confluent Platform." docs.confluent.io, undated continuously-updated documentation, fetched September 11, 2026; the key carries no year for that reason. Vendor documentation, but the normative definition of compatibility modes, which are specified nowhere else: BACKWARD means "consumers using new schema can read data written with old schema", FORWARD means "consumers using old schema can read data written with new schema", and "The default compatibility mode is BACKWARD." Load-bearing for the coordination cost a registry imposes: under BACKWARD you "upgrade all consumers before you start producing new events", and under FORWARD you "first upgrade all producers to using the new schema and make sure the data already produced using the older schemas are not available to consumers, then upgrade the consumers". Cite for the semantics and the upgrade ordering, never as evidence that buying a registry is the right move. - **[CONVOYS]** — Bernhardsson, Erik (at Better). _convoys_: statistical models for time-lagged conversions. MIT licence. Software rather than a publication, so the key carries no year. The repository was verified; its documentation site was fetch-blocked at verification time. +- **[CUI-ALEXANDER-2026]** — Cui, Jiaxin; Alexander, Rohan. "Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs." arXiv:2602.14349, submitted February 15, 2026, read at abstract level (v1). Verified: two prompting strategies, six models, four temperature settings, ten independent executions per configuration, "480 total attempts"; "considerable variation in the analytical results even for consistent configurations"; and the recommendation that "if an LLM is being used to conduct data analysis, then it should be run multiple times independently and the distribution of results considered". A preprint, not peer reviewed; the per-configuration figures are in the unread PDF. - **[CUMMINGS-2017]** — Cummings, David. "Daily Active Users in B2B SaaS." davidcummings.org, May 23, 2017. The post does not restate the author's role (Pardot co-founder, per public knowledge); it does not treat per-seat vs. per-account counting. - **[DATABRICKS-2026]** — Databricks. "Test and monitor a Genie Agent." Databricks documentation, last updated August 10, 2026, fetched September 6, 2026. Product documentation for a paid feature, cited for what an execution-comparison benchmark looks like in a shipped product and never for how well it works. Verified: "Each Genie Agent can contain up to 500 benchmark questions"; a response is rated correct when its result set exactly matches the stored SQL answer's, differs only in sort order, or holds numerics that "round to the same 4 significant digits"; comparison covers "up to 5,000 rows of each result set"; questions without a SQL answer are marked "Manual review needed"; "Benchmark questions run as new conversations", separate from the example queries used to instruct the agent; and users with edit permission "can run a benchmark evaluation at any time". No cost information appears on the page. The tolerance conventions are one product's choices, not an industry standard. - **[DAVIDSONPILON-2019]** — Davidson-Pilon, Cameron. "lifelines: survival analysis in Python." _Journal of Open Source Software_ 4(40):1317, 2019. doi:10.21105/joss.01317. Open access, CC-BY-4.0. The censoring argument and the survival-function definition used in the metrics library come from the documentation intro page rather than the JOSS paper, and that page does not define the Kaplan-Meier estimator itself. and - **[DBT-CONTRACTS]** — dbt Labs. "Model contracts." dbt Developer Hub documentation, page states last updated September 10, 2026. The key carries no year because the page is a living document. Vendor documentation from a transformation-tool company; cited for what the tool category can enforce, never as a product recommendation. A contract is "a set of upfront 'guarantees' that define the shape of your model"; when enforced it "must include every column's `name` and `data_type`"; at build time "dbt will run a 'preflight' check to ensure that the model's query will return a set of columns with names and data types matching the ones you have defined", and a mismatch means the model "will fail to build". Breaking changes detected against a previous project state include removing an existing column and changing an existing column's `data_type`. Two caveats the docs state themselves: views support column names and data types "but not `constraints`", and several platform constraints are "Definable and not enforced", so "a model can still build even if building the model violates the constraint". - **[DBT-DATA-TESTS]** — dbt Labs. "Add data tests to your DAG." dbt Developer Hub, page last updated September 4, 2026; fetched September 6, 2026. Product documentation for the dominant open-source transformation tool in its category; the institutional key flags the interest. Verified: data tests "are `select` statements that seek to grab 'failing' records, ones that disprove your assertion," passing when zero rows return; the four generic tests are `unique`, `not_null`, `accepted_values`, and `relationships`; singular tests are one-off SQL assertions. The page documents no test that compares a computed metric to an expected total or reconciles to an external number, which is the negative claim the pattern rests on. The key carries no year because the page is a living document. - **[DBT-UNIT-TESTS]** — dbt Labs. "Unit tests." dbt Developer Hub, page last updated September 3, 2026; fetched September 6, 2026. Product documentation, same interest as [DBT-DATA-TESTS]. Verified: unit tests "validate your SQL modeling logic on a small set of static inputs before you materialize your full model in production," with mock inputs (`given`) and expected output (`expect`); available from dbt v1.8. Cited for what a unit test checks, the author's logic against inputs the author supplied, and not for a judgment about the feature. The key carries no year because the page is a living document. +- **[DELGADO-TIBAU-2019]** — Delgado, Rosario; Tibau, Xavier-Andoni. "Why Cohen's Kappa should be avoided as performance measure in classification." _PLoS ONE_ 14(9), e0222916, September 26, 2019. doi:10.1371/journal.pone.0222916. Read from the publisher. Verified: kappa "penalizes judges with similar marginals compared with judges with different ones", the paper constructs cases in which "a worse classifier gets higher Kappa score", and the Matthews correlation coefficient is proposed instead because it decreases monotonically as classification worsens. Cited for the direction of kappa's misbehaviour; the setting is classifier evaluation, and the framework applies the point to two human judges. - **[DEMING-FUNNEL]** — The W. Edwards Deming Institute. "The Funnel Experiment." deming.org, fetched August 31, 2026. The key carries no year because the page is an undated living document. Carries the definition of tampering as process-worsening adjustment and the finding that adjusting a stable process to compensate for an undesirable result increases variation. The page does not name which of Deming's books describes the experiment, so the citation stops at the institute's account; [TODO: verify] against _Out of the Crisis_ before attributing the experiment to a specific book. - **[DICKEY-2017]** — Dickey, Megan Rose. "Uber says it accidentally underpaid drivers in New York City." TechCrunch, May 23, 2017. Secondary source; Uber's statement was given to the press and first reported by the Wall Street Journal, and no first-party post has been located. Verified: Uber's 2014 driver agreement set the commission at 25 percent of the fare after taxes and fees, but in New York the commission was calculated on the full fare before them, for about two and a half years; the error was found while implementing "a nationwide change to how drivers see their earnings"; the company's quoted commitment is to pay "every driver every penny they are owed — plus interest." The "tens of millions of dollars" total is the company's estimate as reported; no per-driver figure is in the article. - **[DORA-2026]** — DORA. "DORA's software delivery performance metrics." Google Cloud, last updated January 5, 2026. The change lead time definition and the speed-and-stability finding. DORA is a program run by Google Cloud; the metric definitions are its research output rather than product claims. Free availability of the underlying State of DevOps reports was not confirmed. @@ -77,6 +85,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[GEORGE-2002]** — George, Michael L. _Lean Six Sigma: Combining Six Sigma Quality with Lean Production Speed_. McGraw-Hill, 2002. ISBN 9780071501903. Bibliography verified only; the Little's Law and process-cycle-efficiency formulations attributed to it come from secondary summaries and are unverified. Cite at thesis level, and check against a copy before quoting. - **[GOODHART-1975]** — Goodhart, Charles. "Problems of Monetary Management: The UK Experience." In _Papers in Monetary Economics_, Reserve Bank of Australia, 1975. Citation verified via independent bibliographic sources; original not freely available. Quote only the 1975 sentence, and credit the popular "measure becomes a target" phrasing to [STRATHERN-1997]. - **[GORDON-2019]** — Gordon, Brett R.; Zettelmeyer, Florian; Bhargava, Neha; Chapsky, Dan. "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook." _Marketing Science_ 38(2), 2019. Verified at abstract/DOI level; full text paywalled. doi:10.1287/mksc.2018.1135. +- **[GOULD-2025]** — Gould, Elliot; Fraser, Hannah S.; Parker, Timothy H.; Nakagawa, Shinichi; et al. "Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology." _BMC Biology_ 23, article 35, February 6, 2025. doi:10.1186/s12915-024-02101-x. Abstract verified through an indexed copy because the publisher gates automated fetching; the journal record verified through Crossref and Europe PMC. Verified: "174 analyst teams, comprising 246 analysts", two unpublished datasets, 141 and 85 usable effects; "substantial heterogeneity among results for both datasets"; for the first dataset "near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction"; and "no strong relationship" between peer-review ratings of the analyses, variable choice or random-effects structure and deviation from the meta-analytic mean. Cited for the peer-rating finding. The index record carries the 2020 registered-report date; the published article is 2025. - **[GUSTAFSON-2023]** — Gustafson, Erin. "Meaningful metrics: How data sharpened the focus of product teams." Duolingo Blog, February 17, 2023. - **[HANDY-2017]** — Handy, Tristan. "The Startup Founder's Guide to Analytics." 2017; first published on thinkgrowth.org (Medium), read at the dbt Labs republication, which is dated March 30, 2017. A consultant's stage model by employee count, recommending the first full-time analytics lead at 20 to 50 employees, stated from work with "over a dozen companies"; no survey or dataset behind it, and the author does not claim one. The author later ran dbt Labs, an analytics tooling vendor. The widely repeated hire-at-20-to-50 threshold traces here and no further. - **[HE-2025]** — He, Horace; with others at Thinking Machines Lab. "Defeating Nondeterminism in LLM Inference." Thinking Machines Lab blog, September 10, 2025. Verified: 1,000 completions from one open model (Qwen3-235B-A22B-Instruct-2507) at temperature 0 produced "80 unique completions, with the most common of these occuring 78 times", identical through the 102nd token and first diverging at the 103rd. The stated primary cause is that inference kernels are not batch-invariant, so an answer depends on the server load the request landed in, on top of floating-point non-associativity. The fix, batch-invariant kernels released as thinking-machines-lab/batch-invariant-ops, requires owning the inference stack, which a team on a hosted API does not. The author's employer is an AI lab; cited for the measurement. @@ -86,8 +95,10 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[HIGHALPHA-2025]** — High Alpha. "2025 SaaS Benchmarks Report." Ninth annual edition, 2025; read in full at the published report PDF, reached from the landing page. A survey of more than 800 private software companies (69 percent US, half of respondents CEOs or founders), run by a venture studio with more than 40 venture and platform partners. States its definition: software gross margin is subscription revenue less cost of goods sold divided by subscription revenue, as of Q2 2025. Median software gross margin by ARR band: 74 percent under $1M, 77 percent at $1-5M, 80 percent at $5-20M, 78 percent at $20-50M, 79 percent above $50M; upper quartiles run 80 to 86 percent and lower quartiles 60 to 71 percent. Caveats: respondents are self-selected, figures are self-reported, and the publisher is an investor in the category it benchmarks. - **[HSU-2015]** — Hsu, Jonathan. "Diligence at Social Capital Part 1: Accounting for User Growth." Medium (The Startup), September 22, 2015. Growth-accounting identities (MAU as new + retained + resurrected; prior MAU as retained + churned) and the quick ratio with rough ranges. - **[HUANG-2024]** — Huang, Renee (Snowflake). "Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI." Snowflake engineering blog, August 29, 2024. Vendor material by a product manager for the product being scored, so the conflict is total. Cited for two things only: that a model reported above 90 percent on public leaderboards measured "51%" on the vendor's internal set of "150 questions that mirror the real-world tasks business users encounter", and the evaluation design, in which human evaluators "manually select all the correct SQL queries to add to our set of acceptable gold queries" because "one question can have multiple correct answers". The product's own "90%+" figure is measured on that same private set and is not used. +- **[HUNTINGTON-KLEIN-2021]** — Huntington-Klein, Nick; Arenas, Andreu; Beam, Emily; Bertoni, Marco; et al. "The influence of hidden researcher decisions in applied microeconomics." _Economic Inquiry_ 59(3), pp. 944–960, 2021 (online March 22, 2021). doi:10.1111/ecin.12992. Abstract verified from the open discussion-paper version (IZA DP No. 13233, May 2020) because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: two published causal results, each re-derived from raw data by seven replicators; "No two replicators reported the same sample size"; "Statistical significance varied across replications, and for one of the studies the effect's sign varied as well"; and the spread of estimates across replications was three to four times the typical reported standard error. Cited for the sample-size finding, the closest published analogue to two people choosing different filters. Twelve authors in all. - **[HUSAIN-2024]** — Husain, Hamel. "Your AI Product Needs Evals." hamel.dev, March 29, 2024. A consultant's essay from building evaluation for a real-estate software client. Three levels: unit-test assertions run on every code change, human and model evaluation over logged traces, and A/B tests. Verified sentences: "Rechat has hundreds of these unit tests. We continuously update them based on new failures we observe in the data"; for model-based judging, "track the correlation between model-based and human evaluation"; and "You can never stop looking at data". Cited for the unit-test framing and for growing the set from observed failures. The author sells evaluation consulting and courses. - **[INSIDESALES-2007]** — Oldroyd, James B., for InsideSales.com. "Lead Response Management" study, 2007. Vendor-funded: the sponsor sold response-time software. Two hosted PDF copies were retrieved but could not be parsed, so every figure attributed to this study is unverified, and no independent replication was found. Cite the existence and the funding of the study, never its numbers. +- **[ISO-2023]** — International Organization for Standardization, Technical Committee ISO/TC 69. _ISO 5725-1:2023, Accuracy (trueness and precision) of measurement methods and results — Part 1: General principles and definitions._ Second edition, July 2023; it replaces the 1994 edition. Read from the publisher's preview pages (front matter, introduction and the clause 3 definitions); the full standard is paid, and the ISO site refuses automated fetching. Verified verbatim from the introduction: "Two conditions of precision, termed repeatability and reproducibility conditions, have been found necessary"; "Thus repeatability and reproducibility are the two extremes of precision, the first describing the minimum and the second the maximum variability in results"; and "comparing test results from two batches of product will not indicate a fundamental quality difference if the difference between them can be attributed to the inherent variation in the measurement procedure". Clause 3.14 defines repeatability conditions as the same method, items, facility, operator and equipment "within short intervals of time", and clause 3.19 defines reproducibility conditions as the same method and items "in different test or measurement facilities with different operators using different equipment". The standard covers continuous measurements; carrying the vocabulary to reference answers is the framework's extension. - **[JIN-2026]** — Jin, Tengjun; Choi, Yoojin; Zhu, Yuxuan; Kang, Daniel. "Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards." arXiv:2601.08778, submitted January 13, 2026, read at abstract level (v3, January 19, 2026). Verified: "BIRD Mini-Dev and Spider 2.0-Snow have error rates of 52.8% and 62.8%, respectively"; re-scoring 16 open-source agents on corrected data gave "performance changes range from -7% to 31% (in relative terms) and rank changes range from −9 to +9 positions"; and rankings on the uncorrected subset track the full leaderboard closely while the corrected subset tracks it weakly. A conference version, "Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors" (CIDR 2026), prints different figures on its abstract page: 66.1 percent for Spider 2.0-Snow, five methods re-evaluated, changes of −3 to 31 percent, and up to three rank positions. Cite the preprint's figures and name the version. - **[JIN-CHEN-2018]** — Jin, Li; Chen, Andrew. "The Power User Curve: The best way to understand your most engaged users." a16z / andrewchen.com, August 2018 (on-page day not independently confirmed; year solid). DAU/MAU shortcomings and the L30 histogram. (also ) - **[JONES-2021]** — Jones, Andrew (GoCardless). "Improving Data Quality with Data Contracts." Tech @ GoCardless on Medium, December 24, 2021. First-party account by the engineer who introduced the practice at the company. Read in full. Carries the diagnosis: critical data streamed out of service databases by an in-house change-data-capture service forces every consumer onto an internal data model that "has not been designed for consumption", "lacks documentation", and "will change as the service evolves, causing unexpected breaking changes for the data consumers"; his verdict is "Ultimately, we believe this data is of poor quality." Load-bearing for the practice this framework argues with: catalogs and anomaly detection "do nothing to improve the quality of your data. To do that, you need to go to the source of the data and how it is generated." Direct fetching is blocked (HTTP 403), so verification was against the rendered page. The author's own site carries a stub dated 28 December 2021; the Medium original's date is the one recorded here. @@ -103,12 +114,11 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[KENT-2021]** — Kent, Brian Patrick. "Conversion rate modeling: worth the effort?" crosstab.io, January 9, 2021. - **[KERR-1975]** — Kerr, Steven. "On the Folly of Rewarding A, While Hoping for B." _Academy of Management Journal_ 18(4), 1975. Citation verified via independent bibliographic sources; full text paywalled. doi:10.2307/255378. - **[KHUNE-2024]** — Khune, Abhi; Busch, Callie; Johnson, Jeffrey; Chakka, Pradeep. "QueryGPT – Natural Language to SQL Using Generative AI." Uber Engineering Blog, September 19, 2024. First-party account of an internal natural-language-to-SQL service: naive similarity search over schema and SQL samples produced "declining accuracy in the generated queries" as more tables were onboarded, and was replaced by curated per-domain workspaces plus an intent-classification step. Cite as an existence proof about retrieval degradation, not as a template for small teams. Verified negative: the widely circulated hour- and dollar-savings figures attributed to this system do not appear in the post; do not attribute them to it. -- **[KHUNE-2024]** — Khune, Abhi; Busch, Callie; Johnson, Jeffrey; Chakka, Pradeep. "QueryGPT – Natural Language to SQL Using Generative AI." Uber Engineering Blog, September 19, 2024. First-party account of an internal natural-language-to-SQL service: naive similarity search over schema and SQL samples produced "declining accuracy in the generated queries" as more tables were onboarded, and was replaced by curated per-domain workspaces plus an intent-classification step. Cite as an existence proof about retrieval degradation, not as a template for small teams. Verified negative: the widely circulated hour- and dollar-savings figures attributed to this system do not appear in the post; do not attribute them to it. - **[KNIGHT-LEVESON-1986]** — Knight, John C.; Leveson, Nancy G. "An Experimental Evaluation of the Assumption of Independence in Multiversion Programming." _IEEE Transactions on Software Engineering_ 12(1), 1986, pp. 96–109. Cited at abstract level, read in a course-hosted copy. Verified from the abstract: 27 versions of a program written independently from one specification at two universities and run against one million tests; "the programs were individually extremely reliable but that the number of tests in which more than one program failed was substantially more than expected" under independence. Cited for the caveat that two computations share mistakes more often than chance predicts, so a second path must start from different material; it is a software-reliability experiment on student programs, not an analytics study, and it predates the framework's recency preference by design, because it is the canonical result. - **[KOHAVI-2007]** — Kohavi, Ron; Henne, Randal M.; Sommerfield, Dan. "Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO." KDD 2007, author's copy. - **[KOHAVI-2010]** — Kohavi, Ron; Longbotham, Roger. "Unexpected Results in Online Controlled Experiments." SIGKDD Explorations 12(2), 2010. - **[KOHAVI-2017]** — Kohavi, Ron. "Twyman's Law and Controlled Experiments." Slide deck, written November 5, 2013, updated May 7, 2017, with a companion LinkedIn article of the same date. Verified: the law as "Any figure that looks interesting or different is usually wrong"; the earliest scholarly reference Kohavi found is A. S. C. Ehrenberg, "The Teaching of Statistics," _Journal of the Royal Statistical Society_ Series A 138(4), 1975, which quotes the law by name; the attribution is to W. A. (Tony) Twyman, a British radio and television audience-measurement researcher who, per Kohavi, never published it himself; a 1996 quotations dictionary lists the origin as unknown. The deck's examples include default-value artifacts (a birth date of 11/11/11; a departure city of Adak Island from an alphabetical default), a daylight-saving hour of zero traffic, and a newspaper stock table with eleven identical zero returns, corrected three days later. Cite for the law and its provenance; the examples are anecdotes from the author's experience. and -- **[KOHAVI-2020]** — add after "Cited at book level": Chapter 3 is titled "Twyman's Law and Experimentation Trustworthiness" and Chapter 21 "Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics," both verified from the Cambridge University Press chapter listings on September 6, 2026; the pattern cites the book at chapter-title level only, and no page-level claim is made. +- **[KOHAVI-2020]** — Kohavi, Ron; Tang, Diane; Xu, Ya. _Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing_. Cambridge University Press, 2020. Cited at book level; check page-level claims against a copy. Chapter 3 is titled "Twyman's Law and Experimentation Trustworthiness" and Chapter 21 "Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics," both verified from the Cambridge University Press chapter listings on September 6, 2026; the pattern cites the book at chapter-title level only, and no page-level claim is made. - **[KRAMER-2023]** — Kramer, Emily. "How to measure what marketing activities are actually driving revenue." MKT1 Newsletter, September 20, 2023. Partially paywalled — cite free-section claims only. - **[LEI-2025]** — Lei, Fangyu; Chen, Jixuan; Ye, Yuxiao; Cao, Ruisheng; et al. "Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows." ICLR 2025. Read from the benchmark's project site and leaderboard, fetched September 6, 2026; the paper itself was not read. Verified from the site: 632 workflow problems over databases "often containing over 1,000 columns"; GPT-4o at "10.1% on Spider 2.0 tasks, compared to 86.6% on Spider 1.0", and o1-preview at 17.1 percent. On the fetch date the Snow split (547 examples) was topped by self-submitted commercial agents at 96.70, 96.53 and 94.15 percent with no linked papers, the Lite split (547 examples) at 76.23 percent, and the DBT split (68 examples) at 65.6 percent. A leaderboard entry is a vendor's own submission of a purpose-built agent with undisclosed per-database tuning; cite the launch figures as the paper's and the leaderboard rows as self-reported, never as measured capability. The leaderboard moves, so any citation carries the access date. - **[LEI-LEXICON]** — Lean Enterprise Institute. Lean Lexicon terms: cycle time and takt time. The key deviates from the AUTHOR-YYYY convention because the pages carry no publication year and no edition marker. The lexicon's lead-time term URL returned 404; the lead-time ordering quoted in the metrics library appears on the cycle-time page itself. and @@ -129,10 +139,12 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[MAZAL-2023]** — Mazal, Jorge. "How Duolingo reignited user growth." Lenny's Newsletter, February 28, 2023. - **[MCCLURE-2007]** — McClure, Dave. "Startup Metrics for Pirates" (AARRR), long-version deck. First presented 2007 (date per secondary sources). - **[MEHTA-2025]** — Mehta, Ravi. "Confidence engineering: Why your onboarding is probably too short." Ravi on Product, December 18, 2025. Free. First-party numbers for a checkout expanded from 3 steps to 25 steps, plus the decision-consequence and information-gap frame. His exact role at the company described is not stated on the page, so describe him as a product leader who worked on it. +- **[MENKVELD-2024]** — Menkveld, Albert J.; Dreber, Anna; Holzmeister, Felix; et al. "Nonstandard Errors." _The Journal of Finance_ 79(3), pp. 2339–2390, June 2024 (online April 17, 2024). doi:10.1111/jofi.13337. Abstract verified through an indexed copy because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: nonstandard errors are defined as the uncertainty that variation in the evidence-generating process across researchers adds to a result, studied "by letting 164 teams test the same hypotheses on the same data"; "NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants." Cited for the underestimation finding. - **[META-2024]** — Meta Platforms, Inc. 2023 Form 10-K, filed February 2, 2024 (accession 0001326801-24-000012). Daily active people (DAP) definition, the ML-estimate caveat for cross-app people counting, the duplicate/false-account discussion, and the announced retirement of per-app DAU, MAU, ARPU, and MAP from periodic reports. - **[MODIG-2012]** — Modig, Niklas; Åhlström, Pär. _This is Lean: Resolving the Efficiency Paradox_. Rheologica Publishing, 2012. ISBN 9789198039306. Both authors at the Stockholm School of Economics. Bibliography verified; the flow-versus-resource-efficiency content is unverified at page level, so cite the thesis rather than passages. No flow-efficiency benchmark number outside vendor blogs survived verification. - **[MOHAMMED-2008]** — Mohammed, M. A.; Worthington, P.; Woodall, W. H. "Plotting basic control charts: tutorial notes for healthcare practitioners." _Quality and Safety in Health Care_ 17(2), 2008, pp. 137–145. doi:10.1136/qshc.2004.012047. Read in full from a hosted copy. Peer-reviewed source for the XmR arithmetic: limits at the mean ± 2.66 × mean moving range, with 2.66 = 3/1.128 and the range limit at 3.267 × mean moving range; the roughly 1-in-370 false-alarm rate of three-sigma limits under ideal conditions; the guidance that 20–25 points firm the limits up while fewer give "soft" provisional ones worth using; the three widely recommended run rules; and the warnings that day-of-week patterns and seasonality need preliminary work before charting. The examples are healthcare, and every formula is general. - **[MONTECARLO-2024]** — MacDonald, Lindsay. "Data Quality Anomaly Detection: Everything You Need to Know." Monte Carlo blog, updated June 20, 2024. Vendor material from a data-observability company; the institutional key flags the conflict. Describes automatically applied freshness, volume, and schema monitors "without the need for any threshold setting," with baselines learned from each table's history. The post does not discuss false positives or alert fatigue, which is the cost side of what it sells; cite it for what the category does, never for effectiveness. The URL redirected from montecarlodata.com at fetch time. +- **[MONTGOMERY-2019]** — Montgomery, Douglas C. _Introduction to Statistical Quality Control_, 8th edition. Wiley, August 2019. ISBN 978-1-119-39930-8. Cited at book level for Chapter 8, "Process and Measurement System Capability Analysis", whose title was verified from the publisher's table of contents; the chapter's text was not read for this entry, so the citation rests on the chapter title and stands for the textbook lineage of gauge repeatability and reproducibility studies rather than for any figure. - **[OLDROYD-2011]** — Oldroyd, James B.; McElheran, Kristina; Elkington, David. "The Short Life of Online Sales Leads." _Harvard Business Review_, March 2011. Authorship, affiliations, date, and framing verified; the article body is paywalled and the widely repeated response-time multipliers could not be confirmed on the page. Co-author David Elkington was chairman and CEO of InsideSales.com, which funded the underlying study [INSIDESALES-2007]; disclose that at every use. - **[OPENAI-2026]** — OpenAI. "Assistants migration guide." OpenAI developer documentation, fetched August 26, 2026, the Assistants API shutdown date the page itself states. Verbatim and load-bearing: "We will not provide an automated tool for migrating Threads to Conversations." Cited as a dated first-party record that conversation objects held inside one vendor's platform had no automated export path at retirement. An undated living document, so the fetch date does the work; re-check before quoting again, since the page may change or disappear after the shutdown. - **[ORDONEZ-2009]** — Ordóñez, Lisa D.; Schweitzer, Maurice E.; Galinsky, Adam D.; Bazerman, Max H. "Goals Gone Wild: The Systematic Side Effects of Overprescribing Goal Setting." _Academy of Management Perspectives_ 23(1), 2009, pp. 6-16. Read in full at the open-access repository copy of the working-paper version, whose title hyphenates "Over-Prescribing". The standard caution read alongside [LOCKE-LATHAM-2002]: named side effects include "a narrow focus that neglects non-goal areas, a rise in unethical behavior, distorted risk preferences, corrosion of organizational culture, and reduced intrinsic motivation", and the authors argue goal setting should be treated as a prescription-strength medication rather than an over-the-counter one. An argument piece built on cases and prior studies, not a meta-analysis. @@ -147,6 +159,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[POLYZOTIS-2017]** — Polyzotis, Neoklis; Roy, Sudip; Whang, Steven Euijong; Zinkevich, Martin (Google Research). "Data Management Challenges in Production Machine Learning." SIGMOD '17, Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1723-1726. Peer-reviewed tutorial paper; read from an open copy of the PDF, with the page headers confirming pp. 1723-1724. Cited for one claim, which is the theoretical ground under this framework's position: "the schema describing the constraints may be a property of the pipeline but may not be part of the process generating the data, which means that the data and the schema can evolve separately. (Constrast this with traditional DBMS technology where the data always conforms to the schema.)" The misspelling of "Constrast" is the original's. Also carries the encoding-mismatch example, features that "may use different value encodings (e.g., using 0 and 1 for boolean values in the training data vs. '0' and '1' in the serving data)". - **[POLYZOTIS-2018]** — Polyzotis, Neoklis; Roy, Sudip; Whang, Steven Euijong; Zinkevich, Martin. "Data Lifecycle Challenges in Production Machine Learning: A Survey." _SIGMOD Record_ 47(2), June 2018, pp. 17-28. Google Research, with Whang at KAIST. Peer-reviewed survey, freely readable, read in full. Carries the semantic-drift mechanism this framework leans on hardest: "a feature may suddenly change its unit (e.g., age changes from days to hours) or even disappear. Unfortunately, model training is resilient to such errors, and instead of crashing, the model accuracy may simply decrease." Also the new-category case worked through in the original: a country feature known to hold four values gains "SS", and since "'SS' could also be a valid country (say South Sudan), which means the constraint should now include five countries", the right move may be to widen the constraint rather than reject the batch. And the case-normalization example, where "US" arriving as "us" leaves the model thinking "there is a new country". Same context caveat as [BRECK-2019]: machine-learning pipelines, not metric pipelines. - **[QU-2023]** — Qu, Hila (guest); Rachitsky, Lenny (host). "The ultimate guide to adding a PLG motion." Lenny's Podcast, April 2, 2023. Funnel auditing at 38:24, activation and conversion at 47:30. +- **[QUARFOOT-LEVINE-2016]** — Quarfoot, David; Levine, Richard A. "How Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution?" _The American Statistician_ 70(4), pp. 373–384, 2016 (issue dated October 2016, online November 21, 2016). doi:10.1080/00031305.2016.1141708. Abstract verified through an indexed copy because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: the article "exposes limitations of these indices when the units to be rated are not well-distributed across the rating categories", the indices being Fleiss's kappa, Conger's kappa and Krippendorff's alpha, and argues for the Brennan–Prediger coefficient and Gwet's AC2 because their agreement levels "are more robust to variation in the distribution of units that raters encounter". Cited for the instability of kappa-type indices under skew; the indices studied are the multi-rater forms, and the two-rater Cohen's kappa shares their dependence on the marginals. - **[RACHITSKY-2022]** — Rachitsky, Lenny; Timen, Yuriy. "What is a good activation rate." Lenny's Newsletter, October 25, 2022. - **[RACHITSKY-WINTERS-2020]** — Rachitsky, Lenny; Winters, Casey. "What is good retention." Lenny's Newsletter, June 9, 2020. Survey of roughly twenty growth practitioners; six-month retention benchmarks segmented by product category. Key includes both names to disambiguate from [RACHITSKY-2022] and credit the co-author. - **[REICHHELD-SASSER-1990]** — Reichheld, Frederick F.; Sasser, W. Earl, Jr. "Zero Defections: Quality Comes to Services." _Harvard Business Review_, September-October 1990. The root of the retention-profit folklore. Its prose states "companies can boost profits by almost 100% by retaining just 5% more of their customers", with industry cases of 85 percent (a bank's branch system), 50 percent (an insurance brokerage) and 30 percent (an auto-service chain); the analysis is Bain client work in service industries of that era, and the client data is unpublished. The widely circulated 25-to-95-percent range is not in this article's prose; it appears in [REICHHELD-SCHEFTER-2000]. Full text paywalled; authorship and opening verified on the HBR page, profit sentences verified against a verbatim-quotes reproduction. @@ -158,8 +171,10 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[RYDBERG-2020]** — Rydberg, Johan. "Spotify's New Experimentation Platform (Part 1)." Spotify Engineering, October 29, 2020. Cited for the Metrics Catalog: before it, experiment analyses ran manually in notebooks and there was no consistency across the company in how experiments were analyzed; the catalog made metric pipelines managed and served their results to UIs and notebooks from one place. Verified at article level. The consistency claim is about experiment analysis specifically, not company reporting; do not stretch it further. - **[SANDERSON-2024]** — Sanderson, Chad. "Implementing Data Contracts In The Data Warehouse." Monte Carlo blog, page states updated January 25, 2024. Double-flag: the author led the data-contract programme at Convoy, so it is a practitioner account, and it is hosted by a data-observability vendor with a commercial stake in warehouse-side monitoring, with product links and a closing call to action. Load-bearing concession, verified verbatim: "starting in the data warehouse doesn't require the help of application engineers or all data producers", and "Overall, it requires a smaller culture shift in order to succeed." His own recommendation remains both: "The best way to ensure data products can be effectively produced and maintained is to implement data contracts upstream at the service level as well as in the data warehouse." Cite the concession and the recommendation together; quoting the first without the second misrepresents him. - **[SANDERSON-KREUZIGER-2022]** — Sanderson, Chad; Kreuziger, Adrian (both Convoy). "An Engineer's Guide to Data Contracts - Pt. 1." Data Products newsletter on Substack, October 5, 2022. The byline carries both names; Sanderson introduces Kreuziger, a principal engineer at Convoy, as the author of the body, so the key names both. First-party implementation account, part one of three. Load-bearing verbatim: the stated requirement "Data contracts must be enforced at the producer level", with the sentence that follows it, "If there's nothing enforcing a contract on the producer side, you don't have a contract. That's a handshake agreement at best"; that enforcement "must be automated as part of the existing CI/CD pipeline"; that a meaning change is a breaking change, since "changing the underlying meaning of the data being produced should break the data contract", illustrated by an entity whose length and width switch "from being stored in inches to centimeters"; and the admission that "There are simply some aspects of semantic enforcement that cannot be reliably managed prior to deployment." +- **[SAPARINA-LAPATA-2024]** — Saparina, Irina; Lapata, Mirella. "AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries." NeurIPS 2024, Datasets and Benchmarks track, spotlight; arXiv:2406.19073, submitted June 27, 2024, read at abstract level (v2, October 31, 2024). Verified: the benchmark covers "scope ambiguity, attachment ambiguity, and vagueness", and "even the most advanced models struggle to identify and interpret ambiguity in questions". Cited for the finding that a model does not notice a question with two readings. - **[SCHELTER-2018]** — Schelter, Sebastian; Lange, Dustin; Schmidt, Philipp; Celikel, Meltem; Biessmann, Felix (Amazon Research); Grafberger, Andreas (University of Augsburg). "Automating Large-Scale Data Quality Verification." _PVLDB_ 11(12), 2018, pp. 1781-1794. doi:10.14778/3229863.3229867. Peer-reviewed and first-party; read from the VLDB PDF. Cited for the check vocabulary the framework's semantic half borrows: the vision that users "should be able to write 'unit-tests' for data ... analogous to established testing practices in software engineering"; Completeness as "the fraction of non-missing values in a column"; and "Consistency is defined as the degree to which a set of semantic rules are violated", where intra-relation constraints may span columns, the paper's own example being "if category is t-shirts, then the range of size is {S, M, L}". Also the syntactic-versus-semantic accuracy split, where for a product attribute colour "a value blue can be considered syntactically accurate in the given domain even if the correct value would be red". Constraints compile to aggregation queries over a distributed engine, which is why the same checks are cheap in a warehouse. - **[SCHULTZ-2014]** — Schultz, Alex. "Lecture 6: Growth." How to Start a Startup (Y Combinator / Stanford CS183B), 2014. +- **[SCHWEINSBERG-2021]** — Schweinsberg, Martin; Feldman, Michael; Staub, Nicola; et al. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis." _Organizational Behavior and Human Decision Processes_ 165, pp. 228–249, July 2021. doi:10.1016/j.obhdp.2021.02.003. Abstract verified through an indexed copy because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: independent analysts tested two hypotheses on one dataset with both the analytic approach and the operationalization of the variables left to them; 29 analyses survived quality screening; the analysts reported "radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question"; and a multiverse analysis "demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices". Cited for the operationalization finding, the closest published analogue to a metric definition. - **[SEC-2016]** — U.S. Securities and Exchange Commission, Division of Corporation Finance. Staff comment letters to Snap Inc. (Evan Spiegel), November 29, 2016 (accession 0000000000-16-102863) and December 23, 2016 (accession 0000000000-16-106388), re draft registration statement. Metric comments: the "quarterly average" DAU computed from the last month alone, averaging-window disclosure, and the third-party-to-internal analytics transition. and - **[SEC-2019]** — U.S. Securities and Exchange Commission, Division of Corporation Finance. Staff comment letter to Slack Technologies (Stewart Butterfield), May 7, 2019, re S-1 filed April 26, 2019 (accession 0000000000-19-008166). Comment 5: clarify that daily active users include paying and non-paying users. - **[SEGNER-2026]** — Segner, Michael. "Data Quality Statistics & Insights From Monitoring +11 Million Tables In 2026." Monte Carlo blog, page states updated June 1, 2026. Vendor material, and the institutional conflict is direct: the figures come from one observability product's own troubleshooting agent classifying incidents inside its customers' environments, so the denominator is what that product detects rather than what occurs. Verified verbatim: "Schema drift (7.8%) highlights the ongoing tension between agility and control"; "Intentional changes, including activities like data backfilling, comes in at 14.2%"; "Pipeline execution faults top the list at 26.2%". Cite the ordering of the categories, never the percentages as incidence rates. Note also that "+11 million tables" appears in the page title while the body says millions of tables. @@ -168,9 +183,9 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[SEUFERT-2017]** — Seufert, Eric Benjamin. "Mobile's post-attribution era." Mobile Dev Memo, May 15, 2017. - **[SEUFERT-2023]** — Seufert, Eric Benjamin. "The emerging marketing economist." Mobile Dev Memo, February 15, 2023. - **[SHANMUGAM-2020]** — Shanmugam, Santhosh; Pasqua, Andrea; Agrawal, Ritesh; Li, Ye Henry. "Monitoring Data Quality at Scale with Statistical Modeling." Uber Engineering Blog, May 7, 2020. First-party account of an internal data quality monitor that sets per-table thresholds from Holt-Winters one-step-ahead forecasts checked against prediction intervals, on the stated grounds that tables differ in level and seasonality and that metric-level anomalies are "too noisy for everyday use, and induce alert fatigue." Cite as an existence proof that history-derived limits replaced static thresholds at pipeline scale, not as a small-team recipe; the machinery described needs a data science team. -- **[SHEN-2025]** — Shen, Jiawei; Wan, Chengcheng; Qiao, Ruoyi; et al. "Understanding, Detecting, and Repairing Real-World ICL-Based Text-to-SQL Errors." arXiv:2501.09310; submitted January 16, 2025, read at v2 (July 1, 2025); a v3 (June 15, 2026) exists and was not re-read. Error taxonomy over 3,655 errors from in-context-learning text-to-SQL techniques: 389 of the errors caused execution failures, and within those, syntax errors are 18.8 percent and schema errors 81.2 percent; semantic errors are 36.1 percent (BIRD) and 17.7 percent (Spider) of all errors. The 81.2 figure is real but is a share of execution failures only; do not let it be quoted as a share of all failures, since most wrong queries execute. -- **[SHEN-2025]** — the paper has since been published: Shen, Jiawei; Wan, Chengcheng; Qiao, Ruoyi; Zou, J.; Xu, H.; Shao, Y.; Zhang, Y.; Miao, W.; Pu, Geguang. "Understanding, Detecting, and Repairing Real-World ICL-Based Text-to-SQL Errors." _Proceedings of the ACM on Software Engineering_ 3, FSE, Article FSE164, July 2026; arXiv v3 of June 15, 2026 read in full on September 6, 2026. The v3 study covers 4,602 incorrect queries from four in-context-learning systems (MAC-SQL, DIN-SQL, CHESS, DEA-SQL) on the Bird and Spider dev splits, with a taxonomy of 27 error types in 7 categories: syntax, schema, logic, convention, semantic, not-an-error, and others. Verified figures: 47.8 percent of 6,136 Bird queries with reliable judgments contain at least one true error, 26.8 percent on Spider; intent-independent errors (syntax, schema, logic, convention) are 26.0 percent of errors on Bird and 25.8 percent on Spider; semantic errors are 30.9 percent overall (36.1 percent Bird, 17.7 percent Spider); "Syntax errors constitute 18.8% of all errors that cause execution failures" and "Schema errors cause more than 80% execution failures"; and, in the background section, "While the compiler reports syntax errors, semantic errors are likely to escape." The earlier caveat stands and sharpens: the 18.8 and 81.2 figures are shares of execution failures, not of all errors, and v3 no longer prints the absolute count of execution failures that v2 gave as 389 of 3,655; do not quote that count against the v3 totals. Semantic-error types verified: incorrect table selection, projection error, improper condition, sub-query scope inconsistency, unaligned aggregation structure, wrong COUNT object, ORDER-BY error, missing DISTINCT, comparing wrong columns. Around 15.4 percent of queries have not-an-error problems, which the authors exclude. +- **[SHEN-2025]** — Shen, Jiawei; Wan, Chengcheng; Qiao, Ruoyi; et al. "Understanding, Detecting, and Repairing Real-World ICL-Based Text-to-SQL Errors." arXiv:2501.09310; submitted January 16, 2025, read at v2 (July 1, 2025); a v3 (June 15, 2026) exists and was not re-read. Error taxonomy over 3,655 errors from in-context-learning text-to-SQL techniques: 389 of the errors caused execution failures, and within those, syntax errors are 18.8 percent and schema errors 81.2 percent; semantic errors are 36.1 percent (BIRD) and 17.7 percent (Spider) of all errors. The 81.2 figure is real but is a share of execution failures only; do not let it be quoted as a share of all failures, since most wrong queries execute. The paper has since been published: Shen, Jiawei; Wan, Chengcheng; Qiao, Ruoyi; Zou, J.; Xu, H.; Shao, Y.; Zhang, Y.; Miao, W.; Pu, Geguang. "Understanding, Detecting, and Repairing Real-World ICL-Based Text-to-SQL Errors." _Proceedings of the ACM on Software Engineering_ 3, FSE, Article FSE164, July 2026; arXiv v3 of June 15, 2026 read in full on September 6, 2026. The v3 study covers 4,602 incorrect queries from four in-context-learning systems (MAC-SQL, DIN-SQL, CHESS, DEA-SQL) on the Bird and Spider dev splits, with a taxonomy of 27 error types in 7 categories: syntax, schema, logic, convention, semantic, not-an-error, and others. Verified figures: 47.8 percent of 6,136 Bird queries with reliable judgments contain at least one true error, 26.8 percent on Spider; intent-independent errors (syntax, schema, logic, convention) are 26.0 percent of errors on Bird and 25.8 percent on Spider; semantic errors are 30.9 percent overall (36.1 percent Bird, 17.7 percent Spider); "Syntax errors constitute 18.8% of all errors that cause execution failures" and "Schema errors cause more than 80% execution failures"; and, in the background section, "While the compiler reports syntax errors, semantic errors are likely to escape." The earlier caveat stands and sharpens: the 18.8 and 81.2 figures are shares of execution failures, not of all errors, and v3 no longer prints the absolute count of execution failures that v2 gave as 389 of 3,655; do not quote that count against the v3 totals. Semantic-error types verified: incorrect table selection, projection error, improper condition, sub-query scope inconsistency, unaligned aggregation structure, wrong COUNT object, ORDER-BY error, missing DISTINCT, comparing wrong columns. Around 15.4 percent of queries have not-an-error problems, which the authors exclude. - **[SHEWHART-1931]** — Shewhart, Walter A. _Economic Control of Quality of Manufactured Product_. D. Van Nostrand Company, 1931. Bibliography verified via WorldCat, HathiTrust, and an Internet Archive scan; the book was not read in this round, so cite at book level as the origin of control limits computed from the process's own behavior, and check any page-level claim against a copy. Wheeler's revision column [WHEELER-2012] works a dataset it attributes to page 21 of this book. +- **[SILBERZAHN-2018]** — Silberzahn, Raphael; Uhlmann, Eric L.; Martin, Daniel P.; et al. "Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results." _Advances in Methods and Practices in Psychological Science_ 1(3), pp. 337–356, August 23, 2018. doi:10.1177/2515245917747646. Abstract verified through an indexed copy because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: "Twenty-nine teams involving 61 analysts used the same data set to address the same research question"; effect sizes "ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units"; "Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship"; the analyses used "21 unique combinations of covariates"; "Neither analysts' prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses"; and "Peer ratings of the quality of the analyses also did not account for the variability". The original many-analysts study. - **[SINGH-BEDATHUR-2025]** — Singh, Rajat; Bedathur, Srikanta. "Can the Rookies Cut the Tough Cookie? Exploring the Use of LLMs for SQL Equivalence Checking." arXiv:2412.05561, submitted December 7, 2024, read at abstract level (v2, June 8, 2025). Introduces the SQLEquiQuest benchmark. Verified: language models reach "up to 82% accuracy on Spider+DIN" as equivalence checkers and extend coverage "from a mere 30% supported query pairs to full coverage" against formal methods, but show "a strong bias for equivalence predictions, with consistently poor performance over" non-equivalent pairs. Cited for the direction of the bias, which is toward passing wrong queries; the per-model figures on non-equivalent pairs are in the unread PDF. The affiliation (Indian Institute of Technology Delhi) comes from search listings, not the abstract page. - **[SIROKER-2010]** — Siroker, Dan. "How Obama Raised $60 Million by Running a Simple Experiment." November 29, 2010. First-party account by the campaign's Director of Analytics; hosted on the blog of Optimizely, the testing company Siroker later founded — cited as the story's primary source, not as a vendor reference. - **[SKOK-2013]** — Skok, David (with Gill, Ron; Coffey, Brad). "SaaS Metrics 2.0." forEntrepreneurs. @@ -207,4 +222,6 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[YAKUBENKOV-2019]** — Yakubenkov, Oleg. GoPractice, October 2, 2019. Classic Day-N retention against rolling or unbounded retention, and calendar days against rolling 24-hour windows. Hard number: for one app, day-1 retention is 32 percent on 24-hour windows, 43 percent on calendar days, or 59 percent on the rolling definition, from the same app, the same users and the same day. Also notes that rolling retention produces a curve that can never decrease, a definitional artifact that can masquerade as a smile. Caveat: GoPractice sells paid courses, so the author is commercially interested, though not an analytics vendor. [TODO: verify] the exact article title. - **[YU-2021]** — Yu, Will; Wu, Yun; Wang, Xiaodong; Meng, Wenrui (Uber). "The Journey Towards Metric Standardization." Uber Engineering Blog, January 12, 2021. The uMetric account, read in full. Verified stories and figures: the Shopping Session metric read 6.53M in one internal tool and 6.20M in another for the same city and time duration, root-caused to stale filters in one tool's static query; popular metrics spun off "10X or 100X" named instances; the standardization target is that "a metric and the business logic have a strictly ONE to ONE mapping without ambiguities during the entire lifecycle," built working with 30+ teams and 10K+ metrics. Also carries the enforcement machinery: SQL-equivalence deduplication via canonical transformation (with SQL equivalence noted as NP-complete, hence heuristic), organization-level Verification Committees the post itself calls time-consuming, tiered governance, and definitions kept lean by having consumers supply dimensions and filters at run time. Caveat: first-party platform advocacy; no adoption or accuracy outcomes are published. - **[ZHANG-2024]** — Zhang, Hugh; Da, Jeff; Lee, Dean; et al. (Scale AI). "A Careful Examination of Large Language Model Performance on Grade School Arithmetic." arXiv:2405.00332, submitted May 1, 2024, read at v4 (November 22, 2024): abstract, construction section, and first page. The GSM1k contamination audit: a new test set of 1,205 problems, built by fresh human annotation to mirror the established GSM8k benchmark despite the 1000 in the name; scoring existing models on it produced "accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting", while models at the frontier showed minimal signs of overfitting. Cited to show that a benchmark with published tasks can be audited from outside. The authors work at an AI evaluation vendor, so the audit is not disinterested; the finding cited here does not favor the vendor's products. +- **[ZHAO-2022]** — Zhao, Xinshu; Feng, Guangchao Charles; Ao, Song Harris; Liu, Piper Liping. "Interrater reliability estimators tested against true interrater reliabilities." _BMC Medical Research Methodology_ 22, article 232, 2022. Read from the PubMed Central copy. A simulation study that tests chance-corrected indices against known true reliabilities. Verified: "Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement"; in the tested conditions the kappa family (Scott's pi, Cohen's kappa and Krippendorff's alpha) "were crude predictors of reliability and evenness, and 31-point under-approximators of reliability"; and Gwet's AC1 was "the most accurate approximator". Cited for the under-estimation of kappa under skew. The framework's use of raw agreement for numeric values within a tolerance, where chance agreement is negligible, is its own reading and not this paper's. +- **[ZHENG-2023]** — Zheng, Lianmin; Chiang, Wei-Lin; Sheng, Ying; Zhuang, Siyuan; et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023, Datasets and Benchmarks track; arXiv:2306.05685, submitted June 9, 2023, read at abstract level (v4, December 24, 2023). Verified: "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans", and the paper names "position, verbosity, and self-enhancement biases". Cited for the human-to-human agreement level as the ceiling a model judge is measured against, and for the biases; the setting is chat-assistant responses rather than numeric answers. Thirteen authors in all. - **[ZHONG-2020]** — Zhong, Ruiqi; Yu, Tao; Klein, Dan. "Semantic Evaluation for Text-to-SQL with Distilled Test Suites." EMNLP 2020; arXiv:2010.02840, submitted October 6, 2020, read at abstract level. Verified: the Spider benchmark's exact-set-match metric "leads to a 2.5% false negative rate on average and 8.1% in the worst case", meaning correct queries marked wrong; the proposed alternative executes queries over a distilled suite of databases to bound semantic accuracy. The rates belong to exact-set match on academic benchmarks, measured with 21 leaderboard models and manually verified examples, not to execution comparison on a company warehouse. diff --git a/docs/modules/01-ai-data-quality/README.md b/docs/modules/01-ai-data-quality/README.md index 7475d17..5d042d3 100644 --- a/docs/modules/01-ai-data-quality/README.md +++ b/docs/modules/01-ai-data-quality/README.md @@ -43,7 +43,7 @@ Whether the number the pipeline returns is the number the definition asks for. R | M1-03 | [Detecting plausible-but-wrong outputs](plausible-but-wrong.md) | v0.1.0 | | M1-04 | [Golden question sets for AI analytics validation](golden-question-sets.md) | v0.1.0 | | M1-07 | [Schema drift detection and data contracts](schema-drift-and-data-contracts.md) | v0.1.0 | -| M1-05 | Agreement measurement for analytics QA | v0.2.0 | +| M1-05 | [Agreement measurement for analytics QA](agreement-measurement.md) | v0.2.0 | | M1-06 | Data quality SLOs and error budgets | v0.2.0 | | M1-08 | The reconciliation protocol | v0.2.0 | | M1-09 | Measuring trust, not just accuracy | v0.2.0 | diff --git a/docs/modules/01-ai-data-quality/agreement-measurement.md b/docs/modules/01-ai-data-quality/agreement-measurement.md new file mode 100644 index 0000000..274a98a --- /dev/null +++ b/docs/modules/01-ai-data-quality/agreement-measurement.md @@ -0,0 +1,212 @@ +--- +id: M1-05 +title: Agreement measurement for analytics QA +module: 1B +type: pattern +status: drafted # full sourced draft; awaiting author voice pass per AGENTS.md constraint 6 +sources: + - ACZEL-2021 + - AUST-PONS-2022 + - BAVARESCO-2025 + - BHASKAR-2023 + - BOTVINIK-NEZER-2020 + - BREZNAU-2022 + - CHANG-2021 + - CHEN-2024 + - CUI-ALEXANDER-2026 + - DELGADO-TIBAU-2019 + - GOULD-2025 + - HUANG-2024 + - HUNTINGTON-KLEIN-2021 + - HUSAIN-2024 + - ISO-2023 + - JIN-2026 + - MENKVELD-2024 + - MONTGOMERY-2019 + - QUARFOOT-LEVINE-2016 + - RUMIANTSAU-2026 + - SAPARINA-LAPATA-2024 + - SCHWEINSBERG-2021 + - SILBERZAHN-2018 + - WRETBLAD-2024 + - ZHAO-2022 + - ZHENG-2023 +template: ../../../templates/01-ai-data-quality/agreement-measurement-control-plan.md +summary: > + Before an AI assistant is scored against hand-checked answers, have a + second person answer the same twenty questions blind, each answer with + the conditions behind it: what was excluded, which calendar, and when + the data was pulled. Agreement is checked on the conditions first, then + on the value within a tolerance a person set. That rate is the ceiling + the assistant is held to. An assistant inside the band gets no model + work; the disputed questions go to the definitions and the context + file, and the assistant's misses on the settled questions are the only + errors that are its own. +keywords: + - agreement rate + - inter-rater agreement + - measurement system analysis + - gauge repeatability and reproducibility + - attribute agreement analysis + - Cohen's kappa + - many analysts + - reference answers + - evaluation ceiling + - text-to-SQL evaluation +--- + +# Agreement measurement for analytics QA + +## Problem + +Two competent people asked the same business question return different numbers more often than anyone expects. When 29 research teams were handed one dataset and one question, twenty found the effect and nine did not, and neither expertise nor peer ratings explained who landed where.[^silberzahn-2018] Seven economists re-deriving one published result from the same raw data did not produce the same sample size twice.[^huntington-klein-2021] + +Company data behaves the same way. At Airbnb, data science and finance would give the chief executive different answers to which city had the most bookings last week, from slightly different tables and definitions.[^chang-2021] Nobody was careless. Each person got a number that was right by their reading. + +An AI assistant scored against one person's answers inherits every one of those open choices as its own error, and the team reads a 70 percent score as a model problem and spends the week on prompts. I read the failure as a gauge that was never checked, and the rest of this page is how my field checks one. + +In my experience this is where most executive escalations start. Two departments are in a fast discussion, nobody has time to spell out how their number was made, and the numbers do not match. Laid side by side afterwards, the cause is nearly always one of three things. One side excluded records the other side counted. The two sides used different calendars, fiscal week against calendar week, or fiscal year against calendar year. Or one number was pulled two hours before the meeting and the live dashboard had moved by the time the meeting started. None of those is a wrong query, and an assistant fails in exactly the same three ways. People have to agree on the conditions first. + +## When this applies + +Reach for this pattern when a hand-checked answer is about to be used as the truth. The problems that bring teams here look like these: + +- The assistant scored 70 percent on the golden set, and the team cannot agree on whether that is bad. +- Two people gave the board two numbers for the same quarter, both queries look right, and the meeting is about which person to believe. +- The golden set has one owner, and somebody on the team quietly doubts a few of the reference answers. +- A model upgrade moved the score by five points, and nobody knows whether five points on twenty questions is a change or a coin toss. +- A week of prompt work is about to start because the score is low, and nobody has asked whether the score could be higher. + +It needs a second person who can answer the twenty questions from the data without help, and one afternoon. It needs no analyst and no purchase. + +Building the set and comparing answers as values live on the [golden question sets](golden-question-sets.md) page. Whether one number is right before it leaves the room lives on [detecting plausible-but-wrong outputs](plausible-but-wrong.md). Whether the rate moved or only wiggled between two runs is the question [statistical process control](statistical-process-control-for-pipelines.md) answers, and the rate can go on a control chart once it exists. + +Whenever another page in this library asks how repeatable a hand-checked answer is, this is the page it points to. + +## The pattern + +Before the assistant is scored, a second person answers the same twenty questions the owner already answered, without seeing the key or the owner's work. Each answer carries a short list of the conditions behind it: what was excluded, which calendar, and when the data was pulled. Agreement is checked in that order. Two answers agree when their conditions match and their values land within a tolerance a person set for that metric before the study. Two values that match by coincidence under different conditions do not count. Two that differ inside the tolerance under the same conditions do. The share of questions that agree is the agreement rate. That rate is the ceiling for the assistant, with a band, because twenty questions is a small sample. The assistant is then scored against each person's answers and read against the ceiling rather than against 100. A question the two people disagree on is not scored at all. It goes to the definitions and comes back once the definition is written down. + +In industrial engineering this is measurement system analysis. Repeatability is how close one person lands answering the same question twice, and reproducibility is how close two people land on it. The international standard states the rule this page rests on: a difference between two results is not a real difference when the measurement procedure's own variation can explain it.[^iso-2023] Gauge studies are the textbook form for measured values,[^montgomery-2019] and attribute agreement analysis, with Cohen's kappa as its two-judge arithmetic, is the form for pass-or-fail judgements.[^aust-pons-2022] + +```mermaid +flowchart LR + TWO["Two people answer
the same twenty, blind"] --> RATE["Agreement rate:
the ceiling and its band"] + TWO --> DIS["Disputed questions:
not scored, sent
to the definitions"] + RATE --> CMP{"Assistant inside
the band?"} + CMP -->|yes| DEF["The next hour goes to
definitions and
the context file"] + CMP -->|no| MOD["Context first,
then the model"] +``` + +A single person is not a fixed reference either. Fifty aircraft inspectors judging the same 26 blade photographs twice agreed with their own earlier decision 82.5 percent of the time and with the ground truth 67.7 percent of the time.[^aust-pons-2022] + +## Position + +Before you score the assistant, measure how often two of your own people agree on the same twenty questions, and hold the assistant to that rate as its ceiling, not to one person's answers with the model tuned until the score rises. + +The common practice is the single owner. One person writes the reference answers, the assistant is scored against them, and a low score is read as a model problem. The appeal is real, and the golden question set page installs exactly this owner. A test has one expected value,[^husain-2024] the owner usually knows the data best, and a second person costs an afternoon a ten-person company does not have. + +What I take from the evidence is that the owner's answers are a gauge whose error nobody has measured. A second competent person disagrees with the first more often than a score's precision assumes, and neither expertise nor peer-rated quality predicts who is right.[^silberzahn-2018] [^gould-2025] Inside a company the disagreement is about definitions, and definitions are what move an assistant.[^rumiantsau-2026] So when the assistant's agreement with the reference sits inside the human-to-human band, the next hour goes into the definitions and the context file, not into the model. + +The ceiling excuses nothing below it. The assistant's misses on the settled questions are its own, and they go through the usual order of context first and model second. The ceiling rises with every disagreement that resolves into a written definition. + +The quality manuals I trained on grade kappa against fixed bands, with 0.75 the usual line for good agreement. I do not use the bands here. A difference between two answers can be a delay, an exclusion, a calendar, or two different metrics that happened to land near each other, and no band tells those apart. Check the logic first. The tolerance comes after, and a person sets it for each metric as the reporting difference the operation can live with. + +## Implementation + +The [control plan template](../../../templates/01-ai-data-quality/agreement-measurement-control-plan.md) carries the answer sheets, the arithmetic in spreadsheet-ready form, what each rate means, a machine-readable core, and the prompts. The study is one afternoon and an hour of adding up. + +1. **Pick the second person and keep them blind.** Nobody is the better judge here. The point is that both people end up informed and aligned, so pick for independence: someone who can answer from the data without the owner's help, from outside the team that wrote the key. They see neither the key nor the owner's queries, and nobody discusses the questions until the second sheet is in.[^aczel-2021] +2. **Answer the twenty, and write the conditions under each answer.** The second person gets the question text exactly as the assistant would, and records the value, the query, and three bullets: what was excluded, which calendar, and when the data was pulled. An assistant writes those three bullets from its own query in a second, and a person verifies them in a few more. In the crowdsourced studies, operationalization explained more of the spread than statistics did.[^schweinsberg-2021] +3. **Check the logic, then compute the rate and its band.** Compare the conditions first. Two answers whose conditions differ are a disputed row whatever the values say. Two answers whose conditions match agree when the values land within the tolerance a person set for that metric, and the rate is agreements divided by twenty. The band is the rate plus and minus the square root of p(1−p)/n, about two questions either way at twenty, the framework's own rule of thumb. +4. **Score the assistant three ways.** Against each person, and on the settled questions. Percent agreement is the number for values. Kappa is for pass-or-fail verdicts, where chance agreement is large, and it is read beside the raw rate, because it falls when most answers pass.[^quarfoot-levine-2016] [^delgado-tibau-2019] [^zhao-2022] +5. **Read the result and route it.** An assistant inside the band gets no model work. Each disputed question becomes one line in the [context file](../03-ai-agent-integration/context-file.md) and a dated new row in the key, never an edit in place, as the companies running their own sets do.[^chen-2024] [^huang-2024] An assistant below the band is worked on the settled questions, context first and then the model. +6. **Repeat when the key changes, and chart the rate.** Put the rate on a control chart like the pipeline signals, and re-run the study after every change to the key. + +The rows below are illustrative. + +| Question | Two people agree | Assistant matches A | Assistant matches B | Reading | +|---|---|---|---|---| +| 1 | yes | yes | yes | settled, assistant right | +| 2 | yes | yes | yes | settled, assistant right | +| 3 | yes | yes | yes | settled, assistant right | +| 4 | no | yes | no | disputed: the day the quarter closes | +| 5 | yes | yes | yes | settled, assistant right | +| 6 | yes | no | no | settled, assistant wrong | +| 7 | yes | yes | yes | settled, assistant right | +| 8 | yes | yes | yes | settled, assistant right | +| 9 | no | yes | no | disputed: refunds netted or not | +| 10 | yes | yes | yes | settled, assistant right | +| 11 | yes | no | no | settled, assistant wrong | +| 12 | yes | yes | yes | settled, assistant right | +| 13 | no | no | yes | disputed: trial accounts counted as customers | +| 14 | yes | yes | yes | settled, assistant right | +| 15 | yes | no | no | settled, assistant wrong | +| 16 | yes | yes | yes | settled, assistant right | +| 17 | no | yes | no | disputed: the date of the currency conversion | +| 18 | yes | yes | yes | settled, assistant right | +| 19 | yes | no | no | settled, assistant wrong | +| 20 | no | no | yes | disputed: the time zone of the day boundary | +| **Total** | **15 of 20** | **14 of 20** | **13 of 20** | 4 wrong, 5 disputed | + +The two people agree on 15 of 20, a rate of 75 percent, and the band runs from about 13 to 17. The assistant matches A on 14 and B on 13, inside the band either way, and a team scoring against A alone would report 70 percent and start on the model. On the 15 settled questions the assistant is right on 11 and wrong on 4, and those four are its own errors; the other five the two people read differently. Had the two judges marked the assistant's answers pass or fail instead, their verdicts would agree on 15 of 20, an observed agreement of 0.75. With A passing 14 and B passing 13, chance alone gives 0.70 × 0.65 + 0.30 × 0.35 = 0.56, and kappa is (0.75 − 0.56) / (1 − 0.56) = 0.43. The figure reads as moderate for judges who agree three times in four, because the disputed questions are exactly where their verdicts part. Every one of the five disputed rows is a condition, not arithmetic: an exclusion, a calendar, or an as-of time. + +Run it more than once and it becomes a habit with less friction each time. The important calls then spend their time on actions, and less of it on aligning the numbers. + +## How you know it is working + +- The agreement rate is a number the team knows, with a date and two names beside it. +- The assistant's score is never reported without the human rate. +- The important calls spend their time on what to do, and the question of whose number is right has stopped coming up in them. +- Model work waits until the assistant is below the band on the settled questions. +- Every disputed question turns into a context-file line within the week, and the human rate rises on the next run. +- **Anti-signal:** two people agree on all twenty the first time. Either the second person saw the key, or the questions are too easy to catch anything. + +## Failure modes + +- **The second person is briefed by the first.** A walk-through of how the owner reads the questions turns two judges into one, and the study passes by construction.[^aczel-2021] +- **Scoring the disputed questions anyway.** Whichever reading the owner wrote wins by default. Benchmarks assume one correct query per question,[^bhaskar-2023] and models struggle even to notice a question with two readings,[^saparina-lapata-2024] so it is the one the assistant answers confidently either way. +- **Agreeing on the number and not on the conditions.** Two values can match by coincidence, one on the fiscal calendar and one on the calendar year, and split next quarter. Agreement is recorded on the conditions, and a matching value under different conditions is a disputed row. +- **Kappa alone on a skewed set.** When eighteen of twenty answers pass, chance agreement is high and kappa collapses between judges who agree on nearly everything.[^quarfoot-levine-2016] [^zhao-2022] +- **The assistant as the second judge.** Model judges agree with people about as often as people agree with each other, and they lean toward the first answer shown and the longer one.[^zheng-2023] Their agreement varies by task, and the advice is to validate them against people first.[^bavaresco-2025] A model can be the third rater once two people exist, never the second. +- **Copying a public benchmark's answers as your reference.** Two audits found errors in roughly half the reference answers they checked,[^jin-2026] [^wretblad-2024] and a reference nobody at your company verified has an error rate of its own. +- **Treating one answer from one person as that person's answer.** The inspectors disagreed with themselves one time in six,[^aust-pons-2022] and a model used as an analyst returned considerably different results across ten runs of one setup.[^cui-alexander-2026] Where a question matters, have each person answer it twice, a week apart. + +## Sources & Stories + +The many-analysts evidence is five studies in which independent teams analysed one dataset to answer one question: the referee study [^silberzahn-2018], the immigration study [^breznau-2022], the neuroimaging study [^botvinik-nezer-2020], the finance study that named the phenomenon nonstandard errors [^menkveld-2024], and the ecology study, the largest [^gould-2025]. The sample-size finding is from an economics replication with seven replicators per result [^huntington-klein-2021], the operationalization finding from an organizational-behaviour study [^schweinsberg-2021], and the blinding rule from the consensus guidance for multi-analyst studies [^aczel-2021]. Several of these publishers refuse automated fetching, so abstracts were verified through indexed copies and journal records through Crossref; the reference entries say which. + +The measurement-system lens is the author's own field, cited through the current international standard for measurement precision, read from the publisher's preview pages [^iso-2023], and the standard quality-control textbook, cited at book level for its measurement-system chapter [^montgomery-2019]. The inspector study is a peer-reviewed application of attribute agreement analysis, verified at abstract level [^aust-pons-2022]. The three critiques of kappa are peer-reviewed and cited for the direction of the problem [^quarfoot-levine-2016] [^delgado-tibau-2019] [^zhao-2022]. + +The Airbnb story is a first-party account, cited for the disagreement and never for the platform [^chang-2021]. The dated-row practice is first-party from LinkedIn [^chen-2024] and vendor material from Snowflake, cited only for the evaluation design [^huang-2024]. The two annotation audits are the ones the golden question set page cites [^jin-2026] [^wretblad-2024]. The ambiguity benchmarks [^bhaskar-2023] [^saparina-lapata-2024], the model-judge studies [^zheng-2023] [^bavaresco-2025] and the model-as-analyst preprint [^cui-alexander-2026] are verified at abstract level. The context-before-model ordering is the paired benchmark by authors at a semantic-layer vendor [^rumiantsau-2026], and the single-expected-value practice the Position argues with is Husain's [^husain-2024]. + +The three causes of a mismatch, the conditions summary under every answer, the rule to check the logic before any statistic, and the tolerance set by a person are the author's own, from his practice. The blind two-person study as the first step before scoring, the band of two questions at twenty, the rule that disputed questions are not scored, and the three-way score are the framework's own. No published account applies measurement system analysis to the reference answers of an AI evaluation set; the closest is the quarterly expert review of accepted answers at LinkedIn [^chen-2024]. The prior-art row for this pattern is pending in [prior-art.md](../../prior-art.md). + + + +[^aczel-2021]: [[ACZEL-2021]](../../../REFERENCES.md) +[^aust-pons-2022]: [[AUST-PONS-2022]](../../../REFERENCES.md) +[^bavaresco-2025]: [[BAVARESCO-2025]](../../../REFERENCES.md) +[^bhaskar-2023]: [[BHASKAR-2023]](../../../REFERENCES.md) +[^botvinik-nezer-2020]: [[BOTVINIK-NEZER-2020]](../../../REFERENCES.md) +[^breznau-2022]: [[BREZNAU-2022]](../../../REFERENCES.md) +[^chang-2021]: [[CHANG-2021]](../../../REFERENCES.md) +[^chen-2024]: [[CHEN-2024]](../../../REFERENCES.md) +[^cui-alexander-2026]: [[CUI-ALEXANDER-2026]](../../../REFERENCES.md) +[^delgado-tibau-2019]: [[DELGADO-TIBAU-2019]](../../../REFERENCES.md) +[^gould-2025]: [[GOULD-2025]](../../../REFERENCES.md) +[^huang-2024]: [[HUANG-2024]](../../../REFERENCES.md) +[^huntington-klein-2021]: [[HUNTINGTON-KLEIN-2021]](../../../REFERENCES.md) +[^husain-2024]: [[HUSAIN-2024]](../../../REFERENCES.md) +[^iso-2023]: [[ISO-2023]](../../../REFERENCES.md) +[^jin-2026]: [[JIN-2026]](../../../REFERENCES.md) +[^menkveld-2024]: [[MENKVELD-2024]](../../../REFERENCES.md) +[^montgomery-2019]: [[MONTGOMERY-2019]](../../../REFERENCES.md) +[^quarfoot-levine-2016]: [[QUARFOOT-LEVINE-2016]](../../../REFERENCES.md) +[^rumiantsau-2026]: [[RUMIANTSAU-2026]](../../../REFERENCES.md) +[^saparina-lapata-2024]: [[SAPARINA-LAPATA-2024]](../../../REFERENCES.md) +[^schweinsberg-2021]: [[SCHWEINSBERG-2021]](../../../REFERENCES.md) +[^silberzahn-2018]: [[SILBERZAHN-2018]](../../../REFERENCES.md) +[^wretblad-2024]: [[WRETBLAD-2024]](../../../REFERENCES.md) +[^zhao-2022]: [[ZHAO-2022]](../../../REFERENCES.md) +[^zheng-2023]: [[ZHENG-2023]](../../../REFERENCES.md) diff --git a/llms.txt b/llms.txt index e032879..e41064a 100644 --- a/llms.txt +++ b/llms.txt @@ -29,6 +29,7 @@ Every page opens with the problems it is for, in the reader's words. The short v - [The one file your AI should read before it answers anything](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/docs/modules/03-ai-agent-integration/context-file.md): you are connecting an assistant to your warehouse, or choosing which one. Its answers come back as fluent queries that use the wrong table or the wrong meaning of customer. Template: [Business context file](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/templates/03-ai-agent-integration/context-file.md). - [Golden question sets](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/docs/modules/01-ai-data-quality/golden-question-sets.md): people quote the assistant's numbers and nobody has checked one against a number they already trust. The same question asked twice gave two answers. A model or prompt change went out and the only test is waiting for a complaint. A vendor quotes a benchmark score nobody can relate to their own tables. Template: [Golden question set](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/templates/01-ai-data-quality/golden-question-set.md). +- [Agreement measurement for analytics QA](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/docs/modules/01-ai-data-quality/agreement-measurement.md): the assistant scored 70 percent on your golden questions and nobody can say whether that is bad. Two people gave the board two numbers for the same quarter and both queries look right. The score moved five points after an upgrade and nobody knows whether that is noise. A week of prompt work is about to start because the score is low. Template: [Agreement-measurement control plan](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/templates/01-ai-data-quality/agreement-measurement-control-plan.md). ### The metric itself is the question diff --git a/templates/01-ai-data-quality/agreement-measurement-control-plan.md b/templates/01-ai-data-quality/agreement-measurement-control-plan.md new file mode 100644 index 0000000..45821ca --- /dev/null +++ b/templates/01-ai-data-quality/agreement-measurement-control-plan.md @@ -0,0 +1,179 @@ +# Agreement-measurement control plan + +Control artifact for the pattern [Agreement measurement for analytics QA (M1-05)](../../docs/modules/01-ai-data-quality/agreement-measurement.md). MIT licensed — copy this file into your own repository, replace every `[TODO: …]` field, and delete the guidance blockquotes. The blinding rule follows the consensus guidance for multi-analyst studies [ACZEL-2021], the repeatability and reproducibility vocabulary follows the international standard for measurement precision [ISO-2023], and the kappa arithmetic is the two-judge form used in attribute agreement analysis [AUST-PONS-2022] (see [REFERENCES.md](../../REFERENCES.md)); everything organization-specific is a `[TODO]`. + +> **How to use with an AI analytics agent:** the agent never sees section 2 while the study runs, because the answer sheets are reference answers. Section 3 is the arithmetic, section 4 says what each rate means, section 5 is the machine-readable core a scoring harness consumes, and section 6 is the prompt library. The smallest complete study is one second person, twenty questions, and one afternoon. +> +> **What this plan measures:** how repeatable your own reference answers are, before those answers are used to score anything. Building the twenty questions and comparing answers as values is the job of the [golden question set](golden-question-set.md); this plan sits on top of it. + +## 1. Who judges + +| Role | Who | Rule | +|---|---|---| +| Judge A, the owner of the golden set | `[TODO: name]` | The verified expected values in the golden set are A's answers. A does not re-answer for this study. | +| Judge B, the second person | `[TODO: name; someone who can answer from the data without A's help, and who does not report to A if that can be arranged]` | Answers all twenty from the question text alone. Sees neither the key nor A's queries. Does not discuss the questions with anyone until the sheet is submitted. | +| Scorer | `[TODO: name; may be A, but only after B's sheet is in]` | Compares values within tolerance, runs the assistant, fills section 2b. | +| Repeat by A (optional) | `[TODO: date, at least four weeks after the key was written]` | A answers the twenty again without opening the key. This measures repeatability, the same person twice; the main study measures reproducibility, two different people. | + +> Blind means blind. A walk-through of "how we read these questions" before B starts turns two judges into one, and the study will then pass by construction. If B has a question about wording, B writes it on the sheet as a choice made, and does not ask. + +## 2. The answer sheets + +### 2a. The two people + +One row per golden question. The question text is exactly what the assistant would be sent. "Conditions" is three bullets under the value: what was excluded, which calendar, and when the data was pulled. An assistant writes them from its own query; a person verifies them in seconds. "Tolerance" is the reporting difference a person decided the operation can live with for that metric, set before the study. + +| # | Question, as sent to the assistant | Tolerance | A's value (the key) | A's conditions | B's value | B's conditions | Conditions match? | Agree? | +|---|---|---|---|---|---|---|---| +| 1 | `[TODO]` | `[TODO: e.g. 0.1 percent, or exact]` | `[TODO]` | `[TODO: excluded / calendar / as-of]` | `[TODO]` | `[TODO: excluded / calendar / as-of]` | `[TODO: yes / no]` | `[TODO: yes / no]` | +| 2 | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | +| … | | | | | | | | | +| 20 | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | + +> Check the conditions first. Two answers whose conditions differ are a disputed row whatever the values say, because a matching value under different conditions is a coincidence that will split later. Two answers whose conditions match agree when the values land within the tolerance. Write the reason for every disputed row in section 2b before anyone argues about who was right. + +### 2b. The assistant, scored three ways + +Filled by the scorer after both sheets are in. The assistant is sent the question text only, one question per conversation, three repetitions as the golden set requires. + +| # | Two people agree? | Assistant's value | Matches A? | Matches B? | Reading | +|---|---|---|---|---|---| +| 1 | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO: settled, assistant right / settled, assistant wrong / disputed: what the question left open]` | +| … | | | | | | +| 20 | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | + +> A disputed row is not scored. It does not count for or against the assistant, whichever person it sided with. It goes to section 4 as a definitions item. + +## 3. The agreement arithmetic + +Spreadsheet-ready. One row per question, and everything below is a column or a cell. + +1. **Agreement per row.** First `conditions_match`, by hand, from the two conditions columns. Then `agree = conditions_match AND ABS(A_value - B_value) <= tolerance` for an absolute tolerance, or `ABS(A_value - B_value) / ABS(A_value) <= tolerance` for a relative one. Exact rows agree only when equal. Refusal rows agree when both people declined. +2. **Human agreement rate.** `p = COUNTIF(agree, TRUE) / n`, with `n = 20`. This is the ceiling. +3. **The band.** `half_width = SQRT(p * (1 - p) / n)`, and in questions, `ROUND(n * half_width)`. At twenty questions and a rate of 0.75, the half-width is 0.097, which is about two questions either way. The band runs from `p - half_width` to `p + half_width`. This two-question band is the framework's own rule of thumb, chosen because it fits in one cell; a formal interval would be wider and would say less. +4. **Assistant against each person.** `a_A = COUNTIF(matches_A, TRUE) / n` and `a_B = COUNTIF(matches_B, TRUE) / n`. Percent agreement is the number for values: two people landing within a tenth of a percent of each other by chance is not a real risk, so no chance correction is applied. +5. **Assistant on the settled questions.** `settled = COUNTIF(agree, TRUE)` and `a_settled = COUNTIFS(agree, TRUE, matches_A, TRUE) / settled`. The misses inside this subset are the assistant's own errors, and they are the worklist. +6. **Kappa, for pass-or-fail verdicts only.** When A and B have each marked the assistant's answers pass or fail, build the four cells: `a` = both pass, `b` = A pass and B fail, `c` = A fail and B pass, `d` = both fail. Then `po = (a + d) / n`, `pe = ((a + b) / n) * ((a + c) / n) + ((c + d) / n) * ((b + d) / n)`, and `kappa = (po - pe) / (1 - pe)`. Worked: with `a = 11`, `b = 3`, `c = 2`, `d = 4`, `po = 0.75`, `pe = 0.70 × 0.65 + 0.30 × 0.35 = 0.56`, and `kappa = 0.19 / 0.44 = 0.43`. Report kappa beside `po`, never alone: when most answers pass, `pe` climbs and kappa collapses between judges who agree on nearly everything. +7. **Repeatability, if A re-answered.** The same arithmetic as steps 1 and 2 with A's second sheet against the key. A repeatability rate below the reproducibility rate means the questions have moved in the owner's own head, and the key needs its dated rows checked. + +## 4. What a rate means + +| Reading | What it means | What to do | Within | +|---|---|---|---| +| Values match, conditions differ | A coincidence, not an agreement; the two numbers will split later | Record the row as disputed and write the two conditions side by side | Before scoring | +| Any disputed rows | The question has two defensible readings, and the assistant will answer it confidently either way | One line per row in the [context file](../03-ai-agent-integration/context-file.md) naming the chosen reading, and a dated new row in the golden set; the old row is retired, never edited. The row is not scored until then. | `[TODO: e.g. one week]` | +| Human rate of 20 of 20 on the first run | The blind probably broke, or the questions cannot catch anything | Check that B saw nothing; add the traps the golden set asks for; rerun | Before scoring | +| Human rate below 12 of 20 | The questions, not the people, are the problem | Rewrite each question with the period, the filter and the definition stated in the text; rerun both sheets. The floor is the framework's own. | Before scoring | +| Assistant inside the band against either person | Its disagreement with the key is inside the key's own variation | No model work. The next hour goes into the disputed rows above. | Same day | +| Assistant below the band against both people | A real gap beyond what the definitions explain | Work the misses on the settled rows in the golden set's order: context, then model, then data | `[TODO: e.g. two working days]` | +| Misses on settled rows, at any assistant rate | The assistant's own errors | Each is a finding in the golden set's run log, classified in the same order | Same as above | +| The human rate moved between two runs | Maybe a change, maybe noise | Put the rate on a control chart with the [SPC control plan](spc-control-plan.md); react outside the limits, not inside | On the next run | + +> The ceiling excuses nothing below it, and it rises. Every disputed row that becomes a written definition raises the human rate on the next run, and the assistant is held to the new rate. + +## 5. Machine-readable core + +```json +{ + "spec": "agreement-measurement/v1", + "golden_set": "[TODO: path to the golden question set this study sits on]", + "judges": { + "a": { "name": "[TODO]", "role": "owner", "sheet": "the golden set's expected values" }, + "b": { "name": "[TODO]", "role": "second_person", "blind": true, "submitted_on": "[TODO: date]" }, + "scorer": "[TODO: name]" + }, + "n": 20, + "rows": [ + { + "id": "[TODO: golden question id, e.g. q01]", + "tolerance": "[TODO: e.g. relative 0.001, absolute 0, exact, or refusal]", + "a_value": "[TODO]", + "a_conditions": "[TODO: one line]", + "b_value": "[TODO]", + "b_conditions": "[TODO: one line]", + "agree": "[TODO: true | false]", + "assistant_value": "[TODO]", + "matches_a": "[TODO: true | false]", + "matches_b": "[TODO: true | false]", + "reading": "[TODO: settled_right | settled_wrong | disputed]", + "left_open": "[TODO: for a disputed row, what the question did not state; else null]" + } + ], + "results": { + "human_rate": "[TODO: agreements / n]", + "band": { "half_width": "[TODO: sqrt(p(1-p)/n)]", "low": "[TODO]", "high": "[TODO]" }, + "assistant_vs_a": "[TODO]", + "assistant_vs_b": "[TODO]", + "assistant_on_settled": "[TODO: right on settled / settled]", + "verdict_kappa": "[TODO: only if both judges marked pass/fail; else null]", + "disputed_ids": ["[TODO]"], + "settled_miss_ids": ["[TODO]"] + }, + "routing": { + "disputed": "context_file_line_and_dated_key_row", + "inside_band": "no_model_work", + "below_band": ["context", "model", "data"] + }, + "floors": { "human_rate_min": 0.6, "first_run_all_agree_is_anti_signal": true }, + "sources": ["ACZEL-2021", "ISO-2023", "AUST-PONS-2022"] +} +``` + +> The JSON restates sections 1 to 4 in a form any agent can parse. Keep them in step in the same edit; the JSON wins when the prose is ambiguous. Section 2 stays out of anything the assistant reads while the study runs. + +## 6. Agent prompt templates + +### Prompt A — score the sheets + +```text +You are scoring an agreement-measurement study. Using the rows provided: +1. For each row, decide whether A and B agree within the row's tolerance, + and whether the assistant's value matches A and matches B. Show the + comparison for any row that is close to the tolerance. +2. Compute the human agreement rate, the band as sqrt(p(1-p)/n) with n the + number of rows, the assistant's rate against A, against B, and on the + settled rows only. If pass/fail verdicts from both judges are present, + compute observed agreement and kappa exactly as in the plan and report + both together. +3. List every disputed row with what each person's conditions say about + what was excluded, which calendar, and when the data was pulled. Do not decide which reading is right. +4. State whether the assistant is inside the band against either person, + and list the settled rows it missed. Do not propose model changes. + + +[paste section 5 JSON here, with the rows filled in] + +``` + +### Prompt B — turn a disputed row into a definition, for the owner + +```text +A golden question has two defensible readings. Below are the question, the +two people's values, and the choice each of them made. Draft, for the owner +to decide: +1. One line for the business context file that states the reading chosen, + with the period boundary, filter, or definition spelled out so that the + question can no longer be read the other way. +2. The new golden-set row that follows from that reading, with today's + date as effective_from. Do not edit the old row; mark it retired. +3. One sentence on what the other reading would be right for, in case + someone needs that number under a different name. +Do not pick the reading yourself. Present both and stop. + + +[paste the row from section 5, including a_conditions and b_conditions] + +``` + +## 7. Ownership and review + +| Item | Value | +|---|---| +| Owner (one named person) | `[TODO: name]` | +| Where this plan lives | `[TODO: repository path, outside anything the assistant reads]` | +| Run before | The first time the assistant is scored against the golden set | +| Run again after | Any change to the golden set's expected values, and at least `[TODO: e.g. quarterly]` | +| Second person | Rotates if possible; the same pair for a year drifts toward the same readings | +| On a disputed row | A context-file line and a dated key row within `[TODO: e.g. one week]`; not scored until then | +| On a first run of 20 of 20 | Check the blind before celebrating | +| Review rule | Every edit to this file is reviewed like code, because an edit here changes what counts as agreement. | diff --git a/templates/README.md b/templates/README.md index 7773ff6..4bb434d 100644 --- a/templates/README.md +++ b/templates/README.md @@ -6,7 +6,7 @@ In the DMAIC mapping ([pattern template](../docs/pattern-template.md)), the arti | Directory | Module | Contents | |---|---|---| -| `01-ai-data-quality/` | AI data quality | [Retention-state control plan](01-ai-data-quality/retention-state-control-plan.md) (M1-11) · [Metric-computation control plan](01-ai-data-quality/metric-computation-control-plan.md) (M1-01) · [SPC control plan](01-ai-data-quality/spc-control-plan.md) (M1-02) · [Golden question set](01-ai-data-quality/golden-question-set.md) (M1-04) · [Plausibility-check control plan](01-ai-data-quality/plausibility-check-control-plan.md) (M1-03) · [Schema contract control plan](01-ai-data-quality/schema-contract-control-plan.md) (M1-07) | +| `01-ai-data-quality/` | AI data quality | [Retention-state control plan](01-ai-data-quality/retention-state-control-plan.md) (M1-11) · [Metric-computation control plan](01-ai-data-quality/metric-computation-control-plan.md) (M1-01) · [SPC control plan](01-ai-data-quality/spc-control-plan.md) (M1-02) · [Golden question set](01-ai-data-quality/golden-question-set.md) (M1-04) · [Plausibility-check control plan](01-ai-data-quality/plausibility-check-control-plan.md) (M1-03) · [Schema contract control plan](01-ai-data-quality/schema-contract-control-plan.md) (M1-07) · [Agreement-measurement control plan](01-ai-data-quality/agreement-measurement-control-plan.md) (M1-05) | | `02-infrastructure-design/` | Infrastructure design | _pending_ | | `03-ai-agent-integration/` | AI agent integration | [Business context file](03-ai-agent-integration/context-file.md) (M3-06) | | `04-governance-and-financial-reporting/` | Governance and financial reporting | _pending_ |