From 46f2333c2ef6128bc073ab386dffa3992701ca44 Mon Sep 17 00:00:00 2001 From: Heqing Huang Date: Fri, 18 Sep 2026 14:56:18 +0800 Subject: [PATCH] docs: add the data quality SLO pattern, with its register template Most alert thresholds on a pipeline are neither what the pipeline does nor what the reader needs, because nobody computed the first and nobody asked for the second. This page sets the objective with the person who reads the number, as the number of bad days a quarter they will live with, and keeps it apart from the control limits, which describe the pipeline. Comparing the two says whether the pipeline can meet the requirement at all. Nineteen references added, four extended. --- CONTENT_BACKLOG.md | 2 +- DECISIONS.md | 8 + REFERENCES.md | 27 ++- docs/modules/01-ai-data-quality/README.md | 2 +- .../01-ai-data-quality/data-quality-slos.md | 202 ++++++++++++++++++ llms.txt | 1 + templates/01-ai-data-quality/slo-register.md | 163 ++++++++++++++ templates/README.md | 2 +- 8 files changed, 400 insertions(+), 7 deletions(-) create mode 100644 docs/modules/01-ai-data-quality/data-quality-slos.md create mode 100644 templates/01-ai-data-quality/slo-register.md diff --git a/CONTENT_BACKLOG.md b/CONTENT_BACKLOG.md index 72f774d..b90465d 100644 --- a/CONTENT_BACKLOG.md +++ b/CONTENT_BACKLOG.md @@ -18,7 +18,7 @@ The Module column uses `1A` (definition quality) and `1B` (data integrity) for t | M1-03 | 1B | Detecting plausible-but-wrong outputs | pattern | v0.1.0 | drafted | Twyman's law via Kohavi; Shen's taxonomy, where only syntax and schema errors stop a query; control totals from the audit manual; the Twitter and Facebook restatements | 1 session | | M1-04 | 1B | Golden question sets for AI analytics validation | pattern | v0.1.0 | drafted | Half the gold answers in public text-to-SQL benchmarks are wrong; LinkedIn and Uber built hand-verified sets from their own logs; 80 answers from 1,000 identical runs | 1 session | | M1-05 | 1B | Agreement measurement for analytics QA | pattern | v0.2.0 | drafted | Twenty-nine teams on one dataset returned odds ratios from 0.89 to 2.93; ISO 5725's repeatability and reproducibility as the lens; LinkedIn found 60 percent of its questions had more than one right answer | 1 session | -| M1-06 | 1B | Data quality SLOs and error budgets | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session | +| M1-06 | 1B | Data quality SLOs and error budgets | pattern | v0.2.0 | drafted | The reliability books say not to pick a target from current performance, and data teams import the mechanics without that rule; process capability as the comparison of the reader's requirement against the pipeline's limits; Airbnb's owner-committed landing times, Uber's tier defaults and Shopify's eight-hour default as the practice it argues with | 1 session | | M1-07 | 1B | Schema drift detection and data contracts | pattern | v0.1.0 | drafted | Avro resolution as the mechanism behind the quiet change; GoCardless's six-month report on what a producer-side programme costs; the author's addition of the assistant's memory as a drift of its own | 1 session | | M1-08 | 1B | The reconciliation protocol | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session | | M1-09 | 1B | Measuring trust, not just accuracy | pattern | v0.2.0 | candidate | [TODO: prior art] | 1 session | diff --git a/DECISIONS.md b/DECISIONS.md index 5572e0c..4b438a6 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -2,6 +2,14 @@ One entry per judgment call: decision, alternatives, why. +## 2026-09-18 — M1-06 argues with the imported site-reliability objective using the reliability books' own rule, counts bad days instead of nines, and imports process capability as the comparison + +**Decision:** four calls made while drafting the data-quality-SLO pattern. (a) The Position argues with the site-reliability objective as data teams import it, the target set by the engineer from what the system achieves and expressed in nines, and it does so with the reliability literature's own words: the first book's rule not to pick a target based on current performance, and the workbook's rule that the product manager has to agree the threshold is good enough for users. The platform accounts (Uber's tier defaults, Shopify's eight-hour default, Airbnb's owner-committed landing times and its score that credits "a more realistic SLA") are used as the competent practice, never as failures. (b) The budget is a count of bad periods per quarter agreed with the reader, and the percentage is derived from it. The nines are rejected on scale: a daily table has about sixty-five readings a quarter, a tenth of a percent of that is not a morning, and the one first-party nine found sits inside a freshness definition counting rows within a partition. (c) The comparison of requirement to pipeline is process capability, with the one-sided index (Cpk for a single limit) as the register's arithmetic, read first as a gap in the reader's units; the control limits stay in the SPC control plan and are copied into the register, never edited there. (d) The incident-cost figures were traced and not repeated: the Gartner average is a self-estimate by 154 vendor-nominated reference customers, the IBM trillion ends at an undated infographic, and the vendor survey has no published questionnaire; the page says plainly that no figure survived. + +**Alternatives:** argue with doing nothing, which no competent practitioner defends; argue with the data-observability purchase, which this audience mostly has not made; express the objective as a percentage with the count in parentheses, which puts the number nobody agreed to first; teach the two-sided Cp and Cpk, which doubles the arithmetic for requirements that are almost always one-sided. + +**Why:** a Position should contest the best advice the reader will actually hear, and for this unit that is the reliability literature, which is good and which the reader's engineer has read. Using its own rule against the import keeps the argument one a competent practitioner could still contest on scale grounds. The count keeps the requirement in the reader's words, and the capability index is the one piece of arithmetic that lets a team with one chart say whether the requirement is meetable at all. + ## 2026-09-16 — M1-05 argues with the single-owner key its sibling page installed, scores values by raw agreement and verdicts by kappa, and refuses to score disputed questions **Decision:** four calls made while drafting the agreement-measurement pattern. (a) The Position argues with the single owner writing the reference answers, the practice M1-04 itself installs. The page keeps that owner and treats the owner's answers as a gauge with an unmeasured error: a second person answers the same twenty questions blind before the assistant is scored, the human-to-human rate is the ceiling the assistant is held to, and an assistant inside that rate's band gets no model work. (b) Values within tolerance are scored by raw percent agreement, because two numbers landing within a tenth of a percent by chance is not a real risk; kappa is reserved for pass-or-fail verdicts and is reported beside the raw rate, never alone. The fixed kappa bands the quality manuals grade against are not printed, because no source inside the ten-year window could be verified for them, and the question is left for the author. (c) The band is plus and minus the square root of p(1−p)/n, about two questions at twenty, and the floor of twelve agreements in twenty below which the questions rather than the people are the problem is likewise the framework's own; both are marked as such on the page and in the template. (d) A question the two people disagree on is not scored for or against the assistant. It routes to one line in the context file and a dated new row in the key, so the assistant is scored only on settled questions, and its misses there are the only errors that are its own. (e) Three calls from the author's review, folded in on 2026-09-17: the quality manuals' kappa bands are not used, because a percentage difference can be a delay, an exclusion, a calendar, or two different metrics landing near each other, and no band separates those; the logic is checked before any statistic, through a three-bullet conditions summary under every answer (what was excluded, which calendar, when the data was pulled) that an assistant can write and a person can verify; and the per-metric tolerance is set by a person as the reporting difference the operation can live with, before the study. Nobody is the better judge; the study is run for alignment, and its working signal is that important calls spend their time on actions rather than on whose number is right. diff --git a/REFERENCES.md b/REFERENCES.md index 885ae0b..48438a7 100644 --- a/REFERENCES.md +++ b/REFERENCES.md @@ -14,12 +14,14 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[ACZEL-2021]** — Aczel, Balazs; Szaszi, Barnabas; Nilsonne, Gustav; van den Akker, Olmo R.; et al. "Consensus-based guidance for conducting and reporting multi-analyst studies." _eLife_ 10, e72185, November 9, 2021. doi:10.7554/eLife.72185. Read in full from the publisher. A 54-author consensus paper produced by a reactive-Delphi panel of methodologists, defining a multi-analyst study as one that engages multiple analysts to independently analyze the same data. Verified verbatim: "Co-analysts should not communicate with each other about their analyses until after all initial reports have been submitted", and the recommendation to state the research question "without communicating the lead team's preferred analysis choices or expectations about the conclusions". Cited for the blinding rule; the guidance is written for research studies, and the framework carries it to a two-person study inside a company. - **[AGENTSMD]** — AGENTS.md. "A simple, open format for guiding coding agents." agents.md, fetched August 26, 2026. Open format stewarded by the Agentic AI Foundation under the Linux Foundation; the page describes it as "a README for agents: a dedicated, predictable place to provide the context and instructions" and confirms there are no required fields, just standard Markdown. The key carries no year because the page is an undated living document. The adoption figure ("used by over 60k open-source projects") is self-reported with no methodology; attribute it, never assert it. The format targets coding agents; carrying business context in it is this framework's extension, not documented practice. +- **[AHMED-2020]** — Ahmed, Arbab; Deszczynski, Bruno (Shopify). "How to Reliably Scale Your Data Platform for High Volumes." Shopify Engineering, December 8, 2020. First-party; read in full. Verified verbatim: the batch ingestion service's "primary objective is to ensure that a raw dataset is available within the interval defined by its data freshness service level objective (SLO)", and it "is operating reliably so long as every dataset being extracted is no older than eight hours— the default freshness SLO". Cited for a platform-wide default freshness objective set by the platform's own reliability engineers, and for an objective expressed as a per-dataset age limit rather than a rate. The post does not say who consumes the datasets or what they need. - **[ATIL-2024]** — Atil, Berk; Aykent, Sarp; Chittams, Alexa; Fu, Lisheng; Passonneau, Rebecca J.; et al. "Non-Determinism of 'Deterministic' LLM Settings." arXiv:2408.04667, submitted August 6, 2024, read at abstract level (v5, April 2, 2025). Five language models configured for deterministic output, eight tasks, ten runs each: "accuracy variations up to 15% across naturally occurring runs", a best-to-worst gap of up to 70 percent, and no model that "consistently delivers repeatable accuracy across all tasks, much less identical output strings". Cited for the size of run-to-run variation at settings meant to be deterministic; the models are 2024-era and the per-task figures are in the unread PDF. - **[AUST-PONS-2022]** — Aust, Jonas; Pons, Dirk (University of Canterbury). "Assessment of Aircraft Engine Blade Inspection Performance Using Attribute Agreement Analysis." _Safety_ 8(2), article 23, March 29, 2022. doi:10.3390/safety8020023. Read at abstract level through the Crossref record, because the publisher refuses automated fetching. Verified: photographs of 26 blades were shown twice, in random order, to 50 industry practitioners of three skill levels, 2,600 observations, analysed with attribute agreement analysis and kappa; "operators were on average 82.5% consistent with their serviceability decision, while achieving an inspection accuracy of 67.7%"; and "the operators' reproducibility was 15.4%, as was the accuracy of all operators with the ground truth". Cited as a peer-reviewed application of attribute agreement analysis and for the within-person figure. The domain is visual inspection, and the 15.4 percent is all fifty operators agreeing at once, which is not the two-judge measure the framework uses. - **[AVRO-SPEC]** — The Apache Software Foundation. "Apache Avro Specification," version 1.12.0. avro.apache.org, undated page, fetched September 11, 2026. The key carries no year because the page is versioned rather than dated; the same rules were spot-checked against the 1.11.1 spec and the wording is unchanged. Cited for the Schema Resolution section, which makes the structural-versus-semantic asymmetry mechanical: fields are "matched by name" rather than position; "if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored"; "if the reader's record schema has a field that contains a default value, and writer's schema does not have a field with the same name, then the reader should use the default value from its field"; and "if the reader's record schema has a field with no default value, and writer's schema does not have a field with the same name, an error is signalled". Enum resolution behaves the same way: an unknown writer symbol falls back to the reader's default if one exists, and otherwise errors. - **[BALFOUR-2013]** — Balfour, Brian. "The Never Ending Road To Product Market Fit." brianbalfour.com, December 11, 2013. The flattening-curve reading of product-market fit, with his own caveat that flattening only shows fit for some market and that the next job is segmenting the curve to identify who retained. Distinct from [BALFOUR-2018]. - **[BALFOUR-2018]** — Balfour, Brian; Winters, Casey; Kwok, Kevin; Chen, Andrew. "Growth Loops are the New Funnels." Reforge, 2018. Existence verified via the authors' own announcements; the page blocked direct fetching at verification time (2026-08-04), so cite the thesis, and [TODO: verify] any direct quote against the live page. - **[BAVARESCO-2025]** — Bavaresco, Anna; Bernardi, Raffaella; Bertolazzi, Leonardo; Elliott, Desmond; et al. "LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks." ACL 2025, main conference; arXiv:2406.18403, submitted June 26, 2024, read at abstract level (v3, June 2, 2025). The JUDGE-BENCH study: "20 NLP datasets with human annotations" and "11 current LLMs, covering both open-weight and proprietary models". Verified: "Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated", and "LLMs should be carefully validated against human judgments before being used as evaluators". Cited for the validation rule; the per-dataset correlations are in the unread PDF. Twenty authors in all. +- **[BENKOVA-2024]** — Benková, Marta; Bednárová, Dagmar; Bogdanovská, Gabriela. "Process Capability Evaluation Using Capability Indices as a Part of Statistical Process Control." _Mathematics_ 12(11), 1679, May 28, 2024. doi:10.3390/math12111679. Open access; the publisher's site refused automated fetching (HTTP 403), so the article was read from an Internet Archive capture of its publication day. A research article from the Technical University of Košice whose opening sections carry the textbook definitions. Verified verbatim: capability indices "compare what the process "should do" with what the process "actually does""; the assessment aims "to determine the degree of reliability with which the values of the quality attribute under consideration correspond to the specification based on customer requirements"; control limits are computed from the process data (UCL = x̿ + A3·s̄); and "Process capability represents the performance of a process only in a state of statistical control", with the note that applying capability methods to an unstable process "leads to unreliable estimates of process capability and should never be done". Cited for the customer origin of specifications and the control-first rule; its examples are manufacturing. - **[BEREZOVSKY-2022]** — Berezovsky, Olga. "How to measure cohort retention." Lenny's Newsletter, August 30, 2022. N-day against unbounded retention, and the cohort-table layout; stresses that the two give completely different retention data. Possible partial paywall. - **[BEREZOVSKY-2024]** — Berezovsky, Olga. "Measuring Non-Cohorted Retention or Blended Churn." Data Analysis Journal #188, February 21, 2024. Framing citable; article body is paywalled — [TODO: verify] specifics against a subscription copy before quoting. - **[BEREZOVSKY-2025]** — Berezovsky, Olga. "How to Report on DAU/MAU Ratio." Data Analysis Journal #261, June 4, 2025. Free excerpt verified (scope: ratio misconceptions, reporting mistakes, benchmark caveats); full text paywalled — cite excerpt-level claims only, [TODO: verify] deeper claims against a subscription copy. @@ -28,7 +30,8 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[BERNHARDSSON-2018]** — Bernhardsson, Erik. "Waiting time, load factor, and queueing theory: why you need to cut your systems a bit of slack." erikbern.com, March 27, 2018. First-party account from a mortgage lender's CTO: M/M/1 and M/G/1 models, the utilization-versus-waiting-time curve, and the rule of keeping urgent work under 50 percent average load. Verified negative: the post does not mention Little's Law, so do not attribute the law to it. - **[BERNHARDSSON-2019]** — Bernhardsson, Erik. "Modeling conversion rates using Weibull and gamma distributions." erikbern.com, August 5, 2019. Kaplan-Meier on censored data, and a two-parameter split between the conversion rate a cohort approaches and the speed at which it converges. The data is Better.com's, and the author wrote the library the post recommends (convoys). - **[BETTER-2020]** — Better Mortgage. Closing-time claims disclosure page. First-party company marketing claim, not independent measurement: average closing time from January 2019 to May 2020 stated as 32 days against a 42-day industry average attributed to a third-party origination report. Use only as a labelled company claim. -- **[BEYER-2016]** — Beyer, Betsy; Jones, Chris; Petoff, Jennifer; Murphy, Niall Richard (eds.). _Site Reliability Engineering: How Google Runs Production Systems_. O'Reilly Media, 2016. Cited for chapter 6, "Monitoring Distributed Systems," written by Rob Ewaschuk. Verbatim and load-bearing: "We avoid 'magic' systems that try to learn thresholds or automatically detect causality," alongside the rule that the alerting rules catching real incidents "should be as simple, predictable, and reliable as possible." The chapter is freely readable on Google's own sre.google site, whose page metadata shows 2017 in places; the print edition is 2016. Context caveat: the chapter argues from serving systems with load-testable capacities, not data pipelines; cite its position, never a claim about pipelines. +- **[BEYER-2016]** — Beyer, Betsy; Jones, Chris; Petoff, Jennifer; Murphy, Niall Richard (eds.). _Site Reliability Engineering: How Google Runs Production Systems_. O'Reilly Media, 2016. Cited for chapter 6, "Monitoring Distributed Systems," written by Rob Ewaschuk. Verbatim and load-bearing: "We avoid 'magic' systems that try to learn thresholds or automatically detect causality," alongside the rule that the alerting rules catching real incidents "should be as simple, predictable, and reliable as possible." The chapter is freely readable on Google's own sre.google site, whose page metadata shows 2017 in places; the print edition is 2016. Context caveat: the chapter argues from serving systems with load-testable capacities, not data pipelines; cite its position, never a claim about pipelines. Also cited for chapter 3, "Embracing Risk", and chapter 4, "Service Level Objectives", both freely readable on sre.google. Verified verbatim from chapter 3: "Product Management defines an SLO, which sets an expectation of how much uptime the service should have per quarter" and "100% is probably never the right reliability target". From chapter 4, under "Choosing Targets": "Don't pick a target based on current performance", whose body warns that adopting such values "may lock you into supporting a system that requires heroic efforts to meet its targets"; "Have as few SLOs as possible"; and "Start by thinking about (or finding out!) what your users care about, not what you can measure". Appendix A: 99.9 percent allows 2.16 hours of unavailability per quarter. The serving-systems caveat applies to these chapters too. +- **[BEYER-2018]** — Beyer, Betsy; Murphy, Niall Richard; Rensin, David K.; Kawahara, Kent; Thorne, Stephen (eds.). _The Site Reliability Workbook: Practical Ways to Implement SRE_. O'Reilly Media, 2018. Freely readable on sre.google. Cited for three chapters. Chapter 2, "Implementing SLOs" (Thurgood and Ferguson with Hidalgo and Beyer): "An SLO sets a target level of reliability for the service's customers"; the stakeholder rule, "The product managers have to agree that this threshold is good enough for users—performance below this value is unacceptably low and worth spending engineering time to fix"; the concession that "your current performance can be a good place to start if you don't have any other information", paired with "don't let current performance limit you as you refine your SLO"; the worked budget, a 99.9 percent success objective on 3 million requests over four weeks leaving "a budget of 3,000 (0.1%) errors over that period"; the window rule, "We recommend defining this period as an integral number of weeks so it always contains the same number of weekends"; and Table 2-1's pipeline indicators, freshness as "The proportion of the data that was updated more recently than some time threshold" and correctness as "The proportion of records coming into the pipeline that resulted in the correct value coming out". Chapter 5, "Alerting on SLOs": "Burn rate is how fast, relative to the SLO, the service consumes the error budget." Chapter 13, "Data Processing Pipelines" (Sodt and Maravić of Spotify, with Luo, O'Connor and Ward), at : freshness objectives take the forms "X% of data processed in Y [seconds, days, minutes]", "The oldest data is no older than Y", and "The pipeline job has completed successfully within Y"; "A correctness target can be difficult to measure, especially if there is no predefined correct output", with golden data as the comparison; and the backward-looking example "no more than 0.1% of your invoices are incorrect per quarter". Context caveat: the pipeline examples are streaming and batch systems at large companies. - **[BHADAURIA-2026]** — Bhadauria, Divya (Hasso Plattner Institute, University of Potsdam); Harmouch, Hazar (University of Amsterdam); Naumann, Felix (HPI); Srivastava, Divesh (AT&T Chief Data Office); Ehrlinger, Lisa (HPI). "A Catalog of Data Errors." arXiv:2604.09277v1 [cs.DB], submitted April 10, 2026. Preprint with no stated venue; read from the PDF. Catalogs 35 error types across three non-overlapping categories, missing, incorrect and redundant. Cited for the disguised missing value, defined as a value where "𝑡.𝑎.𝑣 = 𝑀(⊥) ∧ 𝑡.𝑎.𝑣 ∈ 𝑑𝑜𝑚(𝑎)", in words a value that "falls within the domain and appears syntactically valid, but whose true real-world counterpart is a missing value". Load-bearing caveat in the paper's own words: "their detection can be challenging for automated approaches because they often fall within the defined domain or expected context of the attribute, thus appearing valid". Preprint status means the taxonomy may still change; cite the definition, not the count. - **[BHASKAR-2023]** — Bhaskar, Adithya; Tomar, Tushar; Sathe, Ashutosh; Sarawagi, Sunita. "Benchmarking and Improving Text-to-SQL Generation under Ambiguity." EMNLP 2023, main conference; arXiv:2310.13659, submitted October 20, 2023, read at abstract level (v1). Verified: existing benchmarks assume "each text query corresponds to one correct SQL", and the AmbiQT benchmark holds over 3,000 examples whose questions can be read as two plausible SQL queries "due to lexical and/or structural ambiguity". Cited for the one-correct-answer assumption that benchmark scoring rests on. - **[BIGEYE-2021]** — Li, Henry (Bigeye). "Anomaly detection part 1: The key to effective data observability." Bigeye blog, October 20, 2021. Vendor material from a data-observability company selling adaptive thresholds; the institutional key flags the conflict. Verified claims: "Thresholds need to adapt to changes in the business behavior that appear as seasonality, trends, and pattern changes," plus the warning that alert overload makes the safety net stop working. Cite as the category's own description of adaptive thresholds, never as evidence they work. Not the Jinyang Li of [LI-2023]. @@ -45,6 +48,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[CHEN-DAUMAU]** — Chen, Andrew. "DAU/MAU is an important metric to measure engagement, but here's where it fails." andrewchen.com, undated; the key deviates from the AUTHOR-YYYY convention because the page carries no publication date. Carries the sentence "Usually apps over 20% are said to be good, and 50%+ is world class", with no source given for either threshold, and credits Facebook with popularizing the ratio. Cite it as the clearest statement of the folklore thresholds and their absent sourcing, never as a benchmark. - **[CHIN-2024]** — Chin, Cedric. "Becoming Data Driven, From First Principles." Commoncog, January 28, 2024, updated October 15, 2024. Free essay presenting XmR charts as the working tool for understanding variation in business metrics; credits Shewhart, Deming, and Wheeler, recommends Wheeler's _Understanding Variation_, and warns against limits computed from a global standard deviation. The site's deeper companion piece is member-gated and was not read. The author sells memberships and runs the free charting tool behind [TAYLOR-2025], so he is commercially interested in the topic's revival, though not a data-tooling vendor. - **[CHOO-2016]** — Choo, H. J. James. "Little's Law – A Practical Approach to Understanding Production System Performance." _Project Production Institute Journal_ Vol. 1, Winter 2016, published December 18, 2016. Free full text. Used as a fetchable stand-in for Hopp and Spearman's _Factory Physics_, which was not verified in this round. +- **[CICHY-RASS-2019]** — Cichy, Corinna; Rass, Stefan. "An Overview of Data Quality Frameworks." _IEEE Access_ 7, pp. 24634–24648, published February 15, 2019. doi:10.1109/ACCESS.2019.2899751. Peer-reviewed and open access; the publisher's article pages refuse automated fetching, so the paper was read from its open-access PDF. Surveys "Twelve different data quality frameworks". Verified verbatim, the definitions it adopts from Wang and Strong's 1996 survey of data consumers: "Completeness: The extent to which data are of sufficient breadth, depth and scope for the task at hand"; "Accuracy: The extent to which data are correct, reliable and certified"; "Timeliness: The extent to which the age of the data is appropriate for the task at hand"; and "The most common dimensions are completeness, timeliness, and accuracy, followed by consistency and accessibility." The paper does not use the phrase "fitness for use". Cited for the three dimensions and for two of their definitions being relative to the task. - **[COLDEWEY-2016]** — Coldewey, Devin. "Facebook miscalculation significantly inflated average video view times for years." TechCrunch, September 22, 2016. Secondary source, cited because the primary, a post to Facebook's advertiser help page, is no longer retrievable. Reproduces the company's statement verbatim: the metric was defined as "total time spent watching a video divided by the total number of people who have played the video" but "erroneously" calculated as "the total time spent watching a video divided by only the number of people who have viewed a video for three or more seconds." The 60 to 80 percent overstatement is a Publicis Media estimate reported by the Wall Street Journal, which first reported the error; it is not a Facebook figure. The error stood for about two years. - **[CONFLUENT-COMPAT]** — Confluent. "Schema Evolution and Compatibility for Schema Registry on Confluent Platform." docs.confluent.io, undated continuously-updated documentation, fetched September 11, 2026; the key carries no year for that reason. Vendor documentation, but the normative definition of compatibility modes, which are specified nowhere else: BACKWARD means "consumers using new schema can read data written with old schema", FORWARD means "consumers using old schema can read data written with new schema", and "The default compatibility mode is BACKWARD." Load-bearing for the coordination cost a registry imposes: under BACKWARD you "upgrade all consumers before you start producing new events", and under FORWARD you "first upgrade all producers to using the new schema and make sure the data already produced using the older schemas are not available to consumers, then upgrade the consumers". Cite for the semantics and the upgrade ordering, never as evidence that buying a registry is the right move. - **[CONVOYS]** — Bernhardsson, Erik (at Better). _convoys_: statistical models for time-lagged conversions. MIT licence. Software rather than a publication, so the key carries no year. The repository was verified; its documentation site was fetch-blocked at verification time. @@ -54,6 +58,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[DAVIDSONPILON-2019]** — Davidson-Pilon, Cameron. "lifelines: survival analysis in Python." _Journal of Open Source Software_ 4(40):1317, 2019. doi:10.21105/joss.01317. Open access, CC-BY-4.0. The censoring argument and the survival-function definition used in the metrics library come from the documentation intro page rather than the JOSS paper, and that page does not define the Kaplan-Meier estimator itself. and - **[DBT-CONTRACTS]** — dbt Labs. "Model contracts." dbt Developer Hub documentation, page states last updated September 10, 2026. The key carries no year because the page is a living document. Vendor documentation from a transformation-tool company; cited for what the tool category can enforce, never as a product recommendation. A contract is "a set of upfront 'guarantees' that define the shape of your model"; when enforced it "must include every column's `name` and `data_type`"; at build time "dbt will run a 'preflight' check to ensure that the model's query will return a set of columns with names and data types matching the ones you have defined", and a mismatch means the model "will fail to build". Breaking changes detected against a previous project state include removing an existing column and changing an existing column's `data_type`. Two caveats the docs state themselves: views support column names and data types "but not `constraints`", and several platform constraints are "Definable and not enforced", so "a model can still build even if building the model violates the constraint". - **[DBT-DATA-TESTS]** — dbt Labs. "Add data tests to your DAG." dbt Developer Hub, page last updated September 4, 2026; fetched September 6, 2026. Product documentation for the dominant open-source transformation tool in its category; the institutional key flags the interest. Verified: data tests "are `select` statements that seek to grab 'failing' records, ones that disprove your assertion," passing when zero rows return; the four generic tests are `unique`, `not_null`, `accepted_values`, and `relationships`; singular tests are one-off SQL assertions. The page documents no test that compares a computed metric to an expected total or reconciles to an external number, which is the negative claim the pattern rests on. The key carries no year because the page is a living document. +- **[DBT-FRESHNESS]** — dbt Labs. "freshness." dbt Developer Hub reference documentation, page states last updated September 16, 2026. The key carries no year because the page is a living document. Vendor documentation from a transformation-tool company; cited for the mechanism only. Verified: `warn_after` is "How old the most recent data can be before a freshness check reports a warning. Requires both `count` and `period`"; `error_after` is "How old the most recent data can be before a freshness check reports an error. Same format as `warn_after`"; `loaded_at_field` is the "Column dbt queries to determine the most recent loaded timestamp"; `period` is one of minute, hour or day; and "One or both of `warn_after` and `error_after` can be provided. If neither is set, dbt will not check freshness for that resource." The documentation supplies no default ages and says nothing about who chooses them, which is the gap the pattern fills. - **[DBT-UNIT-TESTS]** — dbt Labs. "Unit tests." dbt Developer Hub, page last updated September 3, 2026; fetched September 6, 2026. Product documentation, same interest as [DBT-DATA-TESTS]. Verified: unit tests "validate your SQL modeling logic on a small set of static inputs before you materialize your full model in production," with mock inputs (`given`) and expected output (`expect`); available from dbt v1.8. Cited for what a unit test checks, the author's logic against inputs the author supplied, and not for a judgment about the feature. The key carries no year because the page is a living document. - **[DELGADO-TIBAU-2019]** — Delgado, Rosario; Tibau, Xavier-Andoni. "Why Cohen's Kappa should be avoided as performance measure in classification." _PLoS ONE_ 14(9), e0222916, September 26, 2019. doi:10.1371/journal.pone.0222916. Read from the publisher. Verified: kappa "penalizes judges with similar marginals compared with judges with different ones", the paper constructs cases in which "a worse classifier gets higher Kappa score", and the Matthews correlation coefficient is proposed instead because it decreases monotonically as classification worsens. Cited for the direction of kappa's misbehaviour; the setting is classifier evaluation, and the framework applies the point to two human judges. - **[DEMING-FUNNEL]** — The W. Edwards Deming Institute. "The Funnel Experiment." deming.org, fetched August 31, 2026. The key carries no year because the page is an undated living document. Carries the definition of tampering as process-worsening adjustment and the finding that adjusting a stable process to compensate for an undesirable result increases variation. The page does not name which of Deming's books describes the experiment, so the citation stops at the institute's account; [TODO: verify] against _Out of the Crisis_ before attributing the experiment to a specific book. @@ -81,7 +86,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[GALLO-2014]** — Gallo, Amy. "The Value of Keeping the Right Customers." hbr.org, October 29, 2014. The modern citation anchor for the five-to-25-times acquisition-versus-retention claim; the sentence's own hedge is "Depending on which study you believe, and what industry you're in", and no study is named. Also relays the 25-95 percent retention-profit range, crediting Reichheld at Bain. Cite the sentence together with its hedge, never the multiplier as a finding. - **[GANZ-2026]** — Ganz, Jason; Perigaud, Benoit (both dbt Labs). "Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update." dbt Developer Blog, April 7, 2026. Eleven questions run 20 times each over an insurance schema originally built by a different vendor's researchers. Verified figures: claude-sonnet-4-6 90.0% text-to-SQL against 98.2% semantic layer; gpt-5.3-codex 84.1% against 100.0%; text-to-SQL on the full set 32.7% (2023) to 64.5% (2026); the two 2026 models near-identical on both paths. Vendor conflict direct and disclosed: the authors benchmark their employer's own semantic layer. Their stated caveats matter: the whole schema was loaded as context, which they say is impractical for larger datasets; questions needing too many entity hops score 0.0% through the semantic layer, whose full-set figures are 72.7% for both 2026 models against 60.5% in 2023; and three added data models improved both approaches. Cite for direction with the out-of-scope band attached, never the headline percentages alone. - **[GAO-2024]** — U.S. Government Accountability Office. _Federal Information System Controls Audit Manual (FISCAM)_, GAO-24-107026, 2024 revision. Text extracted and searched on September 6, 2026. Verified, control BP.01.01.02 ("Control totals are employed when practicable"): "A control total is the sum of a numerical field contained in a set of records. Control totals are used to verify the completeness of a set of records as it is processed. Control totals are verified by comparing those from a processed set of records (output) to those of the same set of records before processing (input)." Also, under system-interface reconciliation: "Control totals are agreed between the source and target systems." Cited as the audit lineage of the control-total check the framework imports; the manual is written for federal financial audits and says nothing about analytics. Supersedes GAO-09-232G. -- **[GARTNER-2021]** — Sakpal, Manasi. "How to Improve Your Data Quality." Smarter with Gartner, July 14, 2021. States "Every year, poor data quality costs organizations an average $12.9 million" with no methodology, sample, or measurement year; the research behind the figure is a paid Gartner product and is not public. The live page sat behind a bot check at verification (2026-08-26) and was read via an Internet Archive capture. Cite the sentence's existence and its missing methodology, never the figure as a measurement. +- **[GARTNER-2021]** — Sakpal, Manasi. "How to Improve Your Data Quality." Smarter with Gartner, July 14, 2021. States "Every year, poor data quality costs organizations an average $12.9 million" with no methodology, sample, or measurement year; the research behind the figure is a paid Gartner product and is not public. The live page sat behind a bot check at verification (2026-08-26) and was read via an Internet Archive capture. Cite the sentence's existence and its missing methodology, never the figure as a measurement. Traced on September 18, 2026: the figure originates in Gartner's "Magic Quadrant for Data Quality Solutions" of July 27, 2020, read from a third-party licensed reprint, where reference customers nominated by each vendor estimated their own cost of poor data quality; the average of those self-estimates is the $12.9 million. Cite as a failed trace, never as a measurement. - **[GEORGE-2002]** — George, Michael L. _Lean Six Sigma: Combining Six Sigma Quality with Lean Production Speed_. McGraw-Hill, 2002. ISBN 9780071501903. Bibliography verified only; the Little's Law and process-cycle-efficiency formulations attributed to it come from secondary summaries and are unverified. Cite at thesis level, and check against a copy before quoting. - **[GOODHART-1975]** — Goodhart, Charles. "Problems of Monetary Management: The UK Experience." In _Papers in Monetary Economics_, Reserve Bank of Australia, 1975. Citation verified via independent bibliographic sources; original not freely available. Quote only the 1975 sentence, and credit the popular "measure becomes a target" phrasing to [STRATHERN-1997]. - **[GORDON-2019]** — Gordon, Brett R.; Zettelmeyer, Florian; Bhargava, Neha; Chapsky, Dan. "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook." _Marketing Science_ 38(2), 2019. Verified at abstract/DOI level; full text paywalled. doi:10.1287/mksc.2018.1135. @@ -92,6 +97,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[HENDERSON-QUEUEING]** — Henderson, Joel Parker. _queueing-theory_ (open-source reference repository). A personal reference repository, undated and without editorial review, hence no year in the key. Cite only for the line describing a funnel as a queue-of-queues, and as evidence that the idea is in circulation rather than as authority. - **[HENDRYCKS-2020]** — Hendrycks, Dan; Burns, Collin; Basart, Steven; Zou, Andy; Mazeika, Mantas; Song, Dawn; Steinhardt, Jacob. "Measuring Massive Multitask Language Understanding." arXiv:2009.03300, submitted September 7, 2020, read at abstract level (v3, January 12, 2021). The MMLU evaluation suite: a published test of knowledge and problem solving across 57 tasks, from elementary mathematics to law, built to measure a language model's multitask accuracy against a fixed task set. Cited as an example of a benchmark whose tasks are public and rerunnable, never for any model's score. - **[HERCHER-2026]** — Hercher, James. "Google's Meridian And Meta's Robyn: A Gift To Measurement Or Trojan Horses?" AdExchanger, July 14, 2026. Independent reporting on the open-source MMM renaissance; quotes MMM vendors — cite the reporting and conflict-of-interest argument, not vendor product claims. +- **[HIDALGO-2020]** — Hidalgo, Alex. _Implementing Service Level Objectives_. O'Reilly Media, August 2020. ISBN 978-1-4920-7676-6. The publisher's site refuses automated fetching (HTTP 403); the table of contents and chapter 4, "Choosing Good Service Level Objectives", were read from Internet Archive captures of the publisher's pages. Verified verbatim from chapter 4: "If you are exceeding your SLO target, your users are happy with the state of your service. If you are missing your SLO target, your users are unhappy with the state of your service"; and "It doesn't matter what the percentage actually is. Don't let anyone trick you into thinking that you have to be close to 100% in your measurements." Its Table 4-1 translates nines to time, with 99.9 percent as 43 minutes 49.7 seconds a month. Chapter 11, "Data Reliability", is by guest authors Polina Giralt and Blake Bisset; only its first three paragraphs are public and it was not read, so nothing is attributed to it. - **[HIGHALPHA-2025]** — High Alpha. "2025 SaaS Benchmarks Report." Ninth annual edition, 2025; read in full at the published report PDF, reached from the landing page. A survey of more than 800 private software companies (69 percent US, half of respondents CEOs or founders), run by a venture studio with more than 40 venture and platform partners. States its definition: software gross margin is subscription revenue less cost of goods sold divided by subscription revenue, as of Q2 2025. Median software gross margin by ARR band: 74 percent under $1M, 77 percent at $1-5M, 80 percent at $5-20M, 78 percent at $20-50M, 79 percent above $50M; upper quartiles run 80 to 86 percent and lower quartiles 60 to 71 percent. Caveats: respondents are self-selected, figures are self-reported, and the publisher is an investor in the category it benchmarks. - **[HSU-2015]** — Hsu, Jonathan. "Diligence at Social Capital Part 1: Accounting for User Growth." Medium (The Startup), September 22, 2015. Growth-accounting identities (MAU as new + retained + resurrected; prior MAU as retained + churned) and the quick ratio with rough ranges. - **[HUANG-2024]** — Huang, Renee (Snowflake). "Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI." Snowflake engineering blog, August 29, 2024. Vendor material by a product manager for the product being scored, so the conflict is total. Cited for two things only: that a model reported above 90 percent on public leaderboards measured "51%" on the vendor's internal set of "150 questions that mirror the real-world tasks business users encounter", and the evaluation design, in which human evaluators "manually select all the correct SQL queries to add to our set of acceptable gold queries" because "one question can have multiple correct answers". The product's own "90%+" figure is measured on that same private set and is not used. @@ -138,13 +144,16 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[MARTIN-BELLEMARE-2021]** — Martin, John; Bellemare, Adam (Shopify). "Capturing Every Change From Shopify's Sharded Monolith." Shopify Engineering blog, March 12, 2021. First-party. Cited for the structural-break example: schema evolution rules let producers and consumers decouple, so that "consumers are only forced to update if they care about the new fields being defined or about old fields being removed", but "Changing a source table column definition from 'Integer' to 'String' breaks compatibility, requiring all consumers to update immediately to handle these records. A failure to do so can result in erroneous processing." Scale context stated in the post: over 100 MySQL shards behind the core monolith, 400TB-plus of change-data-capture storage, and a Black Friday Cyber Monday 2020 peak of 100,000 records per second. - **[MAZAL-2023]** — Mazal, Jorge. "How Duolingo reignited user growth." Lenny's Newsletter, February 28, 2023. - **[MCCLURE-2007]** — McClure, Dave. "Startup Metrics for Pirates" (AARRR), long-version deck. First presented 2007 (date per secondary sources). +- **[MCPHILLIPS-2023]** — McPhillips, Michael (Lyft). "From Big Data to Better Data: Ensuring Data Quality with Verity." Lyft Engineering on Medium, October 3, 2023. First-party; direct fetching is blocked (HTTP 403), so verification was against the page rendered in a browser. Verified verbatim: data quality is "The measure of how well data can be used as intended. The data is semantically correct, consistent, complete, unique, well-formed, and timely"; "Data producers and consumers can define their data quality checks and verify the data when it is produced or before it is consumed"; a timeliness example, "The raw_events.bike_sessions data should land in Hive within 5 minutes of being generated"; and coverage of "over 70 teams at Lyft, covering 2,500 Hive tables with 140,000 data quality validations per week". Cited for consumers as well as producers writing checks; the post contains no objective, budget or SLA language. - **[MEHTA-2025]** — Mehta, Ravi. "Confidence engineering: Why your onboarding is probably too short." Ravi on Product, December 18, 2025. Free. First-party numbers for a checkout expanded from 3 steps to 25 steps, plus the decision-consequence and information-gap frame. His exact role at the company described is not stated on the page, so describe him as a product leader who worked on it. - **[MENKVELD-2024]** — Menkveld, Albert J.; Dreber, Anna; Holzmeister, Felix; et al. "Nonstandard Errors." _The Journal of Finance_ 79(3), pp. 2339–2390, June 2024 (online April 17, 2024). doi:10.1111/jofi.13337. Abstract verified through an indexed copy because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: nonstandard errors are defined as the uncertainty that variation in the evidence-generating process across researchers adds to a result, studied "by letting 164 teams test the same hypotheses on the same data"; "NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants." Cited for the underestimation finding. - **[META-2024]** — Meta Platforms, Inc. 2023 Form 10-K, filed February 2, 2024 (accession 0001326801-24-000012). Daily active people (DAP) definition, the ML-estimate caveat for cross-app people counting, the duplicate/false-account discussion, and the announced retirement of per-app DAU, MAU, ARPU, and MAP from periodic reports. - **[MODIG-2012]** — Modig, Niklas; Åhlström, Pär. _This is Lean: Resolving the Efficiency Paradox_. Rheologica Publishing, 2012. ISBN 9789198039306. Both authors at the Stockholm School of Economics. Bibliography verified; the flow-versus-resource-efficiency content is unverified at page level, so cite the thesis rather than passages. No flow-efficiency benchmark number outside vendor blogs survived verification. - **[MOHAMMED-2008]** — Mohammed, M. A.; Worthington, P.; Woodall, W. H. "Plotting basic control charts: tutorial notes for healthcare practitioners." _Quality and Safety in Health Care_ 17(2), 2008, pp. 137–145. doi:10.1136/qshc.2004.012047. Read in full from a hosted copy. Peer-reviewed source for the XmR arithmetic: limits at the mean ± 2.66 × mean moving range, with 2.66 = 3/1.128 and the range limit at 3.267 × mean moving range; the roughly 1-in-370 false-alarm rate of three-sigma limits under ideal conditions; the guidance that 20–25 points firm the limits up while fewer give "soft" provisional ones worth using; the three widely recommended run rules; and the warnings that day-of-week patterns and seasonality need preliminary work before charting. The examples are healthcare, and every formula is general. +- **[MONTECARLO-2023]** — Segner, Michael (Monte Carlo). "The Annual State of Data Quality Survey." montecarlo.ai, page states updated May 2, 2023; the page title carried the year 2026 when fetched on September 18, 2026. Vendor material from a data-observability company, and the institutional key flags the conflict: the survey "was commissioned by Monte Carlo and polled 200 data professionals in March 2023", fielded by Wakefield Research. Verified verbatim: monthly incidents rose "from 59 in 2022 to 67 in 2023"; "a 166% increase in average time to resolution, rising to an average of 15 hours per incident"; "68% of respondents reported an average time of detection for data incidents of four hours or more"; and "The average percentage of impacted revenue jumped to 31%, up from 26% in 2022". The linked report is form-gated and no longer downloadable, and no questionnaire, margin of error or respondent qualification for the 2023 wave was found on any page opened; the vendor's own 2022 report shows its incident average was derived from banded answers. Cited only to record that the incident figures in circulation are a vendor-commissioned survey of 200 respondents; none of the figures is repeated as a measurement. - **[MONTECARLO-2024]** — MacDonald, Lindsay. "Data Quality Anomaly Detection: Everything You Need to Know." Monte Carlo blog, updated June 20, 2024. Vendor material from a data-observability company; the institutional key flags the conflict. Describes automatically applied freshness, volume, and schema monitors "without the need for any threshold setting," with baselines learned from each table's history. The post does not discuss false positives or alert fatigue, which is the cost side of what it sells; cite it for what the category does, never for effectiveness. The URL redirected from montecarlodata.com at fetch time. -- **[MONTGOMERY-2019]** — Montgomery, Douglas C. _Introduction to Statistical Quality Control_, 8th edition. Wiley, August 2019. ISBN 978-1-119-39930-8. Cited at book level for Chapter 8, "Process and Measurement System Capability Analysis", whose title was verified from the publisher's table of contents; the chapter's text was not read for this entry, so the citation rests on the chapter title and stands for the textbook lineage of gauge repeatability and reproducibility studies rather than for any figure. +- **[MONTGOMERY-2019]** — Montgomery, Douglas C. _Introduction to Statistical Quality Control_, 8th edition. Wiley, August 2019. ISBN 978-1-119-39930-8. Cited at book level for Chapter 8, "Process and Measurement System Capability Analysis", whose title was verified from the publisher's table of contents; the chapter's text was not read for this entry, so the citation rests on the chapter title and stands for the textbook lineage of gauge repeatability and reproducibility studies rather than for any figure. Also cited, at the same chapter-title level, for the process-capability half of Chapter 8; the chapter text was not read, and the formulas the pages rely on come from [NIST-HANDBOOK] and [WHEELER-2019]. +- **[NIST-HANDBOOK]** — National Institute of Standards and Technology and SEMATECH. _NIST/SEMATECH e-Handbook of Statistical Methods_, section 6.1.6, "What is Process Capability?". doi:10.18434/M32189. The handbook's front page states it was created June 1, 2003, significantly updated April 2012, and last updated April 27, 2022; the key carries no year because the section page is undated. Read in full. Verified verbatim: "Process capability compares the output of an in-control process to the specification limits by using capability indices. The comparison is made by forming the ratio of the spread between the process specifications (the specification "width") to the spread of the process values, as measured by 6 process standard deviation units (the process "width")"; "we compare the natural variability of a stable process with the process specification limits"; and "A process where almost all the measurements fall inside the specification limits is a capable process." Carries Cp = (USL − LSL)/6σ, Cpk = min[(USL − μ)/3σ, (μ − LSL)/3σ], and the one-sided forms Cpu = (USL − μ)/3σ and Cpl = (μ − LSL)/3σ, which the SLO register uses as its margin. The page does not say who sets the specification limits; that comes from [BENKOVA-2024]. - **[OLDROYD-2011]** — Oldroyd, James B.; McElheran, Kristina; Elkington, David. "The Short Life of Online Sales Leads." _Harvard Business Review_, March 2011. Authorship, affiliations, date, and framing verified; the article body is paywalled and the widely repeated response-time multipliers could not be confirmed on the page. Co-author David Elkington was chairman and CEO of InsideSales.com, which funded the underlying study [INSIDESALES-2007]; disclose that at every use. - **[OPENAI-2026]** — OpenAI. "Assistants migration guide." OpenAI developer documentation, fetched August 26, 2026, the Assistants API shutdown date the page itself states. Verbatim and load-bearing: "We will not provide an automated tool for migrating Threads to Conversations." Cited as a dated first-party record that conversation objects held inside one vendor's platform had no automated export path at retirement. An undated living document, so the fetch date does the work; re-check before quoting again, since the page may change or disappear after the shutdown. - **[ORDONEZ-2009]** — Ordóñez, Lisa D.; Schweitzer, Maurice E.; Galinsky, Adam D.; Bazerman, Max H. "Goals Gone Wild: The Systematic Side Effects of Overprescribing Goal Setting." _Academy of Management Perspectives_ 23(1), 2009, pp. 6-16. Read in full at the open-access repository copy of the working-paper version, whose title hyphenates "Over-Prescribing". The standard caution read alongside [LOCKE-LATHAM-2002]: named side effects include "a narrow focus that neglects non-goal areas, a rise in unethical behavior, distorted risk preferences, corrosion of organizational culture, and reduced intrinsic motivation", and the authors argue goal setting should be treated as a prescription-strength medication rather than an over-the-counter one. An argument piece built on cases and prior studies, not a meta-analysis. @@ -152,6 +161,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[OVERGOOR-2014]** — Overgoor, Jan. "Experiments at Airbnb." Airbnb Engineering, May 27, 2014. - **[PALIHAPITIYA-2012]** — Palihapitiya, Chamath. "How we put Facebook on the path to 1 billion users." Conference talk, 2012; video on YouTube. Origin of the seven-friends-in-ten-days formulation. The video's title was verified at its page; upload metadata was not retrievable at verification, the talk's 2012 date rests on secondary accounts, and the formulation is attested across independent transcriptions rather than checked against the video itself. No published analysis accompanies the claim, and Facebook's own growth lecture gives different numbers and a different attribution [SCHULTZ-2014]. Cite as the origin of the folklore, never as evidence for the threshold. - **[PAN-2026]** — Pan, Tian. "Text-to-SQL in Production: Why Correct SQL Is the Easy Part." tianpan.co, April 10, 2026. Practitioner blog post. Verified: names the silent failure modes of queries that execute and return wrong results, a fan-out from joining a one-to-many table before aggregating (the worked example returns revenue "5x too high"), NULL handling, date-range and timezone errors, and business-term ambiguity, and states of the failing queries that they "execute without errors, return rows with the correct schema, and are completely wrong." Caveat: the post's headline claim that "the missing 76%" of queries are of this kind is arithmetic over published Spider 1.0 and Spider 2.0 benchmark scores, not a measurement on any production system; cite the sentence, never the percentage. The recommended defenses in the post include a second model call as a result validator, which this framework does not adopt. +- **[PARKS-2020]** — Parks, Jonathan; Quoss, Vaughn; Ellwood, Paul (Airbnb). "Data Quality at Airbnb. Part 1 — Rebuilding at Scale." The Airbnb Tech Blog on Medium, November 4, 2020. First-party; direct fetching is blocked (HTTP 403), so verification was against the page rendered in a browser. Verified verbatim: leadership "has set high expectations for data timeliness and quality"; the five goals of the 2019 initiative, including "Ensure important data always meets SLAs"; under Operations, "All important datasets are required to have an SLA for landing times, and pipelines are required to be configured with Pager Duty"; under Accountability, "We also require that teams incorporate data pipeline SLAs into their quarterly OKR planning"; and the Midas process, in which "stakeholders ... first align on design specifications before building their pipelines", after which "a data engineer then builds the datasets and pipelines based on the agreed upon specification". Caveats: no numeric objective appears anywhere in the post, and it is a first-party success account with no failure figures. - **[PEDANTICS-2023]** — "A Brief History of Marketing Mix Modeling." Pedantics (pseudonymous practitioner blog), September 16, 2023. Secondary history of MMM's econometrics/OR lineage; the primary papers it cites were not fetched — cite the history through this source. - **[PETHES-2026]** — Pethes, Ambrus (Mitzu). "AI hallucinations in analytics: why SQL transparency is the only way to trust your AI analyst." Mitzu blog, March 20, 2026. A business-intelligence vendor's post ending in a product pitch, cited as the clearest published statement of the practice of validating an AI answer by reading its query: "Without query visibility, there is no meaningful peer review and no defensible sign-off"; reviewers are told to check joins, filters, grouping, and metric references. The post does not discuss the limits of reading SQL as a control. The framework cites it as the position it argues with, never as evidence. - **[PHE-2020]** — Public Health England. "PHE statement on delayed reporting of COVID-19 cases." GOV.UK, published October 4, 2020, updated October 5, 2020. First-party statement. Verified: 15,841 cases between September 25 and October 2, 2020 were not included in reported daily totals; "A technical issue was identified overnight on Friday 2 October in the data load process that transfers COVID-19 positive lab results into reporting dashboards"; "the technical issue was caused by the fact that some files containing positive test results exceeded the maximum file size that takes these data files and loads then into central systems"; the mitigation splits large files. The statement names a file-size limit only; the widely reported explanation that the files were in a legacy spreadsheet format with a 65,536-row ceiling comes from press reporting and is not in the statement. The statement also says the issue "did not affect people receiving their COVID-19 test results." @@ -160,8 +170,10 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[POLYZOTIS-2018]** — Polyzotis, Neoklis; Roy, Sudip; Whang, Steven Euijong; Zinkevich, Martin. "Data Lifecycle Challenges in Production Machine Learning: A Survey." _SIGMOD Record_ 47(2), June 2018, pp. 17-28. Google Research, with Whang at KAIST. Peer-reviewed survey, freely readable, read in full. Carries the semantic-drift mechanism this framework leans on hardest: "a feature may suddenly change its unit (e.g., age changes from days to hours) or even disappear. Unfortunately, model training is resilient to such errors, and instead of crashing, the model accuracy may simply decrease." Also the new-category case worked through in the original: a country feature known to hold four values gains "SS", and since "'SS' could also be a valid country (say South Sudan), which means the constraint should now include five countries", the right move may be to widen the constraint rather than reject the batch. And the case-normalization example, where "US" arriving as "us" leaves the model thinking "there is a new country". Same context caveat as [BRECK-2019]: machine-learning pipelines, not metric pipelines. - **[QU-2023]** — Qu, Hila (guest); Rachitsky, Lenny (host). "The ultimate guide to adding a PLG motion." Lenny's Podcast, April 2, 2023. Funnel auditing at 38:24, activation and conversion at 47:30. - **[QUARFOOT-LEVINE-2016]** — Quarfoot, David; Levine, Richard A. "How Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution?" _The American Statistician_ 70(4), pp. 373–384, 2016 (issue dated October 2016, online November 21, 2016). doi:10.1080/00031305.2016.1141708. Abstract verified through an indexed copy because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: the article "exposes limitations of these indices when the units to be rated are not well-distributed across the rating categories", the indices being Fleiss's kappa, Conger's kappa and Krippendorff's alpha, and argues for the Brennan–Prediger coefficient and Gwet's AC2 because their agreement levels "are more robust to variation in the distribution of units that raters encounter". Cited for the instability of kappa-type indices under skew; the indices studied are the multi-rater forms, and the two-rater Cohen's kappa shares their dependence on the marginals. +- **[QUOSS-2020]** — Quoss, Vaughn; Parks, Jonathan; Ellwood, Paul (Airbnb). "Data Quality at Airbnb. Part 2 — A New Gold Standard." The Airbnb Tech Blog on Medium, November 25, 2020. First-party; direct fetching is blocked (HTTP 403), so verification was against the page rendered in a browser. Carries the six dimensions the certification addresses, verbatim: "Accuracy: Is the data correct? / Consistency: Is everybody looking at the same data? / Usability: Is data easy to access? / Timeliness: Is data refreshed on time, and on the right cadence? / Cost Efficiency: Are we spending on data efficiently? / Availability: Do we have all the data we need?"; the timeliness promise, "certified data has landing time SLAs, backed by a central incident management process"; "Every Midas model requires a Data Engineering and Data Science owner"; and "The certification process gives data consumers from every team the option to sign on as reviewers of new data model designs". The example design spec's SLA table prints "Landing time: < XX hours" with literal placeholders, so the post publishes no actual objective. Reviews are by "Data Architects or Metrics Architects", and the post concedes certified models are "much slower" to build. - **[RACHITSKY-2022]** — Rachitsky, Lenny; Timen, Yuriy. "What is a good activation rate." Lenny's Newsletter, October 25, 2022. - **[RACHITSKY-WINTERS-2020]** — Rachitsky, Lenny; Winters, Casey. "What is good retention." Lenny's Newsletter, June 9, 2020. Survey of roughly twenty growth practitioners; six-month retention benchmarks segmented by product category. Key includes both names to disambiguate from [RACHITSKY-2022] and credit the co-author. +- **[REDMAN-2016]** — Redman, Thomas C. "Bad Data Costs the U.S. $3 Trillion Per Year." _Harvard Business Review_, September 22, 2016. The live page is paywalled after its first paragraph; the full text was read from an Internet Archive capture of February 28, 2017. Verified verbatim: "$3.1 trillion, IBM's estimate of the yearly cost of poor quality data, in the US alone, in 2016", linked to an IBM Big Data & Analytics Hub infographic, "The Four V's of Big Data". That page is gone; an archived capture of the infographic carries "Poor data quality costs the US economy around $3.1 TRILLION A YEAR" with a footer naming nine organisations and no figure tied to any of them, and the same image was already online by October 2013. No methodology exists anywhere in the chain. Cited only to record that the figure fails tracing; never repeat it. - **[REICHHELD-SASSER-1990]** — Reichheld, Frederick F.; Sasser, W. Earl, Jr. "Zero Defections: Quality Comes to Services." _Harvard Business Review_, September-October 1990. The root of the retention-profit folklore. Its prose states "companies can boost profits by almost 100% by retaining just 5% more of their customers", with industry cases of 85 percent (a bank's branch system), 50 percent (an insurance brokerage) and 30 percent (an auto-service chain); the analysis is Bain client work in service industries of that era, and the client data is unpublished. The widely circulated 25-to-95-percent range is not in this article's prose; it appears in [REICHHELD-SCHEFTER-2000]. Full text paywalled; authorship and opening verified on the HBR page, profit sentences verified against a verbatim-quotes reproduction. - **[REICHHELD-SCHEFTER-2000]** — Reichheld, Frederick F.; Schefter, Phil. "E-Loyalty: Your Secret Weapon on the Web." _Harvard Business Review_ 78(4), July-August 2000. Read in full via a course-hosted reprint PDF. Contains the sentence the folklore quotes: "increasing customer retention rates by 5% increases profits by 25% to 95%", stated as the bottom line of the 1990 analysis it revisits, so the range is ten years downstream of the study it is usually credited to. - **[REINERTSEN-2009]** — Reinertsen, Donald G. _The Principles of Product Development Flow: Second Generation Lean Product Development_. Celeritas Publishing, 2009. ISBN 9781935401001. Bibliography verified; the author's own site has an expired TLS certificate and the specific principle wordings circulating in secondary summaries are unverified. Cite at thesis level; do not quote principle text. @@ -172,7 +184,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[SANDERSON-2024]** — Sanderson, Chad. "Implementing Data Contracts In The Data Warehouse." Monte Carlo blog, page states updated January 25, 2024. Double-flag: the author led the data-contract programme at Convoy, so it is a practitioner account, and it is hosted by a data-observability vendor with a commercial stake in warehouse-side monitoring, with product links and a closing call to action. Load-bearing concession, verified verbatim: "starting in the data warehouse doesn't require the help of application engineers or all data producers", and "Overall, it requires a smaller culture shift in order to succeed." His own recommendation remains both: "The best way to ensure data products can be effectively produced and maintained is to implement data contracts upstream at the service level as well as in the data warehouse." Cite the concession and the recommendation together; quoting the first without the second misrepresents him. - **[SANDERSON-KREUZIGER-2022]** — Sanderson, Chad; Kreuziger, Adrian (both Convoy). "An Engineer's Guide to Data Contracts - Pt. 1." Data Products newsletter on Substack, October 5, 2022. The byline carries both names; Sanderson introduces Kreuziger, a principal engineer at Convoy, as the author of the body, so the key names both. First-party implementation account, part one of three. Load-bearing verbatim: the stated requirement "Data contracts must be enforced at the producer level", with the sentence that follows it, "If there's nothing enforcing a contract on the producer side, you don't have a contract. That's a handshake agreement at best"; that enforcement "must be automated as part of the existing CI/CD pipeline"; that a meaning change is a breaking change, since "changing the underlying meaning of the data being produced should break the data contract", illustrated by an entity whose length and width switch "from being stored in inches to centimeters"; and the admission that "There are simply some aspects of semantic enforcement that cannot be reliably managed prior to deployment." - **[SAPARINA-LAPATA-2024]** — Saparina, Irina; Lapata, Mirella. "AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries." NeurIPS 2024, Datasets and Benchmarks track, spotlight; arXiv:2406.19073, submitted June 27, 2024, read at abstract level (v2, October 31, 2024). Verified: the benchmark covers "scope ambiguity, attachment ambiguity, and vagueness", and "even the most advanced models struggle to identify and interpret ambiguity in questions". Cited for the finding that a model does not notice a question with two readings. -- **[SCHELTER-2018]** — Schelter, Sebastian; Lange, Dustin; Schmidt, Philipp; Celikel, Meltem; Biessmann, Felix (Amazon Research); Grafberger, Andreas (University of Augsburg). "Automating Large-Scale Data Quality Verification." _PVLDB_ 11(12), 2018, pp. 1781-1794. doi:10.14778/3229863.3229867. Peer-reviewed and first-party; read from the VLDB PDF. Cited for the check vocabulary the framework's semantic half borrows: the vision that users "should be able to write 'unit-tests' for data ... analogous to established testing practices in software engineering"; Completeness as "the fraction of non-missing values in a column"; and "Consistency is defined as the degree to which a set of semantic rules are violated", where intra-relation constraints may span columns, the paper's own example being "if category is t-shirts, then the range of size is {S, M, L}". Also the syntactic-versus-semantic accuracy split, where for a product attribute colour "a value blue can be considered syntactically accurate in the given domain even if the correct value would be red". Constraints compile to aggregation queries over a distributed engine, which is why the same checks are cheap in a warehouse. +- **[SCHELTER-2018]** — Schelter, Sebastian; Lange, Dustin; Schmidt, Philipp; Celikel, Meltem; Biessmann, Felix (Amazon Research); Grafberger, Andreas (University of Augsburg). "Automating Large-Scale Data Quality Verification." _PVLDB_ 11(12), 2018, pp. 1781-1794. doi:10.14778/3229863.3229867. Peer-reviewed and first-party; read from the VLDB PDF. Cited for the check vocabulary the framework's semantic half borrows: the vision that users "should be able to write 'unit-tests' for data ... analogous to established testing practices in software engineering"; Completeness as "the fraction of non-missing values in a column"; and "Consistency is defined as the degree to which a set of semantic rules are violated", where intra-relation constraints may span columns, the paper's own example being "if category is t-shirts, then the range of size is {S, M, L}". Also the syntactic-versus-semantic accuracy split, where for a product attribute colour "a value blue can be considered syntactically accurate in the given domain even if the correct value would be red". Constraints compile to aggregation queries over a distributed engine, which is why the same checks are cheap in a warehouse. Also verified verbatim, from the learnings section: "It is also a big advantage if producers as well as consumers of datasets leverage the same system for verifying data quality, as they can re-use checks and constraints from each other, e.g., the producer can choose to adapt checks from downstream consumers earlier in the data processing pipeline." - **[SCHULTZ-2014]** — Schultz, Alex. "Lecture 6: Growth." How to Start a Startup (Y Combinator / Stanford CS183B), 2014. - **[SCHWEINSBERG-2021]** — Schweinsberg, Martin; Feldman, Michael; Staub, Nicola; et al. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis." _Organizational Behavior and Human Decision Processes_ 165, pp. 228–249, July 2021. doi:10.1016/j.obhdp.2021.02.003. Abstract verified through an indexed copy because the publisher refuses automated fetching; the journal record verified through Crossref. Verified: independent analysts tested two hypotheses on one dataset with both the analytic approach and the operationalization of the variables left to them; 29 analyses survived quality screening; the analysts reported "radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question"; and a multiverse analysis "demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices". Cited for the operationalization finding, the closest published analogue to a metric definition. - **[SEC-2016]** — U.S. Securities and Exchange Commission, Division of Corporation Finance. Staff comment letters to Snap Inc. (Evan Spiegel), November 29, 2016 (accession 0000000000-16-102863) and December 23, 2016 (accession 0000000000-16-106388), re draft registration statement. Metric comments: the "quarterly average" DAU computed from the last month alone, averaging-window disclosure, and the third-party-to-internal analytics transition. and @@ -192,6 +204,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[SNAP-2017]** — Snap Inc. Form S-1 Registration Statement, SEC, filed February 2, 2017 (accession 0001193125-17-029199). DAU definition (opens the app at least once in a 24-hour period), the daily-window rationale, and first-party measurement caveats. The filing defines no monthly metric and does not argue against one. - **[SNOWFLAKE-VQR]** — Snowflake. "Cortex Analyst Verified Query Repository." Snowflake documentation, undated, fetched September 6, 2026; the key carries no year because the page is undated. Product documentation for a paid feature, cited for one design fact: the repository is "a collection of questions and corresponding SQL queries", and the product "leverages relevant SQL queries from the repository when answering similar questions", so a verified pair is generation material the model reads rather than a held-out test. Also verified: queries "must use the names of the logical tables and columns defined in the semantic model, not those in the underlying dataset", which binds the repository to the vendor's model, and "Invalid or inaccurate queries can negatively impact Cortex Analyst's performance and accuracy". No stated maximum size or similarity threshold. - **[SPOOL-2009]** — Spool, Jared. "The $300 Million Button." UIE, January 14, 2009. +- **[SRINIVAS-2021]** — Srinivas, Suresh; Puttaswamy, Krishna (Uber). "Uber's Journey Toward Better Data Culture From First Principles." Uber Engineering Blog, March 16, 2021. First-party; read in full. Verified verbatim: before the programme, "Datasets were not properly owned — they often didn't come with quality guarantees, SLAs for bug fixes were inconsistent"; the principle "Data artifacts must have SLAs for data quality, SLAs for bugs, and incident reporting and management just like we do for services. The owner is responsible for upholding those SLAs"; "Every dataset must come with these checks and a default SLA configured", with freshness as the "time delay between production of data and when the data is 99.9% complete in the destination system including a watermark for completeness (default set to 3 9s)"; and "Dataset owners can choose to provide different SLAs, with appropriate documentation and reasoning to consumers". Cited for the owner setting the objective and documenting it to consumers, and for the nine inside the freshness default counting rows within a partition rather than days. - **[STANCIL-2021]** — Stancil, Benn. "The missing piece of the modern data stack." benn.substack.com, April 22, 2021. The metric definition-drift mechanism. Personal essay; the author co-founded Mode, a BI vendor, noted here for disclosure. - **[STANCIL-2025]** — Stancil, Benn. "The context layer." benn.substack.com, August 30, 2025. Read in full. The metrics-layer retrospective by the idea's best-known advocate: by the end of 2021 at least six companies were building a metrics-layer product; "Two pivoted, one got acquired, and one stalled," the two still going are "nothing close to a market standard," and the industry is "slowly backing away from it." His diagnosis is economic rather than technical, quoting Fivetran's chief executive that a standalone metric store could not be sold without a BI layer attached, and he concedes "I missed that then, and perhaps deluded myself into thinking a universal standard would work for us." Caveats: his BI-vendor history is disclosed at [STANCIL-2021]; the economic diagnosis is borrowed from an interested rival vendor's CEO; the essay's second half, proposing a context layer for AI agents, is speculative and is not cited here. - **[STRATHERN-1997]** — Strathern, Marilyn. "'Improving ratings': audit in the British University system." _European Review_ 5(3), 1997, pp. 305–321. The "when a measure becomes a target, it ceases to be a good measure" sentence is hers verbatim (p. 308); she credits the "Goodhart's law" naming to Hoskin. @@ -200,6 +213,7 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[TAYLOR-2025]** — Taylor, Sam. "Xmrit Launches Trend and Seasonality Features for XmR Charts." xmrit.com (Commoncog's free charting tool), April 24, 2025. A feature announcement, so treat tool claims as self-description; the statistical content stands alone: significant seasonality makes an XmR chart "either miss exceptional variation, or wrongly detect seasonal variation as exceptional" by widening the limits, and the published fixes are deseasonalizing by seasonal factors or charting the period-over-period change. - **[TENBENSEL-2019]** — Tenbensel, Tim; Jones, Peter; Chalmers, Linda M.; Ameratunga, Shanthi; Carswell, Peter. "Gaming New Zealand's Emergency Department Target: How and Why Did It Vary Over Time and Between Organisations?" _International Journal of Health Policy and Management_ 9(4), 2020, pp. 152–162. doi:10.15171/ijhpm.2019.98. Open access, fully verified. Note the year mismatch: the journal issue is 2020 and the DOI slug is 2019; the key follows the DOI. - **[TENE-2015]** — Tene, Gil. "How NOT to Measure Latency." QCon San Francisco 2015; InfoQ recording posted March 26, 2016. Speaker is CTO and co-founder of Azul Systems. Talk title, speaker, venue, date, and abstract verified; the arguments about averages, percentile-of-percentiles, and coordinated omission are in the video and unverified at source, so cite at talk level. Latency is a different clock from cycle time; use it for the statistical point only. +- **[TREYNOR-2017]** — Treynor, Ben; Dahlin, Mike; Rau, Vivek; Beyer, Betsy (Google). "The Calculus of Service Availability." _ACM Queue_ 15(2), May 17, 2017. The journal's site refuses automated fetching (HTTP 403); read from an Internet Archive capture. Verified verbatim: "Most services offered by Google aim to offer 99.99 percent (sometimes referred to as the "four 9s") availability to users"; "An error budget is simply 1 minus a service's SLO, so the previously discussed 99.99 percent available service has a 0.01 percent "budget" for unavailability"; "If the error budget is spent, the service freezes changes ... until either the service earns back room in the budget, or the month resets"; and the year's budget at four nines, "0.01 percent of 525,600 minutes/year, or 53 minutes". The article carries no nines-to-downtime table and no figure for 99.9 percent, and it argues from serving systems with request counts, which is the scale mismatch the pattern points at. - **[TURNBULL-2015]** — Turnbull, Alex. "How We Reduced Churn by Over 71%." Groove blog, 2015, formerly at groovehq.com/blog/churn. The primary behind the widely repeated 71-percent churn story. At verification (2026-08-26) the old address redirects to the blog's new host and returns a page not found, and the Wayback Machine serves no capture of the post, so the primary is unretrievable; the figures survive only in third-party retellings, which state no baseline churn rate, period, or churn definition. Cite the disappearance and the claim's status as an untraceable first-party anecdote, never its figures. No working link exists. - **[TWITTER-2017]** — Twitter, Inc. Q3 2017 Shareholder Letter, October 26, 2017. Exhibit 99.1 to Form 8-K (accession 0001564590-17-020095). "Note About Our MAU Adjustment": Digits SMS-authentication traffic counted as MAU since Q4 2014, roughly 1-2 million users per quarter; data-retention limits on reconciling earlier periods. - **[TWITTER-2019]** — Twitter, Inc. "Twitter Announces Fourth Quarter and Fiscal Year 2018 Results," press release, February 7, 2019. Exhibit 99.2 to Form 8-K (accession 0001564590-19-002218). The mDAU definition, the stated rationale and non-comparability admission, MAU's announced retirement, and the transition quarter with both numbers. @@ -210,16 +224,21 @@ _The full Phase 0 seeding is pending; entries below were verified individually a - **[VACANTI-2023]** — Vacanti, Daniel S. "A (Very) Brief History of Little's Law." 55 Degrees blog (guest post), March 16, 2023. The provenance chain from Cobham and Morse to Little's proof, and the operations-management restatement of the law as work in progress equals throughput times cycle time. The post names the journal "Operations Review", almost certainly a slip for _Operations Research_; do not reproduce the error. - **[VARNI-2025]** — Varni, Sara. "How to Achieve Pipeline Alignment with Datadog's CMO Sara Varni." SaaStr, 2025. SaaStr's account of her first-person claims. - **[VAUPEL-YASHIN-1985]** — Vaupel, James W.; Yashin, Anatoli I. "Heterogeneity's Ruses: Some Surprising Effects of Selection on Population Dynamics." _The American Statistician_ 39(3), 1985. Abstract verified; full text paywalled, so cite at abstract level only. The general form of the sorting effect: as a cohort ages the individuals at highest risk tend to die or exit first, producing population patterns surprisingly different from the patterns for subpopulations or individuals. +- **[VEIGA-2017]** — Veiga, Allan Koch; Saraiva, Antonio Mauro; Chapman, Arthur David; Morris, Paul John; Gendreau, Christian; Schigel, Dmitry; Robertson, Tim J. "A conceptual framework for quality assessment and management of biodiversity data." _PLOS ONE_, June 28, 2017. doi:10.1371/journal.pone.0178731. Peer-reviewed and open access; read in full. Verified verbatim from the abstract: "DQ assessment and management cannot be performed if we have not clearly established the quality needs from a data user's standpoint"; and from the framework section, "it is essential to understand what "data fitness for use" means according to the data user's perspective to enable the DQ assessment and management". The domain is biodiversity records, and carrying the rule to a company's reporting numbers is the framework's extension. - **[VERNA-2025]** — Verna, Elena (guest); Rachitsky, Lenny (host). "10 growth tactics that never work." Lenny's Podcast, January 19, 2025. Full transcript on the episode page. - **[WHEELER-2010]** — Wheeler, Donald J. "Individual Charts Done Right and Wrong." Quality Digest, February 1, 2010. The right way: limits from 2.66 times the average moving range. The wrong way: limits from a global standard deviation, which "makes a strong assumption of homogeneity, which is essentially what we want to examine," so a real shift widens the limits and hides itself. - **[WHEELER-2012]** — Wheeler, Donald J. "When Should We Compute New Limits?" Quality Digest Daily, April 2, 2012. Read in full from the author's hosted manuscript. Useful limits from as few as six to ten values, referencing his prior column; Perry Regier's four-question filter for revision; recompute after a known, deliberate process change, from data collected after the change; and the warning that automatically recomputing with every new point is "unnecessary at best and may be misleading," with rolling last-30 windows ignoring that the baseline frames the question the chart asks. +- **[WHEELER-2019]** — Wheeler, Donald J. "The Keys to Quality Assurance: It takes more than a good capability ratio." Quality Digest, March 4, 2019; manuscript 345, read in full from the author's PDF. Defines the four indices in words and formula, the capability ratio as "space available" over "space required by process" with Cp = (UWSL − LWSL)/6 Sigma(X), and the centred ratio Cpk from "the distance to the nearer watershed specification, DNS". Verified verbatim, the three conditions for shipping conforming product: "you will need to have a process with more than a minimum capability; you will need to operate that process predictably; and you will need to have some way to detect when the process strays from operating on-target"; "Having a capability in the neighborhood of 1.50 or greater will give you some cushion"; and "the essence of process capability consists of a description what will be produced in the future", which is why "predictable operation is necessary to guarantee conforming product". The worked processes are manufacturing; the transfer to a pipeline's landing time is the framework's. - **[WIDJAJA-2024]** — Widjaja, Crystal. "Stop Data Theater: Use Altitude Maps." FishmanAF Newsletter (Adam Fishman), April 2, 2024. The "data theater" framing for metric reporting detached from decisions. The essay's examples are operations metrics, not active-user counts; cite the general failure only. - **[WIKIMEDIA-2015]** — Wikimedia Foundation Research and Data team. "Research: Refining the definition of monthly active editors." Wikimedia Meta-Wiki, project begun 2013, last substantively updated 2015. Live wiki page — pin a permanent revision link before quoting verbatim; the page records analysis of the definition's failure modes, not a decision to change it. +- **[WILLIAMS-2021]** — Williams, Chris; Chen, Ken; Wongsuphasawat, Krist; Tomiyama, Sylvia (Airbnb). "Visualizing Data Timeliness at Airbnb." The Airbnb Tech Blog on Medium, February 24, 2021. First-party account of SLA Tracker; direct fetching is blocked (HTTP 403), so verification was against the page rendered in a browser. Verified verbatim: "we aim to have owners of each intermediate step commit to Service Level Agreements (SLAs) for the availability of their data by a certain time. For example, the dataset owner promises that the "bookings" metric will have the latest data by 5 AM UTC, and if it is not available by this time, it is considered "late""; "No SLAs were set when we first launched the tool, and SLAs may be unneeded for datasets that are not widely consumed"; "To set a realistic SLA for a dataset, one has to take into account its entire dependency tree — sometimes comprising 100s of entities — and their SLAs"; and that the tool helps producers decide whether to "negotiate with owners of upstream datasets for earlier SLA's". Cited for the dataset owner committing the agreement and for the objective being negotiated among producers; the post does not describe a consumer's requirement being elicited. Twelve months to build; not a small-team recipe. - **[WINTERS-2017]** — Winters, Casey. "Why Onboarding is the Most Crucial Part of Your Growth Strategy." Medium, June 27, 2017. - **[WINTERS-2021]** — Winters, Casey (then CPO at Eventbrite; previously growth at Pinterest and Grubhub). "Casey's Guide to Finding Product/Market Fit." caseyaccidental.com, April 13, 2021. The insufficiency condition: a flattened retention curve alone is not product-market fit, because a company that is not growing definitely does not have it, so product-market fit cannot be measured by retention alone. - **[WORDSTREAM-2026]** — Marino, Susie. "Google Ads Benchmarks 2026: Competitive Data & Insights for Every Industry." WordStream by LocaliQ, last updated May 19, 2026. Tenth annual search-advertising benchmark, computed from the vendor's own client campaigns. States its methodology: "This report is based on a sample of 13,474 US-based search advertising campaigns running between Apr 1, 2025, and March 31, 2026. Each subcategory includes at minimum 52 unique active campaigns. 'Averages' are technically median figures to account for outliers." Figures used on the benchmarks page: $5.42 average cost per click and $66.69 average cost per lead across industries, with industry cost-per-lead medians from $26.84 (Arts & Entertainment) to $131.63 (Attorneys & Legal Services). Caveats: the sample is one vendor's client base, skewing to small US advertisers; a lead is what the ad platform counts as one; and the publisher, part of Gannett's LocaliQ marketing-services arm, sells the advertising services being benchmarked. The live page sat behind a bot check at verification (2026-08-27) and was read via an Internet Archive capture. - **[WRETBLAD-2024]** — Wretblad, Niklas; Riseby, Fredrik Gordh; Biswas, Rahul; Ahmadi, Amin; Holmström, Oskar. "Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark." ACL 2024 (short papers), pp. 356–369; arXiv:2402.12243, read in the arXiv HTML. Verified: the financial domain's development set holds 106 question-and-query pairs, of which 52 (49 percent) carried noise, 44 in the question and 22 with an erroneous gold query; and "when evaluating models on corrected SQL queries, zero-shot baselines surpassed the performance of state-of-the-art prompting methods". The size of the resulting performance underestimation is read off a figure and is not quoted here. One domain of one benchmark, with 2024-era models. +- **[WRIGHT-2023]** — Wright, Clark (Airbnb). "Data Quality Score: The next chapter of data quality at Airbnb." The Airbnb Tech Blog on Medium, November 29, 2023. First-party; direct fetching is blocked (HTTP 403), so verification was against the page rendered in a browser, with the score weights read from the post's figures. Verified verbatim: four dimensions, "Accuracy, Reliability (Timeliness), Stewardship, and Usability", weighted 40, 15, 30 and 15 of 100 points in the figure; the Reliability score "covers whether there are data landing SLA expectations for the data asset, and how consistently the asset meets those expectations", with 7 points for "meeting SLA hit rate for the last 28 days" and 8 for "defined SLAs"; the remediation text "Improve SLA hit rate by improving landing times or setting a more realistic SLA"; the objective to move "beyond a simple binary definition (certified vs uncertified)"; and, on consumers, dimensional scores are "Useful when a particular area of deficiency is not problematic (e.g., the consumer wants the data to be very accurate but is not worried about it landing quickly every day)". Cited for the "more realistic SLA" line, which states the collapse the pattern argues against in a platform's own words, and for consumers' differing requirements. - **[YAKUBENKOV-2019]** — Yakubenkov, Oleg. GoPractice, October 2, 2019. Classic Day-N retention against rolling or unbounded retention, and calendar days against rolling 24-hour windows. Hard number: for one app, day-1 retention is 32 percent on 24-hour windows, 43 percent on calendar days, or 59 percent on the rolling definition, from the same app, the same users and the same day. Also notes that rolling retention produces a curve that can never decrease, a definitional artifact that can masquerade as a smile. Caveat: GoPractice sells paid courses, so the author is commercially interested, though not an analytics vendor. [TODO: verify] the exact article title. +- **[YING-2021]** — Ying, Maggie; Yan, Wei; Zou, Ying; Sundaresan, Sanjay; Geracioti, Isabel; Chintalapani, Sriharsha (Uber). "How Uber Achieves Operational Excellence in the Data Quality Experience." Uber Engineering Blog, August 5, 2021. First-party; read in full. Verified verbatim, the five test categories: freshness is "the delay after which data is 99.9% complete"; completeness is "the row completeness percentage, computed by comparing this dataset to one of its upstream datasets"; duplicates, cross-datacenter consistency, and "Others"; "SLAs are used as thresholds as part of assertion"; "All datasets at Uber are categorized with tiering levels (Tier 0 being the highest to Tier 5 being the lowest), which we use as criteria to define default SLA values, following Uber's business requirements"; the sustain period, "a sustain period indicating the table SLA that allows for test failures", with a four-hour example; and the coverage claim, "over 2,000 critical datasets on this platform, and detected around 90% of data quality incidents". Cited for the platform setting default objectives by tier, for a tolerated failure period expressed in hours, and for a freshness definition with a nine inside it; the 90 percent is the platform's own count against incidents it knows about. - **[YU-2021]** — Yu, Will; Wu, Yun; Wang, Xiaodong; Meng, Wenrui (Uber). "The Journey Towards Metric Standardization." Uber Engineering Blog, January 12, 2021. The uMetric account, read in full. Verified stories and figures: the Shopping Session metric read 6.53M in one internal tool and 6.20M in another for the same city and time duration, root-caused to stale filters in one tool's static query; popular metrics spun off "10X or 100X" named instances; the standardization target is that "a metric and the business logic have a strictly ONE to ONE mapping without ambiguities during the entire lifecycle," built working with 30+ teams and 10K+ metrics. Also carries the enforcement machinery: SQL-equivalence deduplication via canonical transformation (with SQL equivalence noted as NP-complete, hence heuristic), organization-level Verification Committees the post itself calls time-consuming, tiered governance, and definitions kept lean by having consumers supply dimensions and filters at run time. Caveat: first-party platform advocacy; no adoption or accuracy outcomes are published. - **[ZHANG-2024]** — Zhang, Hugh; Da, Jeff; Lee, Dean; et al. (Scale AI). "A Careful Examination of Large Language Model Performance on Grade School Arithmetic." arXiv:2405.00332, submitted May 1, 2024, read at v4 (November 22, 2024): abstract, construction section, and first page. The GSM1k contamination audit: a new test set of 1,205 problems, built by fresh human annotation to mirror the established GSM8k benchmark despite the 1000 in the name; scoring existing models on it produced "accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting", while models at the frontier showed minimal signs of overfitting. Cited to show that a benchmark with published tasks can be audited from outside. The authors work at an AI evaluation vendor, so the audit is not disinterested; the finding cited here does not favor the vendor's products. - **[ZHAO-2022]** — Zhao, Xinshu; Feng, Guangchao Charles; Ao, Song Harris; Liu, Piper Liping. "Interrater reliability estimators tested against true interrater reliabilities." _BMC Medical Research Methodology_ 22, article 232, 2022. Read from the PubMed Central copy. A simulation study that tests chance-corrected indices against known true reliabilities. Verified: "Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement"; in the tested conditions the kappa family (Scott's pi, Cohen's kappa and Krippendorff's alpha) "were crude predictors of reliability and evenness, and 31-point under-approximators of reliability"; and Gwet's AC1 was "the most accurate approximator". Cited for the under-estimation of kappa under skew. The framework's use of raw agreement for numeric values within a tolerance, where chance agreement is negligible, is its own reading and not this paper's. diff --git a/docs/modules/01-ai-data-quality/README.md b/docs/modules/01-ai-data-quality/README.md index 5d042d3..3e85029 100644 --- a/docs/modules/01-ai-data-quality/README.md +++ b/docs/modules/01-ai-data-quality/README.md @@ -44,7 +44,7 @@ Whether the number the pipeline returns is the number the definition asks for. R | M1-04 | [Golden question sets for AI analytics validation](golden-question-sets.md) | v0.1.0 | | M1-07 | [Schema drift detection and data contracts](schema-drift-and-data-contracts.md) | v0.1.0 | | M1-05 | [Agreement measurement for analytics QA](agreement-measurement.md) | v0.2.0 | -| M1-06 | Data quality SLOs and error budgets | v0.2.0 | +| M1-06 | [Data quality SLOs and error budgets](data-quality-slos.md) | v0.2.0 | | M1-08 | The reconciliation protocol | v0.2.0 | | M1-09 | Measuring trust, not just accuracy | v0.2.0 | | M1-10 | Data incident root cause analysis | v0.2.0 | diff --git a/docs/modules/01-ai-data-quality/data-quality-slos.md b/docs/modules/01-ai-data-quality/data-quality-slos.md new file mode 100644 index 0000000..aa183e1 --- /dev/null +++ b/docs/modules/01-ai-data-quality/data-quality-slos.md @@ -0,0 +1,202 @@ +--- +id: M1-06 +title: Data quality SLOs and error budgets +module: 1B +type: pattern +status: drafted # full sourced draft; awaiting author voice pass per AGENTS.md constraint 6 +sources: + - AHMED-2020 + - BENKOVA-2024 + - BEYER-2016 + - BEYER-2018 + - CICHY-RASS-2019 + - DBT-FRESHNESS + - GARTNER-2021 + - HIDALGO-2020 + - MCPHILLIPS-2023 + - MONTECARLO-2023 + - MONTGOMERY-2019 + - NIST-HANDBOOK + - PARKS-2020 + - QUOSS-2020 + - REDMAN-2016 + - SCHELTER-2018 + - SHANMUGAM-2020 + - SRINIVAS-2021 + - TREYNOR-2017 + - VEIGA-2017 + - WHEELER-2019 + - WILLIAMS-2021 + - WRIGHT-2023 + - YING-2021 +template: ../../../templates/01-ai-data-quality/slo-register.md +summary: > + A service-level objective for a number is the requirement of the person + who reads it: how fresh it has to be, how complete, how close to right, + and how many bad days a quarter they will live with. That count is the + error budget. Set it with the reader, as a count, and keep it apart from + the control limits on the pipeline, which describe what the pipeline + does and belong to nobody's requirement. Put the two side by side and + the comparison says whether the pipeline can meet the requirement at + all, which neither an alert threshold nor a chart can say on its own. +keywords: + - service level objective + - SLO + - SLA + - error budget + - process capability + - specification limit + - control limit + - freshness + - completeness + - accuracy + - landing time +--- + +# Data quality SLOs and error budgets + +## Problem + +Most alert thresholds on a data pipeline are neither of the two numbers that matter. They are not what the pipeline does, because nobody computed that from its history, and they are not what the reader needs, because nobody asked. The threshold pages for lateness the reader would never notice and sleeps through the morning that cost them a decision. + +My field has kept the two numbers apart since the 1930s. A control limit is what the process does, computed from its own history, and it belongs to nobody's requirement. A specification limit is what the customer needs, and the customer sets it. Comparing them is process capability: a stable process's output is held against the specification, and the question is whether almost all of it falls inside.[^nist-handbook] + +Data teams collapse the two. Some call the pipeline's usual landing time its objective, or type in a threshold that is neither. Others copy the nines from site reliability practice, where 99.99 percent suits a service answering millions of requests,[^treynor-2017] onto a table that lands once a day. None of these asked the person who reads the number. + +At a company with no analyst, nobody sits between the person who runs the pipeline and the person who reads it, so the requirement is never written down and every late morning is renegotiated on the spot. In my own practice this loses more trust than a wrong number does. No citable figure for what a bad day costs survived checking.[^gartner-2021] [^redman-2016] [^montecarlo-2023] + + + +## When this applies + +Reach for this pattern when somebody reads a number on a schedule and nobody has written down what late, short or wrong would cost them. The problems that bring teams here look like these: + +- The Monday number was late again, and nobody can say whether late means once a month or once a week. +- A freshness alert fires most mornings, and nobody on call opens it. +- The team calls a table's usual landing time its SLA, and the dashboard went red the first week the load slowed. +- Somebody proposed 99.9 percent for a daily table, and nobody could say what a tenth of a percent of a morning is. +- A table has not missed its target in a year, and nobody is sure anyone still reads the number. + +It needs one reader per number, one owner who can say when the load lands, and a control chart on the landing time from [statistical process control](statistical-process-control-for-pipelines.md). It needs no analyst and no purchase. + +Whether the landing time moved or only wiggled is that page's question. Whether an input changed shape or meaning belongs to [schema drift](schema-drift-and-data-contracts.md). Whether one number is right before it leaves the room lives on [detecting plausible-but-wrong outputs](plausible-but-wrong.md). A number has one owner to hold an objective once [single point of metric computation](single-point-of-metric-computation.md) has given it one computation. + +Whenever another page in this library asks how fresh, how complete or how close to right a number has to be, and how many bad days a quarter its reader will live with, this is the page it points to. + +## The pattern + +A service-level objective for a number is the requirement of the person who reads it, in their words: how fresh it has to be, how complete, how close to right, and how many bad periods a quarter they will live with. That count is the error budget. The objective is set with the reader and kept apart from the control limits on the pipeline, and comparing the two says whether the pipeline can meet the requirement at all. + +The vocabulary is site reliability engineering's: an indicator is the measured thing, an objective is its target, and the budget is the gap between the objective and perfection.[^beyer-2016] The books ask of the target what I ask here. The product manager has to agree the threshold is good enough for users,[^beyer-2018] and a target picked from current performance is warned against in so many words.[^beyer-2016] + +The three dimensions are the ones the survey literature finds most common, and its definitions of two of them end in the words "for the task at hand".[^cichy-rass-2019] The task is the reader's, and quality cannot be assessed until the user's needs are known.[^veiga-2017] + +| Dimension | The reader's question | What is measured | Published form | +|---|---|---|---| +| Fresh | Is it here when I look? | Landing time of the feeding load, against the reader's deadline | Timeliness, the extent to which the age of the data suits the task;[^cichy-rass-2019] the share of data updated more recently than a threshold[^beyer-2018] | +| Complete | Is all of it here? | Share of expected sources present, and the row count on its chart | Completeness, sufficient breadth, depth and scope for the task;[^cichy-rass-2019] the fraction of non-missing values in a column[^schelter-2018] | +| Right | Is the number correct? | Reconciliation to an independent total, within a tolerance the reader set | Accuracy, data that are correct, reliable and certified;[^cichy-rass-2019] the share of records that produced the correct value[^beyer-2018] | + +Process capability sits underneath: the specification comes from the customer, the control limits from the process's own data,[^benkova-2024] and Wheeler's conditions for shipping conforming product are a capability above one, predictable operation, and a way to detect when the process strays.[^wheeler-2019] For a pipeline: a requirement it can meet with margin, a control chart, and a reaction plan.[^montgomery-2019] + +```mermaid +flowchart LR + R["The reader says what
they need, as bad
days a quarter"] --> S["The objective
and its budget"] + P["The pipeline's own
history: the
control chart"] --> C{"Requirement outside
the limits?"} + S --> C + C -->|yes| OK["Capable: a bad day
is a special cause"] + C -->|no| NO["Not capable: routine
variation spends
the budget"] +``` + +## Position + +Set the objective with the person who reads the number, as the number of bad days a quarter they will live with, and keep it apart from the control limits, which describe the pipeline and belong to nobody's requirement. + +The practice this argues with is the site-reliability objective as data teams import it: the engineer sets the target from what the system achieves and expresses it in nines. I understand why it travels: the books are good, the nines are familiar, and the workbook allows that current performance is a fair place to start with nothing else to go on.[^beyer-2018] The data platforms took the same route, with default objectives by tier,[^ying-2021] a platform-wide eight hours for freshness,[^ahmed-2020] and a score that credits a table for setting a more realistic objective.[^wright-2023] + +My objection is that the import drops the one thing the books insist on. The first book says not to pick a target from current performance, because it locks you into whatever the system does now,[^beyer-2016] and Hidalgo's test is whether users are happy above the target and unhappy below it.[^hidalgo-2020] A team that sets the target from the pipeline has dropped the customer from the method. Nor do the nines survive the change of scale. A service answers millions of requests a month, so 99.9 percent is a budget of thousands of failures,[^beyer-2018] and where a nine appears in a platform's freshness check it counts rows.[^srinivas-2021] A daily table has about sixty-five readings a quarter, and a tenth of a percent of that is not a morning. A reader can say three bad mornings, and that is the number to write down. + +Two takeaways follow. A budget nobody ever spends is a requirement set too loose or a number nobody reads. A budget spent by the second week is a capability problem, and the chart shows whether the pipeline can meet the requirement at all. + +## Implementation + +The [SLO register](../../../templates/01-ai-data-quality/slo-register.md) carries one row per number that leaves the room, the arithmetic, what happens when a budget is spent, a machine-readable core, and the prompts. I would fill the first row before buying or building anything. + +1. **Start with the number your most senior reader looks at.** One row: the number, the reader, and when they read it. +2. **Ask the reader three questions, and write the third down as a count.** When do you look at it? What do you do differently if it is late, short or wrong? How many such days a quarter could you live with? The count is the budget; the percentage derived from it is never what the reader agreed to. +3. **Translate the requirement into an indicator per dimension.** Fresh is the landing time of the feeding load, against the reader's deadline rather than a tool's default age.[^dbt-freshness] Complete is the share of expected sources present, plus the row count on the [SPC control plan](../../../templates/01-ai-data-quality/spc-control-plan.md). Right is the reconciliation from the [plausibility checks](plausible-but-wrong.md), and running it on a schedule is its own pattern. +4. **Put the control chart beside the requirement.** From the chart's mean and average moving range, read the gap between the nearest natural limit and the requirement. The register turns it into the one-sided capability index.[^nist-handbook] [^wheeler-2019] Above about one and a half, a bad day is news. Below one, routine variation will spend the budget. +5. **Decide the reaction before the first bad day.** A bad day with a chart signal goes to the SPC reaction plan, and the reader hears from the owner first. A spent budget with a quiet chart is a capability problem, fixed in the process or the requirement, never in the limits. Repeated special causes are root-cause work. +6. **Review each row with its reader once a quarter.** A budget untouched for two quarters gets one question: would you notice a bad day? + +The table is illustrative: one requirement, two pipelines, an average moving range of 22 minutes. + +| | The requirement (the reader's) | Pipeline A (the chart's) | Pipeline B (the chart's) | +|---|---|---|---| +| Landed by | 09:00 every weekday, with data through yesterday | | | +| Bad mornings the reader will live with | 3 of about 65 a quarter, or 95.4 percent on time | | | +| Average landing time | | 06:40 | 08:50 | +| Natural process limits | | 05:41 to 07:39 | 07:51 to 09:49 | +| Upper limit against the deadline | | 81 minutes early | 49 minutes late | +| Capability margin | | 2.4 | 0.2 | +| Late mornings expected from routine variation | | none | about 20 a quarter | +| Reading | | Capable. A late morning is a special cause, and the chart will show it. | Not capable. The third late morning arrives in the second week, and the chart shows nothing, because nothing changed. | + +The limits sit 2.66 times the average moving range either side of the mean, 59 minutes. B's late share is the bell-curve tail past 09:00 with a sigma of 22 divided by 1.128, about 19.5 minutes: three mornings in ten. B's alert would be muted, B's chart would show nothing, and neither would say that the pipeline cannot meet the requirement. The fix is the schedule, the source or the scope, or a 10:00 deadline agreed with the reader. Widening the limits changes nothing. + + + +## How you know it is working + +- Every number that leaves the room has a row with a reader's name, a count of bad days, and the date they agreed to it. +- The reader hears about a bad day from the owner first, as the count spent against the count agreed. +- Budgets get spent, a little. +- **Anti-signal:** a table's objective equals its average landing time, or a nine. The pipeline set its own requirement. + +## Failure modes + +- **The objective is the pipeline's habit.** The table lands at 06:40, so the objective becomes 07:00, and a target copied from the process locks the team into supporting it.[^beyer-2016] Ask the reader. +- **Nines on a daily table.** A tenth of a percent of sixty-five mornings is a fraction of a morning. Count the bad days. +- **The threshold that is neither.** An alert age typed in to feel safe is not the reader's requirement and not the pipeline's limit, and one platform found such alerts too noisy for everyday use.[^shanmugam-2020] +- **A budget the reader never agreed to.** An objective set by the owner alone is the practice this page argues with, in the register's format. Airbnb has consumers review the design spec before the pipeline is built;[^quoss-2020] at ten people that is one conversation. +- **Objectives on every table.** The reliability rule is as few objectives as possible,[^beyer-2016] and Airbnb's tracker launched with none set.[^williams-2021] Fifty rows is a register nobody reads. +- **Treating a right row like a fresh row.** A late number can leave the room with a note. A wrong number does not leave until it reconciles. + +## Sources & Stories + +The objective and budget vocabulary is Google's: the site-reliability book's chapters on embracing risk and on objectives [^beyer-2016], the workbook's chapters on implementing objectives and on pipelines [^beyer-2018], and the availability paper in ACM Queue [^treynor-2017]. Hidalgo is cited for his chapter on choosing objectives; his book's guest-written data chapter is paywalled and was not read [^hidalgo-2020]. + +The capability lens is the author's field: the government statistics handbook for the definition and indices [^nist-handbook], a 2024 open-access article for the customer origin of specifications and the control-first rule [^benkova-2024], Wheeler's column for the three conditions [^wheeler-2019], and the textbook at chapter level [^montgomery-2019]. + +The platform stories are first-party. Airbnb's certification posts [^parks-2020] [^quoss-2020], landing-time tracker [^williams-2021] and quality score [^wright-2023], and Lyft's checks platform [^mcphillips-2023], were read in a browser because Medium refuses fetching; Uber's quality platform [^ying-2021], data-culture post [^srinivas-2021] and monitoring account [^shanmugam-2020], and Shopify's platform post [^ahmed-2020], were read in full. None states an objective as a count of allowed misses. The dimension definitions are from the peer-reviewed survey of twelve frameworks [^cichy-rass-2019], completeness as a fraction is Amazon's [^schelter-2018], the user's-standpoint rule is from a biodiversity-data paper and the transfer is the framework's [^veiga-2017], and the freshness check is a transformation vendor's documentation [^dbt-freshness]. + +No figure for the cost of a bad day survived checking: the consultancy's average is a self-estimate by vendor-nominated reference customers [^gartner-2021], the trillion-dollar figure ends at an undated infographic with no method [^redman-2016], and the incident counts are a vendor-commissioned survey of 200 people [^montecarlo-2023]. + +The count of bad days as the unit, the three reader questions, the one-sided margin, and the two takeaways on unspent and early-spent budgets are the framework's own. The prior-art row is pending in [prior-art.md](../../prior-art.md). + + + +[^ahmed-2020]: [[AHMED-2020]](../../../REFERENCES.md) +[^benkova-2024]: [[BENKOVA-2024]](../../../REFERENCES.md) +[^beyer-2016]: [[BEYER-2016]](../../../REFERENCES.md) +[^beyer-2018]: [[BEYER-2018]](../../../REFERENCES.md) +[^cichy-rass-2019]: [[CICHY-RASS-2019]](../../../REFERENCES.md) +[^dbt-freshness]: [[DBT-FRESHNESS]](../../../REFERENCES.md) +[^gartner-2021]: [[GARTNER-2021]](../../../REFERENCES.md) +[^hidalgo-2020]: [[HIDALGO-2020]](../../../REFERENCES.md) +[^mcphillips-2023]: [[MCPHILLIPS-2023]](../../../REFERENCES.md) +[^montecarlo-2023]: [[MONTECARLO-2023]](../../../REFERENCES.md) +[^montgomery-2019]: [[MONTGOMERY-2019]](../../../REFERENCES.md) +[^nist-handbook]: [[NIST-HANDBOOK]](../../../REFERENCES.md) +[^parks-2020]: [[PARKS-2020]](../../../REFERENCES.md) +[^quoss-2020]: [[QUOSS-2020]](../../../REFERENCES.md) +[^redman-2016]: [[REDMAN-2016]](../../../REFERENCES.md) +[^schelter-2018]: [[SCHELTER-2018]](../../../REFERENCES.md) +[^shanmugam-2020]: [[SHANMUGAM-2020]](../../../REFERENCES.md) +[^srinivas-2021]: [[SRINIVAS-2021]](../../../REFERENCES.md) +[^treynor-2017]: [[TREYNOR-2017]](../../../REFERENCES.md) +[^veiga-2017]: [[VEIGA-2017]](../../../REFERENCES.md) +[^wheeler-2019]: [[WHEELER-2019]](../../../REFERENCES.md) +[^williams-2021]: [[WILLIAMS-2021]](../../../REFERENCES.md) +[^wright-2023]: [[WRIGHT-2023]](../../../REFERENCES.md) +[^ying-2021]: [[YING-2021]](../../../REFERENCES.md) diff --git a/llms.txt b/llms.txt index e41064a..1218879 100644 --- a/llms.txt +++ b/llms.txt @@ -24,6 +24,7 @@ Every page opens with the problems it is for, in the reader's words. The short v - [Single point of metric computation](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/docs/modules/01-ai-data-quality/single-point-of-metric-computation.md): the same metric returns different numbers in different tools, and nobody can say which copy is right. A number is quoted outside the team that produces it, in a deck or by an AI assistant. Template: [Metric-computation control plan](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/templates/01-ai-data-quality/metric-computation-control-plan.md). - [Detecting plausible-but-wrong outputs](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/docs/modules/01-ai-data-quality/plausible-but-wrong.md): a revenue or usage figure in a deck that nobody has tied to the billing system. A total that jumped after a model change added a join. A metric that has been quietly counting test accounts or refunded orders. An assistant returns a fluent answer and a query, and nobody can say whether the join is right. Template: [Plausibility-check control plan](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/templates/01-ai-data-quality/plausibility-check-control-plan.md). - [Schema drift detection and data contracts](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/docs/modules/01-ai-data-quality/schema-drift-and-data-contracts.md): the number moved last week and nobody touched the query. A dashboard went blank because a column was renamed. A status field grew a new value and the report files it under other. A field started arriving empty and the total drifted down. Somebody backfilled a table and last quarter no longer matches the deck. Template: [Schema contract control plan](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/templates/01-ai-data-quality/schema-contract-control-plan.md). +- [Data quality SLOs and error budgets](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/docs/modules/01-ai-data-quality/data-quality-slos.md): the Monday number was late again and nobody can say whether late means once a month or once a week. A freshness alert fires most mornings and nobody opens it. A table's usual landing time is being called its SLA. Somebody proposed 99.9 percent for a daily table. A target has not been missed in a year and nobody is sure anyone reads the number. Template: [SLO register](https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/templates/01-ai-data-quality/slo-register.md). ### An AI assistant answers questions about your data diff --git a/templates/01-ai-data-quality/slo-register.md b/templates/01-ai-data-quality/slo-register.md new file mode 100644 index 0000000..642007f --- /dev/null +++ b/templates/01-ai-data-quality/slo-register.md @@ -0,0 +1,163 @@ +# SLO register for the numbers that leave the room + +Control artifact for the pattern [Data quality SLOs and error budgets (M1-06)](../../docs/modules/01-ai-data-quality/data-quality-slos.md). MIT licensed — copy this file into your own repository, replace every `[TODO: …]` field, and delete the guidance blockquotes. The objective-and-budget vocabulary follows the site-reliability books [BEYER-2016] [BEYER-2018], and the comparison of what the pipeline does against what the reader needs follows the process-capability chapter of the standard quality-control textbook and Wheeler's working form [MONTGOMERY-2019] [WHEELER-2019] (see [REFERENCES.md](../../REFERENCES.md)); everything organization-specific is a `[TODO]`. + +> **How to use with an AI analytics agent:** section 1 is the register, section 2 is the arithmetic, section 3 says what happens when a budget is spent, section 4 is the machine-readable core, section 5 is the prompt library, and section 6 stays with the human owner. The smallest complete implementation is one row: the one number your most senior reader looks at, that reader's name, the time they look, and the number of bad mornings a quarter they told you they would live with. +> +> **What this register does not hold:** the control limits. Those describe what the pipeline does and live in the [SPC control plan](spc-control-plan.md). This register holds what the reader needs. The two are compared in section 2 and never merged. + +## 1. The register + +One row per number and per dimension. Most numbers need a `fresh` row first and a `right` row second; add a `complete` row only where a source can drop out without anything failing. The requirement column is written in the reader's words, by the reader, and the owner translates it into an indicator afterwards. + +| # | Number | Reader, and when they read it | Dimension | Requirement, in the reader's words | Window | Bad periods allowed | Indicator (what is measured to call a period bad) | +|---|---|---|---|---|---|---|---| +| 1 | `[TODO: the number]` | `[TODO: name]`, `[TODO: e.g. every weekday at 09:00]` | fresh | Landed by `[TODO: time]` with data through `[TODO: e.g. yesterday]` | Rolling quarter, about `[TODO: 65]` readings | `[TODO: e.g. 3]` | Landing time of the load that feeds it, read at `[TODO: time]` | +| 2 | same | same | complete | Every expected source present for the period, and the count inside the limits | Same | `[TODO]` | Share of expected sources landed, and the row count from the [SPC control plan](spc-control-plan.md) | +| 3 | same | same | right | Reconciles to `[TODO: the independent total, e.g. the billing statement]` within `[TODO: tolerance]` | Same | `[TODO]` | The reconciliation check from the [plausibility-check control plan](plausibility-check-control-plan.md) | +| 4 | `[TODO: the next number]` | `[TODO]` | `[TODO: fresh, complete, right]` | `[TODO]` | `[TODO]` | `[TODO]` | `[TODO]` | + +> Three questions get a row filled with the reader in ten minutes. When do you look at this number? What do you do differently if it is late, short, or wrong that day? How many such days a quarter could you live with? Write the third answer down as a count. The percentage in section 2 is derived from it and is never the thing the reader agreed to. +> +> A requirement copied from what the pipeline usually does is not a requirement. If the reader cannot say what a bad day costs them, the number may not need a row yet. + +## 2. The arithmetic + +Spreadsheet-ready. One row of the register, one column of daily indicator readings, and everything below follows. + +1. **The budget.** `budget = bad_periods_allowed`, straight from the register. The share the site-reliability books would print is `slo_share = 1 - budget / periods_in_window`; three bad mornings in 65 is 95.4 percent. Keep the count as the number people talk about, because 3 is what the reader said and 95.4 is what a spreadsheet said. +2. **Period status.** For each period, `bad = indicator outside the requirement`, for example `landing_time > deadline`, `sources_landed < sources_expected`, or `ABS(number - independent_total) > tolerance`. A period is bad or not; there is no partial credit. +3. **Budget spent and remaining.** Over a rolling window of an integral number of weeks, so every window holds the same number of weekends [BEYER-2018]: `spent = COUNTIF(status, "bad")` and `remaining = budget - spent`. +4. **Pace.** `pace = spent / weeks_elapsed`. When `pace * 13 > budget`, the budget will not last the quarter, and the reader hears that now rather than in week thirteen. This is the framework's plain form of what the reliability books call a burn rate [BEYER-2018]. +5. **Capability.** Take the chart on the indicator from the [SPC control plan](spc-control-plan.md): its `mean` and its `avg_moving_range`, and `sigma = avg_moving_range / 1.128`. For a requirement the indicator must stay below, such as a deadline, `margin = (requirement - mean) / (3 * sigma)`; for one it must stay above, such as a share of sources, `margin = (mean - requirement) / (3 * sigma)`. This is the one-sided capability index of the quality field, Cpk, written for a single limit [WHEELER-2019]. Read it in the reader's units first: the gap between the natural process limit nearest the requirement and the requirement itself, in minutes or in rows. +6. **What the margin says.** Above about 1.5, the pipeline meets the requirement with room to spare and a bad period is a special cause worth investigating. Between 1 and 1.5, it meets the requirement only while it stays predictable and centred, and Wheeler's advice is to want the cushion [WHEELER-2019]. Below 1, the requirement sits inside the pipeline's own limits, routine variation spends the budget, and no alert will change that. The rough share of bad periods to expect is `1 - NORM.S.DIST((requirement - mean) / sigma, TRUE)` for an upper requirement, assuming day-to-day variation is roughly bell-shaped; multiply by the periods in the window for a count. + +Worked, illustrative, for the freshness row: deadline 09:00, budget 3 of 65 mornings. A load that lands at 06:40 on average with an average moving range of 22 minutes has natural limits of 05:41 and 07:39, a margin of 2.4, and 81 minutes between its upper limit and the deadline; a late morning there is news. A load that lands at 08:50 on average with the same moving range has limits of 07:51 and 09:49, a margin of 0.2, and lands late on about three mornings in ten, so the budget of three is gone in the second week while the chart shows nothing, because nothing changed. + +## 3. What happens when a budget is spent + +| Reading | What it means | What to do | Within | +|---|---|---|---| +| A bad period, and the chart on the indicator shows a point outside its limits that day | A special cause: something changed | The SPC reaction plan runs; the reader hears before they read the number, with the cause once found | Same day | +| A bad period, and the chart is quiet | Routine variation crossed the requirement | Nothing to fix today; check the margin in section 2 | Same day | +| Budget spent, and the chart is quiet | A capability problem: the requirement sits inside the pipeline's limits | Change the process (the schedule, the source, the scope of the table) or renegotiate the requirement with the reader. Do not move the limits. | `[TODO: e.g. two weeks]` | +| Budget spent, and the chart signalled on the bad days | Repeated special causes | Each bad day has a found cause in the signal log; the pattern across them goes to root cause work | `[TODO: e.g. one week]` | +| Pace says the budget will not last the quarter | The reader is going to be surprised in week ten | Tell the reader now, with the count, and agree what changes first | Same week | +| Budget untouched for two consecutive quarters | Either the requirement is too loose or nobody reads the number | Ask the reader whether they would notice a bad day; tighten the row or retire it | At the quarterly review | +| A `right` row goes bad | The number is wrong, whatever the dashboard says | The number does not leave the room until it reconciles; the [plausibility checks](plausibility-check-control-plan.md) say how | Before the reader's next reading | + +> The budget is the reader's, not the owner's. Spending it is not a failure and hoarding it is not a success. It is the agreed number of days a quarter the reader has said they will absorb, and the register exists so that nobody has to renegotiate that on a bad morning. + +## 4. Machine-readable core + +```json +{ + "spec": "slo-register/v1", + "owner": "[TODO: name]", + "window": { "kind": "rolling_quarter", "weeks": 13, "periods": "[TODO: e.g. 65 weekday readings]" }, + "rows": [ + { + "id": "[TODO: e.g. weekly_revenue_fresh]", + "number": "[TODO: the number, as the reader names it]", + "reader": { "name": "[TODO]", "reads_at": "[TODO: e.g. weekdays 09:00]" }, + "dimension": "[TODO: fresh | complete | right]", + "requirement": { + "text": "[TODO: in the reader's words]", + "kind": "[TODO: upper_limit | lower_limit | tolerance]", + "value": "[TODO: e.g. 09:00, or 1.0 as a share, or 0.005 as a relative tolerance]" + }, + "budget": { "bad_periods_allowed": "[TODO: e.g. 3]", "slo_share": "[TODO: derived, 1 - budget / periods]" }, + "indicator": { + "signal_id": "[TODO: the matching signal id in the SPC control plan, or null]", + "source": "[TODO: where the daily reading comes from]" + }, + "capability": { + "mean": "[TODO: from the chart]", + "avg_moving_range": "[TODO: from the chart]", + "sigma": "[TODO: avg_moving_range / 1.128]", + "margin": "[TODO: (requirement - mean) / (3 * sigma), sign per section 2]", + "expected_bad_periods": "[TODO: rough count from section 2, step 6]" + }, + "status": { + "as_of": "[TODO: date]", + "spent": "[TODO]", + "remaining": "[TODO]", + "pace_per_week": "[TODO]", + "reading": "[TODO: in_budget | special_cause | capability_problem | untouched]" + }, + "agreed": { "with": "[TODO: the reader's name]", "on": "[TODO: date]" } + } + ], + "policy": { + "capability_problem": "change_process_or_renegotiate_requirement; never move the limits", + "special_cause": "spc_reaction_plan", + "untouched_two_quarters": "ask_reader_then_tighten_or_retire", + "right_row_bad": "number_does_not_leave_the_room" + }, + "sources": ["BEYER-2016", "BEYER-2018", "MONTGOMERY-2019", "WHEELER-2019"] +} +``` + +> The JSON restates sections 1 to 3 in a form any agent can parse. Keep them in step in the same edit; the JSON wins when the prose is ambiguous. The control limits are read from the SPC control plan and copied here for the capability arithmetic; they are edited there, never here. + +## 5. Agent prompt templates + +### Prompt A — fill a row with the reader + +```text +You are helping the owner of a data quality register interview the person +who reads one number. Ask these questions one at a time and wait for each +answer: +1. When do you look at this number, and what do you decide from it? +2. If it were late one morning, what would you do differently? If it + were short (a source missing)? If it were wrong? +3. How many such days in a quarter could you live with before you stopped + trusting it or had to redo something? +Then draft one register row per dimension the reader cares about, with the +requirement written in the reader's own words and the budget as a count of +bad periods. Do not propose a percentage, and do not describe what the +pipeline currently does. The owner reviews the row with the reader and +commits it. + + +[paste section 4 JSON here, with existing rows] + +``` + +### Prompt B — the quarterly reading + +```text +You are an analytics agent reading an SLO register against the daily +indicator series and the chart baselines provided. For each row: +1. Mark each period bad or not against the requirement, and compute spent, + remaining, and pace exactly as in section 2. Show the arithmetic. +2. Using the chart's mean and average moving range, compute sigma, the + margin, and the rough expected bad periods. +3. Classify the row: in budget; special cause (bad periods coincide with + chart signals); capability problem (budget spent or pacing to be spent + with a quiet chart); or untouched for two quarters. +4. Draft a three-sentence note to the reader named on the row: the count + spent against the count agreed, what the chart says about why, and + what the owner proposes. Do not change any requirement, budget, or + limit; propose the change and stop. + + +[paste section 4 JSON here] + + + +[paste dates and daily readings per row, plus the chart baseline from the +SPC control plan] + +``` + +## 6. Ownership and review + +| Item | Value | +|---|---| +| Owner (one named person) | `[TODO: name]` | +| Where this register lives | `[TODO: repository path]` | +| Reviewed with each reader | `[TODO: e.g. quarterly, in the reader's calendar]` | +| A change to a requirement or a budget | An edit to this file with a date and the reader's name, reviewed like code | +| The control limits | Never edited here. They belong to the [SPC control plan](spc-control-plan.md) and change only after a confirmed process change. | +| Standing check | If a row's budget has not been touched in two quarters, the review asks the reader whether they would notice a bad day. | diff --git a/templates/README.md b/templates/README.md index 4bb434d..76ca7fe 100644 --- a/templates/README.md +++ b/templates/README.md @@ -6,7 +6,7 @@ In the DMAIC mapping ([pattern template](../docs/pattern-template.md)), the arti | Directory | Module | Contents | |---|---|---| -| `01-ai-data-quality/` | AI data quality | [Retention-state control plan](01-ai-data-quality/retention-state-control-plan.md) (M1-11) · [Metric-computation control plan](01-ai-data-quality/metric-computation-control-plan.md) (M1-01) · [SPC control plan](01-ai-data-quality/spc-control-plan.md) (M1-02) · [Golden question set](01-ai-data-quality/golden-question-set.md) (M1-04) · [Plausibility-check control plan](01-ai-data-quality/plausibility-check-control-plan.md) (M1-03) · [Schema contract control plan](01-ai-data-quality/schema-contract-control-plan.md) (M1-07) · [Agreement-measurement control plan](01-ai-data-quality/agreement-measurement-control-plan.md) (M1-05) | +| `01-ai-data-quality/` | AI data quality | [Retention-state control plan](01-ai-data-quality/retention-state-control-plan.md) (M1-11) · [Metric-computation control plan](01-ai-data-quality/metric-computation-control-plan.md) (M1-01) · [SPC control plan](01-ai-data-quality/spc-control-plan.md) (M1-02) · [Golden question set](01-ai-data-quality/golden-question-set.md) (M1-04) · [Plausibility-check control plan](01-ai-data-quality/plausibility-check-control-plan.md) (M1-03) · [Schema contract control plan](01-ai-data-quality/schema-contract-control-plan.md) (M1-07) · [Agreement-measurement control plan](01-ai-data-quality/agreement-measurement-control-plan.md) (M1-05) · [SLO register](01-ai-data-quality/slo-register.md) (M1-06) | | `02-infrastructure-design/` | Infrastructure design | _pending_ | | `03-ai-agent-integration/` | AI agent integration | [Business context file](03-ai-agent-integration/context-file.md) (M3-06) | | `04-governance-and-financial-reporting/` | Governance and financial reporting | _pending_ |