From c92bd4965d2fee1d40abf71885d053a612b07af6 Mon Sep 17 00:00:00 2001 From: Heqing Huang <13338360+h3qing@users.noreply.github.com> Date: Mon, 21 Sep 2026 07:12:32 +0000 Subject: [PATCH 1/3] Reader-facing pages showed the maintainer's process instead of the content Before the repository is shared widely, the README, reading guide, module charters and indexes carried visible author notes, phase numbers, backlog IDs, and a status line written before any pattern now on main existed. Author notes move into HTML comments and a section that held only a note is hidden rather than filled. The metrics library index says what each page answers instead of where the drafting stands. The skills index, prior-art review, stubs and Module 1 charter drop phase and backlog references. The roadmap, changelog, README status and module status lines reflect what is on main. Contributing points at issues, since Discussions are not enabled, and its provenance rule matches what CI checks. The reference-architecture and guide indexes point at the roadmap instead of promising counts it disagreed with. The judgment calls are recorded in DECISIONS.md. --- .github/ISSUE_TEMPLATE/pattern-proposal.md | 2 +- .gitignore | 2 +- CHANGELOG.md | 4 +++ CONTENT_BACKLOG.md | 2 +- CONTRIBUTING.md | 4 +-- DECISIONS.md | 8 ++++++ README.md | 6 +++-- REFERENCES.md | 2 +- ROADMAP.md | 4 +-- docs/index.md | 25 +++++++++++++------ docs/metrics/README.md | 16 ++++++------ docs/modules/01-ai-data-quality/README.md | 14 +++++------ .../02-infrastructure-design/README.md | 4 ++- .../modules/03-ai-agent-integration/README.md | 4 +-- .../README.md | 8 +++--- docs/prior-art.md | 6 ++--- implementation-guides/README.md | 2 +- .../metric-definitions-repository.md | 2 +- lychee.toml | 2 +- reference-architectures/README.md | 2 +- .../analytics-stack-ai-native-startup.md | 2 +- skill/README.md | 16 ++++++------ templates/03-ai-agent-integration/.gitkeep | 0 templates/README.md | 2 +- 24 files changed, 84 insertions(+), 55 deletions(-) delete mode 100644 templates/03-ai-agent-integration/.gitkeep diff --git a/.github/ISSUE_TEMPLATE/pattern-proposal.md b/.github/ISSUE_TEMPLATE/pattern-proposal.md index f5885ba..087546a 100644 --- a/.github/ISSUE_TEMPLATE/pattern-proposal.md +++ b/.github/ISSUE_TEMPLATE/pattern-proposal.md @@ -7,7 +7,7 @@ labels: pattern-proposal ## Module -Which of the four modules does this belong to, and why? For Module 1, say which half: 1A (definition quality — what to measure) or 1B (data integrity — whether the number is right). +Which of the four modules does this belong to, and why? Say which half: 1A or 1B, 2A or 2B, 3A or 3B, 4A or 4B. The module charters under `docs/modules/` define them. ## Problem diff --git a/.gitignore b/.gitignore index dae6ecb..3bda752 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,7 @@ # macOS .DS_Store -# Phase 0 working documents — not part of the published structure +# Maintainer working documents — not part of the published structure CANDIDATE_PATTERNS.md # Machine-local working notes diff --git a/CHANGELOG.md b/CHANGELOG.md index e8fe967..fdd0fea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- Six Module 1B pattern drafts, each with a control-plan template: single point of metric computation, statistical process control for data pipelines, detecting plausible-but-wrong outputs, golden question sets for AI analytics validation, agreement measurement for analytics QA, and schema drift detection and data contracts. +- The first Module 3 pattern draft, the business context file an AI assistant reads before it answers, with a copyable context-file template. +- Six metrics-library topic pages: attribution, conversion rate, time to convert, active users, retention, and benchmarks. +- The first guided skill, attribution design, and the skill template every skill follows. - `llms.txt`, the file an AI assistant reads after the reader pastes one line: what the library is, which page to fetch for which problem, and what to hand the reader at the end. The README carries the line. CI fails when a finished page has no row in the file. - Repository skeleton: module charters, pattern template, prior-art and references scaffolding, deliverable-type homes, community files, CI. - First pattern draft: state-based retention measurement (M1-11, Adapted from Duolingo's published growth model), with Module 1's first control-plan template, first prior-art row, and first verified REFERENCES entries. diff --git a/CONTENT_BACKLOG.md b/CONTENT_BACKLOG.md index 72f774d..896bca3 100644 --- a/CONTENT_BACKLOG.md +++ b/CONTENT_BACKLOG.md @@ -8,7 +8,7 @@ The Module column uses `1A` (definition quality) and `1B` (data integrity) for t |---|---|---|---|---|---|---|---| | MX-01 | 1A | Segmentation | topic | — | ready | Author's own thesis: segmentation is where the manufacturing lens stops working | 1 session | | MX-02 | 1A | Benchmarks, and how to check a number before you steer by it | topic | — | drafted | Five failed traces: Reichheld 5%, 5x-cheaper, 7-friends, Groove 71%, DAU/MAU 50% | 1 session | -| MX-03 | 1A | Retention | topic | — | in progress | Fader & Hardie sorting effect, NRR filings, author's two-axis split | 1 session | +| MX-03 | 1A | Retention | topic | — | drafted | Fader & Hardie sorting effect, NRR filings, author's two-axis split | 1 session | | MX-04 | 1A | Time to convert | topic | — | drafted | Little's Law, flow metrics; merged, author answers folded in | — | | SK-01 | 1A | Defining your definitions (guided skill) | skill | — | ready | Facebook single-definition story (Schultz lecture, S-1) | 1 session | | SK-02 | 1A | Is this a vanity metric? (guided skill) | skill | — | candidate | Ries 2009 posts, data-theater critique, Twitter mDAU story | 1 session | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bc8ec72..6c2892a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,12 +6,12 @@ Thank you for considering it. The most valuable contribution this project can re - **Review a pattern.** Open an issue or comment on the pattern's file. Disagreement with a pattern's Position section is especially welcome; that section exists to be argued with. - **Propose a pattern.** Use the pattern-proposal issue template. Proposals need the problem, the position ("do X, not Y"), and honest prior art — a pointer to the closest existing writing, even if it partly covers your idea. -- **Contribute a case study.** Via Discussions. Anonymized is fine; fabricated is not. +- **Contribute a case study.** Open an issue. Anonymized is fine; fabricated is not. - **Fix errors.** PRs for factual errors, dead links, or broken citations are always welcome. ## Ground rules -- **Provenance is load-bearing.** Every normative claim cites prior art via a key in [REFERENCES.md](REFERENCES.md) or is explicitly tagged `Original` with a recorded search in [docs/prior-art.md](docs/prior-art.md). CI enforces this. +- **Provenance is load-bearing.** Every pattern lists its sources in a Sources & Stories section, every citation key resolves in [REFERENCES.md](REFERENCES.md), and a claim that is the framework's own says so in the text. CI checks the section and the keys. The landscape each pattern came from is recorded in [docs/prior-art.md](docs/prior-art.md). - **Vendor neutral.** Patterns describe categories of tool and classes of failure, never named products. - **No confidential material.** Do not contribute your employer's internal metrics, system names, or numbers. Abstract to how the class of problem behaves. - **Template discipline.** Pattern content follows the [eight-section template](docs/pattern-template.md) exactly. diff --git a/DECISIONS.md b/DECISIONS.md index 5572e0c..dba19c3 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -2,6 +2,14 @@ One entry per judgment call: decision, alternatives, why. +## 2026-09-21 — Pre-share cleanup: author notes go invisible, indexes drop the making-of, promises point at the roadmap + +**Decision:** three calls made while cleaning the reader-facing pages before the repository is shared widely. (a) Author notes that were visible as `[TODO(heqing): …]` blocks on the reading guide and the module charters move into HTML comments, and a section whose only content was such a note is hidden whole rather than left as an empty heading. Nothing is written in their place, per hard constraints 6 and 7. (b) Index pages stop carrying maintainer status. The metrics library table now says what each page answers, in the words `llms.txt` already uses, and the skills index, the prior-art review, the stubs and the Module 1 charter lose their phase numbers and backlog IDs; row-level status stays in the backlog. (c) The reference-architecture and implementation-guide indexes no longer promise a count per release, since the roadmap disagreed with them; they point at the roadmap instead. + +**Alternatives:** fill the empty sections from lines the README already carries, which would put generated prose where the author's voice is expected; keep the visible notes as an honesty signal, which the 2026-08-11 entry already rejected; keep the counts and reconcile the roadmap to them. + +**Why:** a page that shows its own outline reads as unfinished whatever its contents, and a reader arriving from a share link should meet the content, not the process that produced it. + ## 2026-09-16 — M1-05 argues with the single-owner key its sibling page installed, scores values by raw agreement and verdicts by kappa, and refuses to score disputed questions **Decision:** four calls made while drafting the agreement-measurement pattern. (a) The Position argues with the single owner writing the reference answers, the practice M1-04 itself installs. The page keeps that owner and treats the owner's answers as a gauge with an unmeasured error: a second person answers the same twenty questions blind before the assistant is scored, the human-to-human rate is the ceiling the assistant is held to, and an assistant inside that rate's band gets no model work. (b) Values within tolerance are scored by raw percent agreement, because two numbers landing within a tenth of a percent by chance is not a real risk; kappa is reserved for pass-or-fail verdicts and is reported beside the raw rate, never alone. The fixed kappa bands the quality manuals grade against are not printed, because no source inside the ten-year window could be verified for them, and the question is left for the author. (c) The band is plus and minus the square root of p(1−p)/n, about two questions at twenty, and the floor of twelve agreements in twenty below which the questions rather than the people are the problem is likewise the framework's own; both are marked as such on the page and in the template. (d) A question the two people disagree on is not scored for or against the assistant. It routes to one line in the context file and a dated new row in the key, so the assistant is scored only on settled questions, and its misses there are the only errors that are its own. (e) Three calls from the author's review, folded in on 2026-09-17: the quality manuals' kappa bands are not used, because a percentage difference can be a delay, an exclusion, a calendar, or two different metrics landing near each other, and no band separates those; the logic is checked before any statistic, through a three-bullet conditions summary under every answer (what was excluded, which calendar, when the data was pulled) that an assistant can write and a person can verify; and the per-metric tolerance is set by a person as the reporting difference the operation can live with, before the study. Nobody is the better judge; the study is run for alignment, and its working signal is that important calls spend their time on actions rather than on whose number is right. diff --git a/README.md b/README.md index 9dfe1e7..55de910 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ The primary worked case is early-stage and AI-native companies; the patterns gen ## What ships -Four deliverable types, each with a visible home here: the framework itself (`docs/modules/`), best-practices documentation, reference architectures (`reference-architectures/`), and implementation guides (`implementation-guides/`). Every module ships at least one copyable control-plan template in `templates/`. A diagnostic skill (`skill/`) assesses an organization's readiness before it deploys an AI analytics agent. +Four deliverable types, each with a visible home here: the framework itself (`docs/modules/`), best-practices documentation, reference architectures (`reference-architectures/`), and implementation guides (`implementation-guides/`). Every module ships at least one copyable control-plan template in `templates/`. Guided skills (`skill/`) walk a team through one decision at a time; the planned diagnostic skill assesses an organization's readiness before it deploys an AI analytics agent. ## Sources @@ -67,7 +67,7 @@ Every pattern lists the stories and research it draws from — the Duolingo grow ## Status -Pre-release. Prior-art review in progress; module content follows. See [ROADMAP.md](ROADMAP.md). +Pre-release, with no version tagged yet. Every page on `main` is a full sourced draft awaiting the author's final pass: the patterns listed in the [Module 1](docs/modules/01-ai-data-quality/README.md) and [Module 3](docs/modules/03-ai-agent-integration/README.md) charters, the topic pages in the [metrics library](docs/metrics/README.md), the control plans in [templates](templates/README.md), and one [guided skill](skill/README.md). The release plan is in [ROADMAP.md](ROADMAP.md). ## Contributing @@ -79,4 +79,6 @@ Documentation: [CC BY 4.0](LICENSE). Code and templates: [MIT](LICENSE). ## Author +Maintained by [Heqing Huang](https://github.com/h3qing). + diff --git a/REFERENCES.md b/REFERENCES.md index 885ae0b..5d8b95b 100644 --- a/REFERENCES.md +++ b/REFERENCES.md @@ -10,7 +10,7 @@ Bibliography for the framework. Every citation key used anywhere in `docs/`, `re ## Entries -_The full Phase 0 seeding is pending; entries below were verified individually as units were synthesized._ +_Entries are added as pages are drafted, each verified individually before it enters._ - **[ACZEL-2021]** — Aczel, Balazs; Szaszi, Barnabas; Nilsonne, Gustav; van den Akker, Olmo R.; et al. "Consensus-based guidance for conducting and reporting multi-analyst studies." _eLife_ 10, e72185, November 9, 2021. doi:10.7554/eLife.72185. Read in full from the publisher. A 54-author consensus paper produced by a reactive-Delphi panel of methodologists, defining a multi-analyst study as one that engages multiple analysts to independently analyze the same data. Verified verbatim: "Co-analysts should not communicate with each other about their analyses until after all initial reports have been submitted", and the recommendation to state the research question "without communicating the lead team's preferred analysis choices or expectations about the conclusions". Cited for the blinding rule; the guidance is written for research studies, and the framework carries it to a two-person study inside a company. - **[AGENTSMD]** — AGENTS.md. "A simple, open format for guiding coding agents." agents.md, fetched August 26, 2026. Open format stewarded by the Agentic AI Foundation under the Linux Foundation; the page describes it as "a README for agents: a dedicated, predictable place to provide the context and instructions" and confirms there are no required fields, just standard Markdown. The key carries no year because the page is an undated living document. The adoption figure ("used by over 60k open-source projects") is self-reported with no methodology; attribute it, never assert it. The format targets coding agents; carrying business context in it is this framework's extension, not documented practice. diff --git a/ROADMAP.md b/ROADMAP.md index 44fa9ad..743f27b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # Roadmap -Each tag below is a real git tag and a GitHub release with notes, aligned to [CHANGELOG.md](CHANGELOG.md). +Each version below becomes a git tag and a GitHub release with notes when it ships, aligned to [CHANGELOG.md](CHANGELOG.md). No version is tagged yet. | Tag | Contents | Status | |---|---|---| @@ -9,7 +9,7 @@ Each tag below is a real git tag and a GitHub release with notes, aligned to [CH | — | One pasted line gets a reader in: `llms.txt`, the line on the README, CI that keeps the file current | done, 2026-09-16 | | — | Diagnostic skill v0.1 published | not started; the first guided skill, attribution design, is drafted | | v0.1.0 | Module 1's core patterns: the 1B integrity patterns, on the 1A topic pages already drafted | drafted and on main: single point of computation, statistical process control, plausible-but-wrong, golden question sets, schema drift; the tag follows the author's voice pass | -| v0.2.0 | Module 1 complete, both halves, and the first reference architecture | in progress: agreement measurement and the segmentation topic page are in review; four 1B patterns, the reference architecture, and the remaining 1A pages are open | +| v0.2.0 | Module 1 complete, both halves, and the first reference architecture | in progress: agreement measurement is drafted and on main; the segmentation topic page, data quality SLOs and the reconciliation protocol are in review; the remaining 1B patterns, the reference architecture, and the remaining 1A pages are open | | v0.3.0 | Second module core patterns, first implementation guide, skill v1.0 | not started; the module 3 context-file pattern and its template are already on main | | v1.0.0 | All four modules at first complete draft | not started | diff --git a/docs/index.md b/docs/index.md index 65510d8..18a4c22 100644 --- a/docs/index.md +++ b/docs/index.md @@ -12,11 +12,9 @@ Read https://raw.githubusercontent.com/h3qing/open-analytics-framework/main/llms These systems do not fail at the model layer. They fail at deployment. Query generation is largely a solved problem — model ability is generalized, but it lacks opinion and industry experience. What breaks is everything around it: inconsistent metric definitions across teams, schemas that drift, outputs that look correct while resting on bad joins or incomplete data, and the resulting collapse of organizational trust that sends everyone back to manual validation. That is a process and systems problem, not a modeling problem, which is why industrial-engineering methodology is the right lens. -[TODO(heqing): expand in your own voice — what this means for the reader, and why the deployment layer is where your five years were spent.] + -## Who this is for - -[TODO(heqing): primary worked case (early-stage and AI-native companies) and how the patterns generalize down. Say it explicitly rather than pretending to be audience-neutral.] + ## How the framework is organized @@ -64,8 +62,21 @@ Patterns cite the stories and research they draw from: see [the prior-art review ## Available now -- [State-based retention measurement](modules/01-ai-data-quality/state-based-retention-measurement.md) — the first pattern, drawn from the Duolingo growth story. Drafted; awaiting the author's voice pass. +Every page below is a full sourced draft awaiting the author's final pass. + +**Patterns**, each with its control-plan template in [`templates/`](../templates/README.md): + +- [State-based retention measurement](modules/01-ai-data-quality/state-based-retention-measurement.md) (1A) +- [Single point of metric computation](modules/01-ai-data-quality/single-point-of-metric-computation.md) (1B) +- [Statistical process control for data pipelines](modules/01-ai-data-quality/statistical-process-control-for-pipelines.md) (1B) +- [Detecting plausible-but-wrong outputs](modules/01-ai-data-quality/plausible-but-wrong.md) (1B) +- [Golden question sets for AI analytics validation](modules/01-ai-data-quality/golden-question-sets.md) (1B) +- [Agreement measurement for analytics QA](modules/01-ai-data-quality/agreement-measurement.md) (1B) +- [Schema drift detection and data contracts](modules/01-ai-data-quality/schema-drift-and-data-contracts.md) (1B) +- [The one file your AI should read before it answers anything](modules/03-ai-agent-integration/context-file.md) (3B) + +**Metric topic pages** in the [metrics library](metrics/README.md): [attribution](metrics/attribution.md), [conversion rate](metrics/conversion-rate.md), [time to convert](metrics/time-to-convert.md), [active users](metrics/active-users.md), [retention](metrics/retention.md) and [benchmarks](metrics/benchmarks.md). -## Reading order +**Guided skill:** [attribution design](../skill/attribution-design/SKILL.md). -[TODO(heqing): recommended paths — e.g., "deploying an agent this quarter" vs. "no analytics function yet". Write after Module 1 content exists.] + diff --git a/docs/metrics/README.md b/docs/metrics/README.md index 1fe2009..a673ba6 100644 --- a/docs/metrics/README.md +++ b/docs/metrics/README.md @@ -16,14 +16,16 @@ Topic pages carry a Sources & Stories section like every other content file; cit ## Topics -| Topic | Status | +| Topic | What it answers | |---|---| -| [Attribution](attribution.md) | drafted; author answers and voice pass done, guided skill shipped | -| [Conversion rate](conversion-rate.md) | drafted; author answers and voice pass done | -| [Time to convert](time-to-convert.md) | drafted; author answers folded in, voice pass pending | -| [Active users](active-users.md) | drafted; author opening written, interview questions open | -| [Retention](retention.md) | drafted; entity retention only, author answers folded in, voice pass pending | -| [Benchmarks](benchmarks.md) | drafted; every trace re-verified, interview pending | +| [Attribution](attribution.md) | Who gets credit for an outcome, and how to line the credit rule up with how people are paid. Has a [guided skill](../../skill/attribution-design/SKILL.md). | +| [Conversion rate](conversion-rate.md) | The rate between two funnel steps, where conversion ends and activation begins, and how it drives acquisition cost and lifetime value. | +| [Time to convert](time-to-convert.md) | How long it takes to move through the pipeline, and what that does to the meaning of a window. | +| [Active users](active-users.md) | What active should mean at your company, the daily, weekly and monthly windows, and when the total is a vanity number. | +| [Retention](retention.md) | Whether the users, accounts or workspaces you won come back, how to read a cohort curve, and why a flattening curve is a sorting effect rather than loyalty. | +| [Benchmarks](benchmarks.md) | What a benchmark is for, how to set a goal from one, and five questions to ask of any number before you steer by it. | + +Every page is a full sourced draft awaiting the author's final pass. Retention was too big for one page. [Retention](retention.md) now covers the entity: users, accounts and workspaces, and whether they come back. Value retention, where the thing retained is contract value, and lifetime value, where retention meets acquisition cost, are separate pages still to be written. diff --git a/docs/modules/01-ai-data-quality/README.md b/docs/modules/01-ai-data-quality/README.md index 5d042d3..8bf8406 100644 --- a/docs/modules/01-ai-data-quality/README.md +++ b/docs/modules/01-ai-data-quality/README.md @@ -2,7 +2,7 @@ > Scope: the quality of the whole measurement system — both whether you are measuring the right thing and whether the number you get back is right. Covers metric definition, instrumentation, statistical control, validation, and the trust signals built on them. Does not cover: model selection, prompting, or benchmark performance; infrastructure design (Module 2); agent deployment mechanics (Module 3); metric change control as a governance process (Module 4). -**Status:** charter plus first drafted pattern. This is the opening module. Patterns land here after prior-art review and interview or source-synthesis sessions; the working list lives in [CONTENT_BACKLOG.md](../../../CONTENT_BACKLOG.md). +**Status:** the opening module. The patterns linked below are drafted, each with its control-plan template in [`templates/01-ai-data-quality/`](../../../templates/01-ai-data-quality/); rows without a link are not yet written. Row-level status lives in [CONTENT_BACKLOG.md](../../../CONTENT_BACKLOG.md). ## Two halves @@ -14,13 +14,13 @@ A measurement system fails in two independent ways. It can measure the wrong cha | Fails as | A precisely computed number that nobody can act on, or three teams reporting three different values for "active" | A correct definition returning a wrong value: bad join, late partition, silent schema change | | DMAIC phase | Define — naming the critical-to-quality characteristic | Measure, Analyze, Improve, Control — measurement-system analysis and SPC | | Knowledge layer | The [metrics library](../../metrics/README.md) | [`templates/01-ai-data-quality/`](../../../templates/01-ai-data-quality/) control plans | -| Guided layer | The [skills](../../../skill/README.md) that walk a team through defining a metric in their own context | [TODO(heqing): validation-harness skill, or none] | +| Guided layer | The [skills](../../../skill/README.md) that walk a team through defining a metric in their own context | — | -Read 1A first. A definition defect propagates into every check 1B can run: statistical control on a metric that measures the wrong thing produces a well-behaved number nobody should steer by. + -## Why this module exists +Read 1A first. A definition defect propagates into every check 1B can run: statistical control on a metric that measures the wrong thing produces a well-behaved number nobody should steer by. -[TODO(heqing): the failure this module prevents, in one paragraph, in your voice. Say which half you have watched fail more often, and why the two are one module rather than two. Interview question to be generated in session.] + ## 1A — Definition quality @@ -30,11 +30,11 @@ What to measure, and how to pin a definition down so it holds. The durable knowl |---|---|---|---| | M1-11 | [State-based retention measurement](state-based-retention-measurement.md) | drafted — awaiting author voice pass | Duolingo growth model (Gustafson, Mazal) | -Topic pages currently drafted: [attribution](../../metrics/attribution.md), [conversion rate](../../metrics/conversion-rate.md), [time to convert](../../metrics/time-to-convert.md), [active users](../../metrics/active-users.md), [retention](../../metrics/retention.md). Planned topics and guided skills are backlog rows `MX-*` and `SK-*`. +Topic pages drafted so far: [attribution](../../metrics/attribution.md), [conversion rate](../../metrics/conversion-rate.md), [time to convert](../../metrics/time-to-convert.md), [active users](../../metrics/active-users.md), [retention](../../metrics/retention.md) and [benchmarks](../../metrics/benchmarks.md); the pages planned next are listed at the end of the [metrics library](../../metrics/README.md#topics). The first guided skill is [attribution design](../../../skill/attribution-design/SKILL.md). ## 1B — Data integrity -Whether the number the pipeline returns is the number the definition asks for. Rows without a link are `candidate` until the Phase 0 prior-art review cuts the list; the backlog is the source of truth. +Whether the number the pipeline returns is the number the definition asks for. Rows without a link are not yet written. | ID | Working title | Target | |---|---|---| diff --git a/docs/modules/02-infrastructure-design/README.md b/docs/modules/02-infrastructure-design/README.md index 08b69da..791b1ae 100644 --- a/docs/modules/02-infrastructure-design/README.md +++ b/docs/modules/02-infrastructure-design/README.md @@ -14,7 +14,9 @@ Infrastructure fails in two independent ways. It can be built wrong, and it can | Fails as | Money spent on a stack that answers nothing, or a foundation you cannot retrofit because the history is already gone | A correct build that drifts, breaks without anyone noticing, or stops working the week one person leaves | | DMAIC phase | Improve — designing the process that produces the number | Control — holding it as volume, headcount and sources grow | | Knowledge layer | The [reference architectures](../../../reference-architectures/README.md): what a whole stack looks like at a stage | [`templates/02-infrastructure-design/`](../../../templates/02-infrastructure-design/) control plans | -| Guided layer | [TODO(heqing): a stack-selection skill, or none] | The verification skill in 2B, and the [skills](../../../skill/README.md) layer generally | +| Guided layer | — | The verification skill in 2B, and the [skills](../../../skill/README.md) layer generally | + + Read 2A first, and inside it start with what you cannot get back. Most of this module is reversible: a warehouse can be migrated, a tool can be cancelled, a model can be rewritten. History cannot be recovered once a source system has overwritten it, which is why the build order below leads with the units that expire rather than the units that matter most. diff --git a/docs/modules/03-ai-agent-integration/README.md b/docs/modules/03-ai-agent-integration/README.md index aa53a84..dc2b2e8 100644 --- a/docs/modules/03-ai-agent-integration/README.md +++ b/docs/modules/03-ai-agent-integration/README.md @@ -2,7 +2,7 @@ > This module covers pointing an AI agent at your company's data: what the agent may reach, how it authenticates, what untrusted rows can make it do, what it records, whether its answers can be trusted, and how a company adopts it. -**Status:** build order set, no units written yet. Statuses live in [CONTENT_BACKLOG.md](../../../CONTENT_BACKLOG.md). +**Status:** build order set. The first pattern, [the context file](context-file.md), is drafted with its template; nothing else is written yet. Statuses live in [CONTENT_BACKLOG.md](../../../CONTENT_BACKLOG.md). ## Two halves @@ -94,7 +94,7 @@ One result matters more than everything else in this half. In a paired evaluatio | Unit | What goes wrong without it | Type | |---|---|---| | [The one file your AI should read before it answers anything](context-file.md) | The agent guesses at ambiguous tables, and the fix everyone reaches for is a better model rather than a written definition | pattern | -| Write your company's context file in an afternoon | You agree the file matters and never sit down to write it | template | +| [Write your company's context file in an afternoon](../../../templates/03-ai-agent-integration/context-file.md) | You agree the file matters and never sit down to write it | template | | What to write first so an AI agent gets your numbers right | The effort goes into describing every column, which is the highest-cost lowest-return work available | pattern | | The pairs you feed it are the pairs you can no longer test it with | Your examples and your answer key are the same set, so the test passes by construction | pattern | | Your agent will inherit your SQL's bugs | An undocumented filter in the transformation is reproduced faithfully, and unlike a person the agent cannot notice | pattern | diff --git a/docs/modules/04-governance-and-financial-reporting/README.md b/docs/modules/04-governance-and-financial-reporting/README.md index 6b233bd..4af0047 100644 --- a/docs/modules/04-governance-and-financial-reporting/README.md +++ b/docs/modules/04-governance-and-financial-reporting/README.md @@ -14,13 +14,13 @@ | Fails as | The number is right and you cannot defend it. The definition that produced last March's figure is unrecoverable, a retention clock expired on the history diligence just asked for, an invoice was edited in place and the old value is gone | ARR that quietly mixes subscription, usage, credits, pilots and implementation fees. A margin that omits inference cost. A definition that changed between two board decks and rewrote a trend with nothing disclosed | | DMAIC phase | Control — the evidence that the process was held, kept as long as someone can still ask | Define and Measure, applied to the money characteristics | | Knowledge layer | [`templates/04-governance-and-financial-reporting/`](../../../templates/04-governance-and-financial-reporting/) control plans | The investor pack and the definitions sheets | -| Guided layer | [TODO(heqing): a reproduction drill, or none] | [TODO(heqing): a revenue-definition skill, or none] | +| Guided layer | — | — | -The money definitions carry a disclosure consequence: changing what counts as an active user is an internal problem, and changing what counts as ARR between two board decks is a restatement. + -## Why this module exists +The money definitions carry a disclosure consequence: changing what counts as an active user is an internal problem, and changing what counts as ARR between two board decks is a restatement. -[TODO(heqing): interview-driven, written in session. Say in your own voice why a company with no compliance function should read any of this, and what you have watched go wrong when the record was not kept. The research supplies the rules; it cannot supply the reason you think this belongs in an analytics framework at all.] + ## 4A — What you must be able to show diff --git a/docs/prior-art.md b/docs/prior-art.md index 6a57886..f701673 100644 --- a/docs/prior-art.md +++ b/docs/prior-art.md @@ -1,6 +1,6 @@ # Prior art and landscape review -> Scope: the coverage matrix — one row per candidate pattern, recording what already exists publicly and what does not. This file is what lets readers trace each pattern's Sources & Stories back to the landscape it came from. Status: **the full Phase 0 review is pending; rows land individually as units are synthesized.** +> Scope: the coverage matrix — one row per candidate pattern, recording what already exists publicly and what does not. This file is what lets readers trace each pattern's Sources & Stories back to the landscape it came from. Status: **rows are added as patterns are drafted. The patterns drafted after the first do not yet have a row here; their sources are listed in each page's Sources & Stories section and in [REFERENCES.md](../REFERENCES.md).** ## Method @@ -11,8 +11,8 @@ For each candidate pattern: search books, peer-reviewed papers, standards docume | Pattern | Closest prior art | Citation key | What it covers | What it does not cover | |---|---|---|---|---| | State-based retention measurement (M1-11) | Duolingo's growth model, documented by the practitioners who built it; per Mazal, itself adapted from state models at Zynga and MyFitnessPal | [GUSTAFSON-2023], [MAZAL-2023] | Seven MECE daily activity states; named transition rates (CURR, NURR, RURR, SURR, reactivation/resurrection); sensitivity simulation to select a focus metric; verifying the metric is movable and moves the topline | Application to organizations without analytics staff; low-volume weekly-window variant; reverse-funnel bottleneck framing and coarse-first instrumentation (original here); LLM-agent operationalization | -| _remaining rows pending Phase 0_ | | | | | +| _rows for the patterns drafted since are pending_ | | | | | ## Uncovered ground -_Pending Phase 0: where the review found genuinely uncovered territory._ +_Pending: where the review found genuinely uncovered territory._ diff --git a/implementation-guides/README.md b/implementation-guides/README.md index 7ab9333..3c918bd 100644 --- a/implementation-guides/README.md +++ b/implementation-guides/README.md @@ -6,4 +6,4 @@ One guide per file. Guides are the long-form counterpart to the patterns in `doc |---|---|---| | [Standing up a metric definitions repository](metric-definitions-repository.md) | Module 1 | stub | -_More added per [CONTENT_BACKLOG.md](../CONTENT_BACKLOG.md); at least two ship by v0.3.0._ +_More follow the [roadmap](../ROADMAP.md)._ diff --git a/implementation-guides/metric-definitions-repository.md b/implementation-guides/metric-definitions-repository.md index c90daa6..d55aa13 100644 --- a/implementation-guides/metric-definitions-repository.md +++ b/implementation-guides/metric-definitions-repository.md @@ -2,7 +2,7 @@ > Scope: step-by-step implementation of single-point metric definition ownership, ending with the control-plan template in `templates/01-ai-data-quality/`. Does not cover: choosing infrastructure (Module 2) or governance of definition changes (Module 4). -**Status:** stub. Content is interview-driven and lands in a Phase 3 session. +**Status:** stub. Not yet written. ## Problem diff --git a/lychee.toml b/lychee.toml index 2542d27..81347bd 100644 --- a/lychee.toml +++ b/lychee.toml @@ -9,7 +9,7 @@ timeout = 30 max_retries = 3 retry_wait_time = 2 -# Phase 0 working document, not part of the published structure. +# Maintainer working document, untracked and not part of the published structure. exclude_path = ["CANDIDATE_PATTERNS.md"] # Hosts that serve readers normally but answer automated checkers with 403 or diff --git a/reference-architectures/README.md b/reference-architectures/README.md index 6f471ea..79d9427 100644 --- a/reference-architectures/README.md +++ b/reference-architectures/README.md @@ -6,4 +6,4 @@ One architecture per file. Category-level, vendor-neutral: each names classes of |---|---|---| | [Analytics stack for an AI-native startup, pre-first-data-hire](analytics-stack-ai-native-startup.md) | Module 2 | stub | -_More added per [CONTENT_BACKLOG.md](../CONTENT_BACKLOG.md); at least two ship by v0.2.0._ +_More follow the [roadmap](../ROADMAP.md)._ diff --git a/reference-architectures/analytics-stack-ai-native-startup.md b/reference-architectures/analytics-stack-ai-native-startup.md index 11bc903..c0eda68 100644 --- a/reference-architectures/analytics-stack-ai-native-startup.md +++ b/reference-architectures/analytics-stack-ai-native-startup.md @@ -2,7 +2,7 @@ > Scope: a category-level architecture for the smallest reliable analytics stack at an AI-native company before its first dedicated data hire. Does not cover: product recommendations, cost comparisons, or organizations past the first-data-hire stage. -**Status:** stub. Content is interview-driven and lands in a Phase 4 session. +**Status:** stub. Not yet written. ## Problem diff --git a/skill/README.md b/skill/README.md index 47467da..aba2fe3 100644 --- a/skill/README.md +++ b/skill/README.md @@ -2,19 +2,19 @@ Guided conversations that apply this framework's knowledge to one company's situation: questions one or two at a time, guidance after each answer, a decision record at the end. Every skill follows [the skill template](../docs/skill-template.md). MIT licensed. +Guided skills built on a topic page are Module 1A work: they walk a team through defining a metric in their own context. The `analytics-readiness` diagnostic is the exception, since it spans all four modules. + | Skill | Status | |---|---| | [attribution-design](attribution-design/SKILL.md) — pick the credit rule for your size, align it with team incentives | draft | -| `analytics-readiness` — the diagnostic run before deploying an AI analytics agent (working name, Open Decision #2) | planned, Phase 2 | -| Defining your definitions (SK-01) and the vanity-metric check (SK-02) | candidate rows in the backlog | +| `analytics-readiness` — the diagnostic run before deploying an AI analytics agent (working name) | planned | +| Defining your definitions, and the vanity-metric check | planned | -The `analytics-readiness` diagnostic outputs a readiness report with a prioritized remediation plan mapped to the four modules, including the narrowest wedge — the single metric that, if made trustworthy enough that nobody re-checks it, changes the most downstream. Its construction is Phase 2, deliberately before the module content: writing the forcing questions is how the content gets articulated. +The `analytics-readiness` diagnostic outputs a readiness report with a prioritized remediation plan mapped to the four modules, including the narrowest wedge — the single metric that, if made trustworthy enough that nobody re-checks it, changes the most downstream. ## Seeded forcing questions -Accumulated as patterns land, ahead of Phase 2 construction: - -- From [M1-11 state-based retention measurement](../docs/modules/01-ai-data-quality/state-based-retention-measurement.md): *Name, from data you already have, the single lifecycle transition where you lose the most users.* If nobody can answer without inventing a new query on the spot, the organization is goaled on aggregates it cannot act on, and the retention-state pattern is a prerequisite for pointing an AI analytics agent at engagement questions. -Guided skills built on a topic page are Module 1A work: they walk a team through defining a metric in their own context. The `analytics-readiness` diagnostic is the exception — it spans all four modules. +Accumulated as patterns land, for the diagnostic to draw on: -- From the [conversion topic page](../docs/metrics/conversion-rate.md): *Ask three teams what "active" and "converted" mean, and compare the answers.* If the definitions differ and none is written down, no metric downstream of them can be trusted. The checkable model is Facebook holding the whole company to one definition of active, inside and out, edge cases documented. This question may grow into its own guided skill on defining definitions; candidate row SK-01 in the backlog. +- From [state-based retention measurement](../docs/modules/01-ai-data-quality/state-based-retention-measurement.md): *Name, from data you already have, the single lifecycle transition where you lose the most users.* If nobody can answer without inventing a new query on the spot, the organization is goaled on aggregates it cannot act on, and the retention-state pattern is a prerequisite for pointing an AI analytics agent at engagement questions. +- From the [conversion topic page](../docs/metrics/conversion-rate.md): *Ask three teams what "active" and "converted" mean, and compare the answers.* If the definitions differ and none is written down, no metric downstream of them can be trusted. The checkable model is Facebook holding the whole company to one definition of active, inside and out, edge cases documented. This question may grow into its own guided skill on defining definitions. diff --git a/templates/03-ai-agent-integration/.gitkeep b/templates/03-ai-agent-integration/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/templates/README.md b/templates/README.md index 4bb434d..50eb420 100644 --- a/templates/README.md +++ b/templates/README.md @@ -11,4 +11,4 @@ In the DMAIC mapping ([pattern template](../docs/pattern-template.md)), the arti | `03-ai-agent-integration/` | AI agent integration | [Business context file](03-ai-agent-integration/context-file.md) (M3-06) | | `04-governance-and-financial-reporting/` | Governance and financial reporting | _pending_ | -Templates are dual-licensed MIT (see [LICENSE](../LICENSE)) so they can be copied into private repositories without ceremony. +Templates are MIT licensed (see [LICENSE](../LICENSE)) so they can be copied into private repositories without ceremony. From 3c7355d078613218357b4c46e1b1aa83a6f10c78 Mon Sep 17 00:00:00 2001 From: Heqing Huang <13338360+h3qing@users.noreply.github.com> Date: Mon, 21 Sep 2026 07:33:14 +0000 Subject: [PATCH 2/3] The code of conduct still carried the template's fill-in notes, and the README had no bio link Conduct reports go through the repository's issue tracker, and the Contributor Covenant's two bracketed notes to the adopter are removed. The README author line links the maintainer's LinkedIn bio. LinkedIn joins the hosts the external link check skips, since it answers automated checkers with a refusal rather than a page. --- CODE_OF_CONDUCT.md | 4 +--- README.md | 4 +--- lychee.toml | 1 + 3 files changed, 3 insertions(+), 6 deletions(-) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 1e07b54..d8ec042 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -47,15 +47,13 @@ We agree to restrict the following behaviors in our community. Instances, threat Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm. -When an incident does occur, it is important to report it promptly. To report a possible violation, **[NOTE: describe your means of reporting here.]** +When an incident does occur, it is important to report it promptly. To report a possible violation, open an issue in [this repository's issue tracker](https://github.com/h3qing/open-analytics-framework/issues). Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution. ## Addressing and Repairing Harm -**[NOTE: The remedies and repairs outlined below are suggestions based on best practices in code of conduct enforcement. If your community has its own established enforcement process, be sure to edit this section to describe your own policies.]** - If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped. 1) Warning diff --git a/README.md b/README.md index 55de910..b034a60 100644 --- a/README.md +++ b/README.md @@ -79,6 +79,4 @@ Documentation: [CC BY 4.0](LICENSE). Code and templates: [MIT](LICENSE). ## Author -Maintained by [Heqing Huang](https://github.com/h3qing). - - +Maintained by [Heqing Huang](https://github.com/h3qing). Bio on [LinkedIn](https://www.linkedin.com/in/hhq/). diff --git a/lychee.toml b/lychee.toml index 81347bd..6753ce0 100644 --- a/lychee.toml +++ b/lychee.toml @@ -24,6 +24,7 @@ exclude = [ "^https://www\\.reforge\\.com/", "^https://www\\.slideshare\\.net/", "^https://a16z\\.com/", + "^https://www\\.linkedin\\.com/", ] accept = ["200", "201", "202", "203", "204", "206", "301", "302", "308", "403", "429"] From b8132cb41011a239e64961f88b1917eab0b62a79 Mon Sep 17 00:00:00 2001 From: Heqing Huang <13338360+h3qing@users.noreply.github.com> Date: Mon, 21 Sep 2026 07:37:12 +0000 Subject: [PATCH 3/3] Every pull request should request the maintainer's review Adds a CODEOWNERS file naming the maintainer for every path, so GitHub requests their review on each pull request and the branch ruleset on main can require that review before a merge. --- .github/CODEOWNERS | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..b0971be --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,2 @@ +# Every pull request requests the maintainer's review before it merges. +* @h3qing