Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Every pull request requests the maintainer's review before it merges.
* @h3qing
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/pattern-proposal.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ labels: pattern-proposal

## Module

Which of the four modules does this belong to, and why? For Module 1, say which half: 1A (definition quality — what to measure) or 1B (data integrity — whether the number is right).
Which of the four modules does this belong to, and why? Say which half: 1A or 1B, 2A or 2B, 3A or 3B, 4A or 4B. The module charters under `docs/modules/` define them.

## Problem

Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# macOS
.DS_Store

# Phase 0 working documents — not part of the published structure
# Maintainer working documents — not part of the published structure
CANDIDATE_PATTERNS.md

# Machine-local working notes
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Added

- Six Module 1B pattern drafts, each with a control-plan template: single point of metric computation, statistical process control for data pipelines, detecting plausible-but-wrong outputs, golden question sets for AI analytics validation, agreement measurement for analytics QA, and schema drift detection and data contracts.
- The first Module 3 pattern draft, the business context file an AI assistant reads before it answers, with a copyable context-file template.
- Six metrics-library topic pages: attribution, conversion rate, time to convert, active users, retention, and benchmarks.
- The first guided skill, attribution design, and the skill template every skill follows.
- `llms.txt`, the file an AI assistant reads after the reader pastes one line: what the library is, which page to fetch for which problem, and what to hand the reader at the end. The README carries the line. CI fails when a finished page has no row in the file.
- Repository skeleton: module charters, pattern template, prior-art and references scaffolding, deliverable-type homes, community files, CI.
- First pattern draft: state-based retention measurement (M1-11, Adapted from Duolingo's published growth model), with Module 1's first control-plan template, first prior-art row, and first verified REFERENCES entries.
4 changes: 1 addition & 3 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,13 @@ We agree to restrict the following behaviors in our community. Instances, threat

Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm.

When an incident does occur, it is important to report it promptly. To report a possible violation, **[NOTE: describe your means of reporting here.]**
When an incident does occur, it is important to report it promptly. To report a possible violation, open an issue in [this repository's issue tracker](https://github.com/h3qing/open-analytics-framework/issues).

Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution.


## Addressing and Repairing Harm

**[NOTE: The remedies and repairs outlined below are suggestions based on best practices in code of conduct enforcement. If your community has its own established enforcement process, be sure to edit this section to describe your own policies.]**

If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped.

1) Warning
Expand Down
2 changes: 1 addition & 1 deletion CONTENT_BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The Module column uses `1A` (definition quality) and `1B` (data integrity) for t
|---|---|---|---|---|---|---|---|
| MX-01 | 1A | Segmentation | topic | — | ready | Author's own thesis: segmentation is where the manufacturing lens stops working | 1 session |
| MX-02 | 1A | Benchmarks, and how to check a number before you steer by it | topic | — | drafted | Five failed traces: Reichheld 5%, 5x-cheaper, 7-friends, Groove 71%, DAU/MAU 50% | 1 session |
| MX-03 | 1A | Retention | topic | — | in progress | Fader & Hardie sorting effect, NRR filings, author's two-axis split | 1 session |
| MX-03 | 1A | Retention | topic | — | drafted | Fader & Hardie sorting effect, NRR filings, author's two-axis split | 1 session |
| MX-04 | 1A | Time to convert | topic | — | drafted | Little's Law, flow metrics; merged, author answers folded in | — |
| SK-01 | 1A | Defining your definitions (guided skill) | skill | — | ready | Facebook single-definition story (Schultz lecture, S-1) | 1 session |
| SK-02 | 1A | Is this a vanity metric? (guided skill) | skill | — | candidate | Ries 2009 posts, data-theater critique, Twitter mDAU story | 1 session |
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ Thank you for considering it. The most valuable contribution this project can re

- **Review a pattern.** Open an issue or comment on the pattern's file. Disagreement with a pattern's Position section is especially welcome; that section exists to be argued with.
- **Propose a pattern.** Use the pattern-proposal issue template. Proposals need the problem, the position ("do X, not Y"), and honest prior art — a pointer to the closest existing writing, even if it partly covers your idea.
- **Contribute a case study.** Via Discussions. Anonymized is fine; fabricated is not.
- **Contribute a case study.** Open an issue. Anonymized is fine; fabricated is not.
- **Fix errors.** PRs for factual errors, dead links, or broken citations are always welcome.

## Ground rules

- **Provenance is load-bearing.** Every normative claim cites prior art via a key in [REFERENCES.md](REFERENCES.md) or is explicitly tagged `Original` with a recorded search in [docs/prior-art.md](docs/prior-art.md). CI enforces this.
- **Provenance is load-bearing.** Every pattern lists its sources in a Sources & Stories section, every citation key resolves in [REFERENCES.md](REFERENCES.md), and a claim that is the framework's own says so in the text. CI checks the section and the keys. The landscape each pattern came from is recorded in [docs/prior-art.md](docs/prior-art.md).
- **Vendor neutral.** Patterns describe categories of tool and classes of failure, never named products.
- **No confidential material.** Do not contribute your employer's internal metrics, system names, or numbers. Abstract to how the class of problem behaves.
- **Template discipline.** Pattern content follows the [eight-section template](docs/pattern-template.md) exactly.
Expand Down
8 changes: 8 additions & 0 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

One entry per judgment call: decision, alternatives, why.

## 2026-09-21 — Pre-share cleanup: author notes go invisible, indexes drop the making-of, promises point at the roadmap

**Decision:** three calls made while cleaning the reader-facing pages before the repository is shared widely. (a) Author notes that were visible as `[TODO(heqing): …]` blocks on the reading guide and the module charters move into HTML comments, and a section whose only content was such a note is hidden whole rather than left as an empty heading. Nothing is written in their place, per hard constraints 6 and 7. (b) Index pages stop carrying maintainer status. The metrics library table now says what each page answers, in the words `llms.txt` already uses, and the skills index, the prior-art review, the stubs and the Module 1 charter lose their phase numbers and backlog IDs; row-level status stays in the backlog. (c) The reference-architecture and implementation-guide indexes no longer promise a count per release, since the roadmap disagreed with them; they point at the roadmap instead.

**Alternatives:** fill the empty sections from lines the README already carries, which would put generated prose where the author's voice is expected; keep the visible notes as an honesty signal, which the 2026-08-11 entry already rejected; keep the counts and reconcile the roadmap to them.

**Why:** a page that shows its own outline reads as unfinished whatever its contents, and a reader arriving from a share link should meet the content, not the process that produced it.

## 2026-09-16 — M1-05 argues with the single-owner key its sibling page installed, scores values by raw agreement and verdicts by kappa, and refuses to score disputed questions

**Decision:** four calls made while drafting the agreement-measurement pattern. (a) The Position argues with the single owner writing the reference answers, the practice M1-04 itself installs. The page keeps that owner and treats the owner's answers as a gauge with an unmeasured error: a second person answers the same twenty questions blind before the assistant is scored, the human-to-human rate is the ceiling the assistant is held to, and an assistant inside that rate's band gets no model work. (b) Values within tolerance are scored by raw percent agreement, because two numbers landing within a tenth of a percent by chance is not a real risk; kappa is reserved for pass-or-fail verdicts and is reported beside the raw rate, never alone. The fixed kappa bands the quality manuals grade against are not printed, because no source inside the ten-year window could be verified for them, and the question is left for the author. (c) The band is plus and minus the square root of p(1−p)/n, about two questions at twenty, and the floor of twelve agreements in twenty below which the questions rather than the people are the problem is likewise the framework's own; both are marked as such on the page and in the template. (d) A question the two people disagree on is not scored for or against the assistant. It routes to one line in the context file and a dated new row in the key, so the assistant is scored only on settled questions, and its misses there are the only errors that are its own. (e) Three calls from the author's review, folded in on 2026-09-17: the quality manuals' kappa bands are not used, because a percentage difference can be a delay, an exclusion, a calendar, or two different metrics landing near each other, and no band separates those; the logic is checked before any statistic, through a three-bullet conditions summary under every answer (what was excluded, which calendar, when the data was pulled) that an assistant can write and a person can verify; and the per-metric tolerance is set by a person as the reporting difference the operation can live with, before the study. Nobody is the better judge; the study is run for alignment, and its working signal is that important calls spend their time on actions rather than on whose number is right.
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,15 +59,15 @@ The primary worked case is early-stage and AI-native companies; the patterns gen

## What ships

Four deliverable types, each with a visible home here: the framework itself (`docs/modules/`), best-practices documentation, reference architectures (`reference-architectures/`), and implementation guides (`implementation-guides/`). Every module ships at least one copyable control-plan template in `templates/`. A diagnostic skill (`skill/`) assesses an organization's readiness before it deploys an AI analytics agent.
Four deliverable types, each with a visible home here: the framework itself (`docs/modules/`), best-practices documentation, reference architectures (`reference-architectures/`), and implementation guides (`implementation-guides/`). Every module ships at least one copyable control-plan template in `templates/`. Guided skills (`skill/`) walk a team through one decision at a time; the planned diagnostic skill assesses an organization's readiness before it deploys an AI analytics agent.

## Sources

Every pattern lists the stories and research it draws from — the Duolingo growth model, practitioner writing, prior-art research recorded in [docs/prior-art.md](docs/prior-art.md) — so you can go back to the originals if you want to go deeper. All citations resolve to [REFERENCES.md](REFERENCES.md).

## Status

Pre-release. Prior-art review in progress; module content follows. See [ROADMAP.md](ROADMAP.md).
Pre-release, with no version tagged yet. Every page on `main` is a full sourced draft awaiting the author's final pass: the patterns listed in the [Module 1](docs/modules/01-ai-data-quality/README.md) and [Module 3](docs/modules/03-ai-agent-integration/README.md) charters, the topic pages in the [metrics library](docs/metrics/README.md), the control plans in [templates](templates/README.md), and one [guided skill](skill/README.md). The release plan is in [ROADMAP.md](ROADMAP.md).

## Contributing

Expand All @@ -79,4 +79,4 @@ Documentation: [CC BY 4.0](LICENSE). Code and templates: [MIT](LICENSE).

## Author

<!-- TODO(heqing): author bio -->
Maintained by [Heqing Huang](https://github.com/h3qing). Bio on [LinkedIn](https://www.linkedin.com/in/hhq/).
2 changes: 1 addition & 1 deletion REFERENCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Bibliography for the framework. Every citation key used anywhere in `docs/`, `re

## Entries

_The full Phase 0 seeding is pending; entries below were verified individually as units were synthesized._
_Entries are added as pages are drafted, each verified individually before it enters._

- **[ACZEL-2021]** — Aczel, Balazs; Szaszi, Barnabas; Nilsonne, Gustav; van den Akker, Olmo R.; et al. "Consensus-based guidance for conducting and reporting multi-analyst studies." _eLife_ 10, e72185, November 9, 2021. doi:10.7554/eLife.72185. Read in full from the publisher. A 54-author consensus paper produced by a reactive-Delphi panel of methodologists, defining a multi-analyst study as one that engages multiple analysts to independently analyze the same data. Verified verbatim: "Co-analysts should not communicate with each other about their analyses until after all initial reports have been submitted", and the recommendation to state the research question "without communicating the lead team's preferred analysis choices or expectations about the conclusions". Cited for the blinding rule; the guidance is written for research studies, and the framework carries it to a two-person study inside a company. <https://elifesciences.org/articles/72185>
- **[AGENTSMD]** — AGENTS.md. "A simple, open format for guiding coding agents." agents.md, fetched August 26, 2026. Open format stewarded by the Agentic AI Foundation under the Linux Foundation; the page describes it as "a README for agents: a dedicated, predictable place to provide the context and instructions" and confirms there are no required fields, just standard Markdown. The key carries no year because the page is an undated living document. The adoption figure ("used by over 60k open-source projects") is self-reported with no methodology; attribute it, never assert it. The format targets coding agents; carrying business context in it is this framework's extension, not documented practice. <https://agents.md/>
Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Roadmap

Each tag below is a real git tag and a GitHub release with notes, aligned to [CHANGELOG.md](CHANGELOG.md).
Each version below becomes a git tag and a GitHub release with notes when it ships, aligned to [CHANGELOG.md](CHANGELOG.md). No version is tagged yet.

| Tag | Contents | Status |
|---|---|---|
Expand All @@ -9,7 +9,7 @@ Each tag below is a real git tag and a GitHub release with notes, aligned to [CH
| — | One pasted line gets a reader in: `llms.txt`, the line on the README, CI that keeps the file current | done, 2026-09-16 |
| — | Diagnostic skill v0.1 published | not started; the first guided skill, attribution design, is drafted |
| v0.1.0 | Module 1's core patterns: the 1B integrity patterns, on the 1A topic pages already drafted | drafted and on main: single point of computation, statistical process control, plausible-but-wrong, golden question sets, schema drift; the tag follows the author's voice pass |
| v0.2.0 | Module 1 complete, both halves, and the first reference architecture | in progress: agreement measurement and the segmentation topic page are in review; four 1B patterns, the reference architecture, and the remaining 1A pages are open |
| v0.2.0 | Module 1 complete, both halves, and the first reference architecture | in progress: agreement measurement is drafted and on main; the segmentation topic page, data quality SLOs and the reconciliation protocol are in review; the remaining 1B patterns, the reference architecture, and the remaining 1A pages are open |
| v0.3.0 | Second module core patterns, first implementation guide, skill v1.0 | not started; the module 3 context-file pattern and its template are already on main |
| v1.0.0 | All four modules at first complete draft | not started |

Expand Down
Loading
Loading