Skip to content

Measure how often your own people agree before you score the assistant - #41

Merged
h3qing merged 1 commit into
mainfrom
agreement-measurement
Sep 17, 2026
Merged

h3qing merged 1 commit into
mainfrom
agreement-measurement

Conversation

@h3qing

@h3qing h3qing commented Sep 16, 2026 •

Copy link
Copy Markdown
Owner

An assistant scored 70 percent against one person's reference answers, and the team is about to spend a week on prompts. Nobody has asked whether two of their own people would agree on those twenty questions. In the published studies where independent teams analysed the same data, they did not, and inside a company the cause is nearly always one of three things: an exclusion, a calendar, or the time the data was pulled.

This page has a second person answer the same twenty questions blind before the assistant is scored, with three bullets under each answer saying what was excluded, which calendar, and when the data was pulled. Agreement is checked on the conditions first, then on the value within a tolerance a person set for that metric. The rate at which the two people agree is the ceiling the assistant is held to. A question the two people read differently is not scored; it goes to the definitions and the context file. The template carries the answer sheets, the spreadsheet arithmetic, what each rate means, and the prompts.

Files:

  • docs/modules/01-ai-data-quality/agreement-measurement.md
  • templates/01-ai-data-quality/agreement-measurement-control-plan.md
  • REFERENCES.md
  • llms.txt, CONTENT_BACKLOG.md, DECISIONS.md, docs/modules/01-ai-data-quality/README.md, templates/README.md

@h3qing
h3qing force-pushed the agreement-measurement branch 4 times, most recently from 93c3f98 to 6148322 Compare September 17, 2026 03:40
…mplate

An assistant scored 70 percent against one person's reference answers,
and the team is about to spend a week on prompts. Nobody asked whether
two of their own people would agree on those twenty questions. In the
published studies where independent teams analysed the same data, they
did not, and inside a company the cause is nearly always an exclusion,
a calendar, or the time the data was pulled. This page has a second
person answer the twenty questions blind with the conditions under each
answer, checks the conditions before the values, holds the assistant to
the rate at which the two people agree, and sends every disputed
question to the definitions instead of the model. Nineteen references
added.
@h3qing
h3qing force-pushed the agreement-measurement branch from 6148322 to 9b3f281 Compare September 17, 2026 03:41
@h3qing
h3qing merged commit 7666d4a into main Sep 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant