Repository navigation
Measure how often your own people agree before you score the assistant - #41
Merged
Merged
Conversation
h3qing
force-pushed
the
agreement-measurement
branch
4 times, most recently
from
September 17, 2026 03:40
93c3f98 to
6148322
Compare
…mplate An assistant scored 70 percent against one person's reference answers, and the team is about to spend a week on prompts. Nobody asked whether two of their own people would agree on those twenty questions. In the published studies where independent teams analysed the same data, they did not, and inside a company the cause is nearly always an exclusion, a calendar, or the time the data was pulled. This page has a second person answer the twenty questions blind with the conditions under each answer, checks the conditions before the values, holds the assistant to the rate at which the two people agree, and sends every disputed question to the definitions instead of the model. Nineteen references added.
h3qing
force-pushed
the
agreement-measurement
branch
from
September 17, 2026 03:41
6148322 to
9b3f281
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An assistant scored 70 percent against one person's reference answers, and the team is about to spend a week on prompts. Nobody has asked whether two of their own people would agree on those twenty questions. In the published studies where independent teams analysed the same data, they did not, and inside a company the cause is nearly always one of three things: an exclusion, a calendar, or the time the data was pulled.
This page has a second person answer the same twenty questions blind before the assistant is scored, with three bullets under each answer saying what was excluded, which calendar, and when the data was pulled. Agreement is checked on the conditions first, then on the value within a tolerance a person set for that metric. The rate at which the two people agree is the ceiling the assistant is held to. A question the two people read differently is not scored; it goes to the definitions and the context file. The template carries the answer sheets, the spreadsheet arithmetic, what each rate means, and the prompts.
Files:
docs/modules/01-ai-data-quality/agreement-measurement.mdtemplates/01-ai-data-quality/agreement-measurement-control-plan.mdREFERENCES.mdllms.txt,CONTENT_BACKLOG.md,DECISIONS.md,docs/modules/01-ai-data-quality/README.md,templates/README.md