Skip to content

Coherent agent-facing text: one inventory, a CI lint, and protocol friction as a release metric #487

Closed

Description

@renmengye

Agents learn the kernel's protocol only from the text it shows them: the brief, each verb's help, refusals, wake messages and kernel notes. That text is written in many places and has drifted (see #484: one verb named end while every help line says "end your turn", and one refusal message used for two opposite situations, with kernel-internal words like "leg").

The cost is measurable. In a recent study (single-agent runs on one benchmark, six author models), the share of runs that hit a protocol refusal and the refusals per run were:

Condition Runs hitting a refusal Refusals per run
main author model, depth cap 1 7 / 15 5.1
main author model, depth cap 2 4 / 15 2.7
main author model, default cap 1 / 20 0.2
a second open model, default cap 6 / 15 6.0 (never completed a submit)

Friction is highest exactly where the agent has the fewest turns to recover, so incoherent text costs research, not just tokens.

Proposal

  1. One reviewable inventory. Every agent-facing string (brief sections, verb help and descriptions, refusals, wake and kernel messages) is generated from, or registered in, one module that can be printed and reviewed as a whole.
  2. A CI lint over that inventory:
    • one term per concept (verbs, turn vs run, sleep/wake), checked against a small glossary;
    • no kernel-internal vocabulary;
    • every refusal names the next step.
  3. Friction as a release metric. The pre-release canary runs report refusals per run by author model; a release that raises it does not ship without an explanation in the release PR.

Related: #484 (end → stop), which is the first concrete fix this inventory would have caught. Planned for 0.4; sequenced with the attempt.py split so the inventory module is designed once.

Activity

  1. added 2 commits that reference this issue on Oct 24, 2026
  2. renmengye commented on Oct 24, 2026

    @renmengye
    MemberAuthor

    All three parts are merged: the agent-text inventory and lint (#574), refusal wording fixes (#574), and refusals per run by author model in outerloop trajectory with the RELEASING.md canary step (#577). Closing; the next release's canary is the first to report the metric.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions