Skip to content

[roadmap] Range and list line-anchors are silently dropped by the review validator #83

Description

@azevedo

Problem

skills/ba-review/SKILL.md Bullet-anchor grammar requires line to be a single positive integer.
Reviewers routinely emit line ranges (:4-6, :20-37) and comma lists (:21,27) instead.
When the consolidation step enforces the grammar literally, every such finding is dropped as
dropped_no_fileline — silently, with only a counter to show for it.

Observed on a dry-run of the new comment-quality-reviewer
(docs/reviews/2026-08-09-221623-staged/):

⚠ Dropped: 10 findings (no file:line)
  comment-quality-reviewer: 6  (:4-6, :10-13, :8-9, :15-18, :32-34, :1-14)
  complexity-reviewer:      4  (:20-37, :17-18, :15-37, :21,27)

The six lost from comment-quality were its highest-value findings — a High/100 return-shape
enumeration, Medium/75 type-restating params, a Medium/50 missing caller contract, and a Medium/50
standard-mechanics comment. What rendered was the tail. Worse, the consolidated summary.md then
carried a Looks Good entry praising the exact block another reviewer had condemned at High/100 —
because only the surviving verdict made it through.

Why it went unnoticed until now

The "validator" is prompt-implemented, not code — the consolidating model reads the grammar and
applies it. So enforcement is stochastic, and that is the actual bug surface:

Run Range/list anchors present Reported dropped
2026-08-02-224403 6 (complexity 4, error-handling 2) 1
2026-08-04-112242 2 (error-handling 2) 0 — "No records dropped"
2026-08-09-221623 10 10

Range anchors have been emitted in every persisted run on record. Earlier runs tolerated or quietly
normalised them; the 2026-08-09 run enforced the letter of the grammar. So this is not a new
regression — it is a long-standing silent-loss channel that only became visible when a reviewer that
quotes multi-line blocks made the volume large enough to notice.

Why comment-quality-reviewer surfaced it

Structural, not accidental. The other seven anchor to a single statement. A doc comment is a
multi-line block, so quoting one invites a span. It will keep hitting this hardest.

Already mitigated in that one agent (v0.45.0): its ## Principles now require one positive
integer and name the failure. The other seven and the shared grammar are untouched.

Options

  1. Tighten the shared grammar + the Task dispatch template so every reviewer is told
    "single integer, never a range" once. Fixes all eight; touches a consumer file.
  2. Make the consolidation step normalise :4-6 to :4 instead of dropping. Recovers findings
    from reviewers that ignore the rule, and removes the stochastic-enforcement gap.
  3. Both — normalise defensively, instruct explicitly.

Item 2 is the one that closes the silent-loss channel: instruction alone still depends on eight
agents complying every run.

Not in scope of the port

docs/plans/2026-08-09-feat-comment-quality-reviewer-builtin-plan.md explicitly excluded
dispatch-template edits, so only the single-agent mitigation shipped with v0.45.0.

Metadata

Metadata

Assignees

No one assigned

    Labels

    cluster:review-quality/ba:review & review-plan qualityreadyClear starting point — can build now

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions