Alignment tuning against the benchmark: move recall and correspondence F1

[houfu]

The first tuning pass against the benchmark ADR-0021 built for it. One rule changed in the aligner, one default moved, nothing changed in the benchmark's measurement.

Before / after

Synthetic tier, difflib backend, all labelled blocks. The hand tier is 1.0000 on every column before and after.

Metric	Before	After	Gate	Standing
correspondence F1	0.9412	0.9750	≥ 0.95	met (was xfail(strict=True), mark removed)
precision	0.9457	0.9753	—
recall	0.9367	0.9747	—
spurious-match rate	0.0000	0.0000	—
renumber recall	0.9795	0.9932	—
move recall	0.7391	0.7826	≥ 0.90	not met, and unreachable — see below
move precision	1.0000	1.0000	no wrong move	met, both tiers
exact wrong matches	84 / 1349	37 / 1349	—
hand tier F1	1.0000	1.0000	≥ 0.85	not assessed (draft labels)
hand tier unreviewed moves	3	3	0	unchanged — no new move on the hand tier

Per plan, synthetic, F1 and move recall:

Plan	F1 before	F1 after	Move recall before	Move recall after
heavy	0.9163	1.0000	n/a	n/a
light	0.9333	1.0000	n/a	n/a
mixed	0.9204	0.9474	0.8571	0.8571
move-heavy	0.8880	0.9094	0.6875	0.7500
renumber-storm	1.0000	1.0000	n/a	n/a
structure	0.9973	0.9973	n/a	n/a
table	1.0000	1.0000	n/a	n/a

The four pairs that carried almost all of the loss:

Pair	F1 before	F1 after
repetitive-schedule-heavy	0.2500	1.0000
repetitive-schedule-light	0.3448	1.0000
repetitive-schedule-mixed	0.1818	0.5000
repetitive-schedule-move-heavy	0.0345	0.2000
msa-markdown-move-heavy	0.9852	1.0000 (moves 3/4 → 4/4)

What changed, and why

1. A structural-nearness step in the tie-break (redlines/alignment.py). ADR-0032 said "highest score, then equal role, then document order". Document order is right until a bucket holds more than one equal, and the report found the case where it is not: a schedule of thirty byte-identical paragraphs puts all thirty in one exact bucket, an edited test paragraph matches no key and so consumes no source, and every identical pair below it is off by one. That single rule was the report's 84 wrong exact matches.

The order is now: highest score, then equal role, then the structurally nearer candidate — the one whose sibling position is closest, in whole blocks — then earliest source, then earliest test. It applies in exact, label and fuzzy alike. Two things make it safe rather than clever:

• Nearness never creates a match. It is consulted only between candidates that already scored the same, exactly as role is.
• Across one exact bucket the assignment is order-preserving, so two exact pairs can never cross each other and produce a move on no evidence.

Distance is integer arithmetic on sibling positions, never a float, so determinism is unchanged and the tie-break is total.

Scaling positions to a common length was tried and rejected on measurement, and the rejection is recorded in the ADR: it assumes a sibling group grew evenly, and on govinfo-hr4668-ih-to-rh — ten sections becoming twenty by insertion in one place — it sends source section 6 to test section 16 rather than test section 7, costing four hand-tier correspondences and inventing seven moves on a tier that had been perfect. Absolute distance says "as near as it was", which is the weaker and truer claim.

2. label_min_similarity 0.35 → 0.50. The one default that moved. 0.35 rejected the sample pair's 0.20 case, and the report found the case it was too low for: in msa-markdown-move-heavy, clause 7.1 moved to 5.4 with byte-identical text while a different clause took the number 7.1. The two 7.1s score 0.3636 — just over the old floor — so the label pass consumed both blocks and the move pass never saw the perfect cross-scope match. The sweep is flat from 0.37 to 0.65 on both tiers; 0.50 is the middle of that plateau rather than its edge.

3. Every other default was swept and left alone. fuzzy_min_similarity, positional_min_similarity, move_min_similarity, move_tie_margin, move_min_tokens, move_kinds, fuzzy_window, table_fuzzy, max_comparisons — each one across the whole committed corpus. None of them is suppressing a true move: loosening any move knob bought zero synthetic moves and cost the hand tier correspondences and new unreviewed moves, which is ADR-0009's asymmetry working. budget_exhausted is false on every pair of both tiers.

4. ADR-0032 amended in place, dated 2026-09-05. The tie-break paragraph and the label_min_similarity row are rewritten where they stand; a new Amendment, 2026-09-05: what the benchmark said section records the before/after, the scaled-position alternative and its rejection, and the move-recall ceiling. The Revisit when section is updated to point at it.

5. benchmark/REPORT.md narrative updated (benchmark/report.py) — its two now-stale paragraphs said the exact pass "takes the first" and blamed move recall on the move-heavy plan. Both are rewritten to say what is actually true, including the ceiling. Numbers and metric definitions are untouched: the benchmark was not edited to flatter the engine.

6. Three new regression tests in tests/test_alignment.py, one per property: an edited block does not shift the identical pairs below it, equally exact candidates never cross, and nearness is absolute rather than scaled. No existing test was removed or weakened.

Which goldens changed, and which nodes

No change-tree node changed. No matched_by changed. No confidence changed.

tests/corpus/sample_pair/expected/change_tree.contract.json and change_tree.markdown.json were regenerated with tests/corpus/sample_pair/regenerate.py. Each file has exactly two changed lines, both the same recorded configuration value in two places — config.alignment.label_min_similarity and the top-level alignment.config.label_min_similarity — 0.35 → 0.5. Nothing else in either file moved, and the M1 block-tree goldens (source.*.json, test.*.json) are byte-identical.

The eight promises were re-checked and all eight hold unchanged, on both profiles: the widened definition is one modify with one inline insert; the moved clause is one move plus a separate modify on its body; the inserted clause renumbers the two clauses below it; the cross-reference edit touches only the cross-reference span; the deleted sub-clause is one delete that keeps its label; the inserted table row is one row-level insert; the whitespace-only change produces no node; and the edit inside the repetitive schedule is found. The pair still produces exactly ten change nodes.

redlines/schemas/comparison-v2.json's examples block carries the same default and was updated with it.

What is still red, and why

Move recall is 0.7826 against ADR-0009's 0.90, and it cannot reach the bar on this corpus without inventing a move. Its gate test stays xfail(strict=True) with the reason rewritten to say so. The numbers:

• 23 labelled moves on the synthetic tier; 18 found; move precision 1.0000.
• All five misses are in repetitive-schedule-mixed (1) and repetitive-schedule-move-heavy (4). Both source documents contain exactly one distinct paragraph text, repeated thirty times — no label, no heading, no parent to tell one sibling from another. Each missed move's source block is one of 30 byte-identical candidates, and its true partner is one of 24 or 28 byte-identical candidates on the test side.
• There is no property of those documents that separates the labelled partner from the other 29. Reporting one would be a one-in-thirty guess. The move pass's uniqueness rule sees the ambiguity and stays silent, which is what it was built to do.
• 18/23 = 0.7826 is the ceiling that keeps move precision at 1.0000 on both tiers.

ADR-0009 says the bar is not to be lowered quietly, and a tuning pass is not where it would be lowered. Three ways out exist and all of them are the maintainer's call, not this PR's: extend the corpus so the repetitive-schedule pairs are not 22% of the labelled moves; label those five as unscored on the grounds that they are unrecoverable (a change to the benchmark, which this PR was told not to make); or reopen ADR-0009 and ship moves with the number published, which is what its own Revisit when section contemplates.

Two smaller things for the record:

• repetitive-schedule-mixed (0.5000) and repetitive-schedule-move-heavy (0.2000) still carry the corpus's remaining correspondence loss, for the same reason. With 30 identical siblings, nearness is the only evidence there is, and where the generator's truth is not the nearest candidate the exact pass is wrong and cannot know it. The remaining 37 wrong exact matches are all of this shape.
• ADR-0008's review gate, one reading: with nearness in, the label pass's unique contribution on the synthetic tier is 0 — every pair it made, some other pass would have made too — while its wrong-match count is also 0 over 85 pairs. Low unique and high wrong is the cut signal; low unique and zero wrong is not, so label stays. Noted in the ADR so the next reading has a baseline.

Checks

uv run mypy redlines tests benchmark clean. uv run pytest 1637 passed, 2 skipped, 6 xfailed — including the two N2 timing tests (the new bucket walk is O(k) and adds no similarity computation; 2,000 identical paragraphs in one sibling group align in 9 ms). benchmark/REPORT.md and benchmark/results/latest.json are regenerated and committed, and the byte-stability test passes.

🤖 Generated with Claude Code