Skip to content

Duplication check: keep one qualifying place per token run so the window cannot hide a real pair (#42) - #44

Merged
kikashy merged 1 commit into
mainfrom
duplication-cap-42
Oct 26, 2026
Merged

kikashy merged 1 commit into
mainfrom
duplication-cap-42

Conversation

@kikashy

@kikashy kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Closes #42.

What changed

lib/duplication.py: each run of min_tokens tokens still keeps its 16 most recent places and compares them nearest first. Besides them it now keeps the most recent place that itself spans at least min_lines lines: when that place leaves the 16 (and none of the 16 spans min_lines lines) it is held apart, compared after the recent places, and never evicted until a newer place that spans min_lines lines replaces it. Two such places always make a reported pair, so 16 or more short copies between two real copies no longer hide the real pair. Nearest-first comparison, the once-per-block examination, the same-file rule and the ignore blocks are unchanged.

README: the sentence that states the 16-place bound now says what is kept beyond it.

Scope of the guarantee: it holds when a run of the earlier real copy spans min_lines lines on its own. A copy longer than min_tokens in which no single run spans min_lines lines is still covered by the 16-place window only.

Fixture

  • test_a_copy_is_named_past_more_compact_places_than_the_recent_window: a spread copy of a 31-token function (7 lines), 17 files with the same tokens on 3 lines, and the spread copy again. Expected: exactly lib/a.js:2-8 and lib/c.js:2-8: 31 tokens.
  • ..._in_one_file: the same in one file, each compact copy after a line holding one name of its own so no copy joins its neighbours.

Both fail on main (exit 0, no pair) and pass on this branch.

Mutant

In a scratch copy, the kept place was evicted (the kept[value] = earlier[0] assignment replaced by pass): both new tests fail, the other 41 pass.

Cost

No synthetic-tree test in the suite; on the kit itself the check takes 0.13-0.15 s on main and 0.14-0.15 s on this branch, peak memory about 27.7 MB on both (three runs each).

Check summary

jpdev check --changed (lib/ and tests/ are unmapped, so the whole list ran): 8 passed, 0 failed, 0 skipped, 0 not run. diff-check, local-paths, agent-docs, bash-n, duplication (no duplicate block, 21 files), json, shellcheck, tests (430 passed, 0 failed). tests/duplication_test.py: 43 tests OK (41 existing + 2 new).

🤖 Generated with Claude Code

…dow cannot hide a real pair (#42)

Each run of min_tokens tokens keeps its 16 most recent places and, once
it has left them, the most recent place that itself spans min_lines
lines. That place is compared after the recent ones and is never evicted
until a newer qualifying place replaces it, so 16 or more short copies
between two real copies no longer drop the real pair.

Two fixtures put 17 compact copies between two spread copies, across
files and in one file; both fail on main and on a mutant that evicts the
kept place.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Review record: devkit PR #44 (issue #42), head fe2adb5 on base 1f02688

The fix does what it claims and nothing else: the kept place is tried only after every recent place fails, so it can only add pairs main hid. Both new tests fail on main and on a mutant, and the stated limit is real but narrow.
Narrow review under a budget. Same-vendor review, not a cross-vendor claim. 0 HIGH, 0 MEDIUM, 2 LOW.

# Finding Severity Evidence What to do
1 Stated limit confirmed: a copy with no single 100-token run that spans 10 lines is still hidden behind 16+ compact copies LOW (it is documented, and the fix is not hollow) Probe at 100/10: a 12-line copy at 13 tokens/line (156 tokens, every run spans at most 9 lines), with 17 reflowed 3-line copies between the two real ones: no pair on the branch or on main. The same copy with 0 copies between is reported (156 tokens); with 16 copies it is hidden on both. At 11 tokens/line (12 lines) and 9 tokens/line (14 lines), the branch reports the pair and main does not. Kit density (python and shell): mean 11.2 tokens/line, median 9, p90 23; 60.4% of all 73,173 100-token runs span at least 10 lines None for merge. Optional: state the limit in the README next to the guarantee
2 README "copies whose runs span min_lines lines" reads as if every run must span. The real condition is weaker: the earlier copy needs one run that spans on its own. The sentence is true as a sufficient condition, but the limit is left implicit LOW (docs) duplicates(): only the earlier place's span gates kept; the later place's span only clears it Optional wording: "two copies of which the earlier has a run that spans min_lines lines"; add "a copy with no such run is compared only with the 16 most recent places"

Not hollow: hiding a pair takes 16 or more compact copies of the same 100+ tokens between the two real copies, which is rare in real code. Even then, any real copy with a stretch near the kit's median density (9 tokens/line) has a spanning run and is now reported. Only copies at the threshold that stay dense throughout (above about 10 tokens/line over every 100-token stretch) remain window-only.

Checked in the diff:

  • Candidate order: the recent places come first, nearest first, then the kept place (an itertools.chain).
  • One examination per block: the examined key and state are unchanged, and the kept comparison goes through the same _examine.
  • Same-file rule: skip_file also applies to the kept place.
  • Ignore blocks: barrier ids are untouched.
  • kept is cleared only when a newer place that spans arrives, and is set only when no place that stays in the window spans. Its invariant holds: a kept place exists only while no recent place spans.
  • No other behaviour changes. Public text (diff, commit) has no paths, names or organizations.

What I ran (Python 3.12, memory cap and timeout on every run, scratch copies deleted after each run):

  • python tests/duplication_test.py: 43 tests OK on the branch.
  • Branch tests against main's lib/duplication.py: the 2 new tests FAIL. Six other errors appear because the scratch copy has no templates/ or jpdev.json; they occur identically on the branch copy and are not related.
  • Mutant (the kept place evicted by any newer place: if older is not None: without and spans(here)): both new tests FAIL.
  • The check over the kit, 3 runs each: branch 0.14 s, main 0.15–0.17 s. Cost is unchanged.
  • A probe tree (19 small files) for finding 1, and a density count over the kit's tracked python and shell files.

merge: yes

@kikashy
kikashy merged commit d1a00b8 into main Oct 26, 2026
3 checks passed
@kikashy
kikashy deleted the duplication-cap-42 branch October 26, 2026 03:42
@github-actions

Copy link
Copy Markdown

🎉 Thank you for your contribution to Judgment Pack!

If you’d like another task, choose a path that interests you:

@kikashy kikashy mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplication check: keep one qualifying place per token run so the 16-place cap cannot hide a real pair

1 participant