[kikashy]

What it adds

• The duplication check-list step (lib/duplication.py, python3 standard library, vendored by a repository as scripts/dev/duplication.py). It is off, and its summary line says duplication: off: ..., until the repository's jpdev.json declares a duplication member: languages (any of go, javascript, python, shell, typescript), optional paths and exclude globs (* and ? within one segment, ** as a whole segment; default **), min_tokens (10 to 10000) and min_lines (1 to 10000). The files read are tracked or untracked-and-not-ignored files of the declared languages' extensions; symbolic links are not followed.
• Detection. Each language is read as tokens with its own comment and string syntax, so layout and comments do not hide a copy (identifiers are not renamed: a copy with changed names is not reported). Every run of min_tokens tokens is hashed with a rolling hash and compared with the previous place the same run appeared; the match is extended both ways to the whole shared block, examined once, and reported when each place spans at least min_lines lines. Three copies make two pairs (each against the one before it). Two places in one file may not overlap, so a repeated short pattern is not a copy of itself. A generated tree of about 3.6 million Go tokens ran in about 5 s and 380 MB.
• The report names each pair with both file:first-last ranges and the shared token count, and exits 1:

  duplication: 1 duplicate pair (at least 20 tokens over at least 5 lines, 2 files of python)
    src/ledger.py:4-9 and tools/report.py:8-14: 29 tokens
  

• The ignore block. The code between a comment holding duplication:ignore-start and a comment holding duplication:ignore-end is left out, in any comment syntax of the language (//, /* */, #); a barrier stands in for it so no block joins the code around it. The marker text in a string is not a marker. A start that is never closed, an end with no start and a start inside an open block are reported (exit 1, file:line).
• Refusals. An invalid declaration is refused with exit 2 before any file is read, and the message names the member, for example duplication: refused: jpdev.json duplication.min_tokens: an integer from 10 to 10000 is required (got 5).
• Public surface, changed with its vectors and documentation: templates/dev.json carries the step (after generated-docs); templates/jpdev.json shows the member; templates/jpdev.schema.json adds it, and the root now requires version and one of generators, changed or duplication (anyOf). lib/generated_docs.py and its copy scripts/dev/generated_docs.py accept the member and do not read it. README has a "The duplication check" section, docs/adopting.md a "Duplication check" section, docs/policy.md one sentence (duplication is where two copies drift; a repository opts in with its thresholds and marks an intentional copy with an ignore block), and docs/generated-docs.md names the member.
• On an existing tool. jscpd is the more complete detector (many more languages and normalization modes, HTML and JSON reports). This ships the standard-library detector because the kit's checks run with python3 alone and a Go-only or Python-only repository has no Node toolchain in CI.

Checks

Candidate: head 7008b8a, no uncommitted diff. On top of 5e8c374 (the feature), 91ebc02 merges main at c122806 (#40) and 7008b8a opts the kit itself in. The whole list ran once on that head:

scripts/dev/check.sh     (python3 = Python 3.12.11 first on PATH, GIT_EDITOR unset, TMPDIR in session scratch)
== check summary (dev.json) ==
PASS     diff-check               0s
PASS     local-paths              0s
PASS     agent-docs               0s
    sync-agent-docs: CLAUDE.md matches AGENTS.md
    sync-agent-docs: CLAUDE.md matches AGENTS.md
PASS     bash-n                   0s
    bash -n: every script parses
PASS     duplication              0s, python <home>/.pyenv/versions/3.12.11/bin/python3 (Python 3.12.11)
    duplication: no duplicate block (at least 100 tokens over at least 10 lines, 21 files of python, shell)
PASS     json                     0s, python <home>/.pyenv/versions/3.12.11/bin/python3 (Python 3.12.11)
    parsed 11 JSON documents
PASS     shellcheck               5s
    shellcheck: clean
PASS     tests                    89s
    430 passed, 0 failed
== 8 passed, 0 failed, 0 skipped, 0 not run ==

The earlier run on 5e8c374 alone (before the merge) was 7 passed, tests 429 passed, 0 failed.

The merge with #40 (91ebc02). Both pull requests extend jpdev.json. Git flagged two conflicts, the member set of config() in lib/generated_docs.py and its copy, and the root anyOf of templates/jpdev.schema.json; both are resolved to the union ({version, generators, changed, duplication}; version plus one of generators, changed, duplication). Two more did not conflict textually but broke a feature, and are fixed in the merge: lib/changed_checks.py refused every member except version, generators and changed, so jpdev check --changed would have refused templates/jpdev.json, and now accepts duplication; the standard-library schema reader in tests/changed_checks_test.py did not know enum, which the languages member uses, and now reads it. Probes on a scratch copy: dropping duplication from the changed-path members fails the shipped-mapping test for templates/jpdev.json and jpdev.json (refused at load); ignoring enum in the reader fails the duplication schema test (unknown language accepted); dropping either member from the generated-document set fails that member's test.

The kit on itself (7008b8a). jpdev.json declares {"languages": ["python", "shell"], "exclude": ["scripts/dev/**"], "min_tokens": 100, "min_lines": 10}, and dev.json runs python3 lib/duplication.py as the duplication step. The CI lint job runs it, and .github/ci-evidence.json lists it among the lint checks, with lib/duplication.py and jpdev.json as inputs. At those thresholds the only pairs in the tree are the vendored copies under scripts/dev/ (scripts/dev/check.sh and templates/check.sh, 1816 tokens; lib/generated_docs.py and its copy, 1518; lib/handover_check.sh and its copy, 184), which are excluded. At 50 tokens and 5 lines, five pairs inside test files would be reported. The schema vector test now also runs through the standard-library reader, so it no longer skips where jsonschema is missing.

Focused runs during development: tests/duplication_test.py 29 tests OK under Python 3.12 (jsonschema installed, so the schema vector test ran) and under 3.8; tests/generated_docs_test.py 20 tests OK against both helper copies; scripts/dev/check.sh --only diff-check,local-paths,json,bash-n,shellcheck.

tests/duplication_test.py (run by tests/run.sh) holds handwritten fixture repositories with hand-counted ranges and token counts: a copy above the threshold fails and names both places; the token threshold is inclusive (29 reported, 30 not); the shorter place decides the line threshold; a Go copy laid out and commented differently is found; three copies in one file make two pairs; a repeated short pattern is not reported; the ignore block suppresses a copy in Python (#) and Go (/* */ start, // end); the marker in a string is not a marker; the three marker problems; per-language tokenizer cases; off with no jpdev.json and with no member; languages, paths, exclude, ignored files and links; 23 refused declarations by exact message (empty, missing and oversized among them) and through the command; and the template manifest's own step run by templates/check.sh in a fixture repository (FAIL row with the pair in the summary block). The schema and the command agree on every vector (members, enum and bounds compared directly; the good and the 23 bad vectors validated with jsonschema when it is installed).

Safeguard probes. Each broke a scratch copy of lib/duplication.py once (run with DUPLICATION_HELPER), the clone unchanged:

Broken	Caught by
the detector never reports	9 tests, at their exit-status assertion (0, "no duplicate block", where 1 is expected), the template-step test among them
an ignore start is not honored	4 ignore-block tests (the copy is reported, the barrier is missing, the end markers become problems)
comment handling applied to strings	the string-marker test (a "never closed" problem instead of the pair) and 7 tokenizer cases
two places in one file may overlap	the repeated-pattern test (an overlapping pair reported)
the longer place decides the line threshold	the line-threshold test (reported at min_lines 7)
no bounds on the thresholds	4 refusal cases (min_tokens 9 and 10001, min_lines 0 and 10001: Refused not raised) and the command refusal test (exit 1 instead of 2)
no barrier for an ignored region	the barrier test (the code around the region joins)
an unknown member accepted	the refusal case for threshold (Refused not raised)

Schema probes (self-check b), each on a scratch copy of the schema: removing the shell enum value, the exclude member, the duplication branch of anyOf, the glob pattern, or the min_lines maximum each fails the schema tests.

Self-checks: (a) not applicable, dev.json declares no handover paths; (b) above; (c) the schema and the command read the same vectors and agree; (d) no mutation rows in this repository; (e) refusals tested for empty, missing and oversized members; (f) no checkpoint; (g) README, adopting guide, policy sentence and module docstring read against the code. CI owns the exact-head run of the same list.

Not in this PR

• No identifier renaming (a copy with changed names is not reported), no further languages, no machine-readable report.
• The five pairs that test files show at 50 tokens and 5 lines are not refactored; the kit's thresholds are 100 and 10.

Fix round

Candidate: head 5dd5f44 (one commit on 7008b8a), no uncommitted diff. It answers the clean-room review of 7008b8a.

Finding	Fix	Test, and the probe that breaks it
HIGH: only the most recent place of a run was compared, so a copy was dropped when the nearer place spanned too few lines	The 16 most recent places of each run are kept and walked nearest first until one makes a reported pair; examined still extends each diagonal block once. After an unreported run that repeats itself in one file, that file's other places are passed over for the run; other files are still compared	Three files (lib/a.js, compact lib/b.js, lib/c.js; also with the compact one sorting last) and the same-file variant (spread, compact, spread): a.js:2-8 and c.js:2-8 and one.js:2-8 and one.js:14-20, 31 tokens each. With 1 recent place (the old behaviour) both fail. With the same-file pass-over removed, the repeated-pattern test fails
MEDIUM: no template changed rule selected duplication	Every template rule that can match a file of a declared language selects it (including docs/**), as does the kit's own docs rule. docs/adopting.md says to do so	A test checks every rule of templates/jpdev.json and jpdev.json against sample paths the declaration reads. Dropping the step from the template's go/** rule or from the kit's docs rule fails it
MEDIUM: two surviving ignore-block mutants	New fixtures: a valid block plus a copy outside it in the same file (reported, report.py:9-15); two blocks whose surrounding code is the same 8 + 8 tokens (not joined); a block inside a copy splits it (14 tokens reported, not 29)	The mutant that drops pairs naming a file with a block fails 2 tests. The mutant where all barriers share one id fails the join test
MEDIUM: a tracked directory replaced by a link was followed	A path is read only when path.resolve() equals the path below the resolved root	A tracked inner/ replaced by a link to a directory outside the root: not read. Removing the check fails the test
MEDIUM: no size bound; MemoryError ended in a traceback	max_file_bytes (1024 to 104857600, default 1048576; schema and refusals included): a larger file is named on a not read: line and is not read. MemoryError is the refusal out of memory reading the files in scope; put large generated files in exclude or lower max_file_bytes (exit 2)	Over and at the limit; MemoryError raised by a mock of the detector, because it cannot be produced reliably in CI. Removing the limit fails the size test. Removing the catch fails the refusal test
LOW	A NUL byte in the first 8192 bytes: named and not read. multipleOf: 1 on the integers (redundant with type: integer, so no vector separates it; the standard-library reader in tests/changed_checks_test.py reads it). README sentences on the split copy, here-document markers and carriage-return-only files, each with a test	Removing the binary check fails the binary test

The detector on the earlier generated tree (about 3.6 million tokens) took 7.8 s and 380 MB after the change, against 5.4 s before.

The whole list ran once on 5dd5f44:

scripts/dev/check.sh     (python3 = Python 3.12.11 first on PATH, GIT_EDITOR unset, TMPDIR in session scratch)
== check summary (dev.json) ==
PASS     diff-check               0s
PASS     local-paths              0s
PASS     agent-docs               0s
    sync-agent-docs: CLAUDE.md matches AGENTS.md
    sync-agent-docs: CLAUDE.md matches AGENTS.md
PASS     bash-n                   0s
    bash -n: every script parses
PASS     duplication              0s, python <home>/.pyenv/versions/3.12.11/bin/python3 (Python 3.12.11)
    duplication: no duplicate block (at least 100 tokens over at least 10 lines, 21 files of python, shell)
PASS     json                     0s, python <home>/.pyenv/versions/3.12.11/bin/python3 (Python 3.12.11)
    parsed 11 JSON documents
PASS     shellcheck               4s
    shellcheck: clean
PASS     tests                    79s
    430 passed, 0 failed
== 8 passed, 0 failed, 0 skipped, 0 not run ==

Focused runs: tests/duplication_test.py 41 tests OK, tests/changed_checks_test.py OK, tests/generated_docs_test.py OK. CI owns the exact-head run.

Closes #35

🤖 Generated with Claude Code

[kikashy]

Clean-room review (same vendor): Claude Opus 5.5, effort xhigh, 2026-10-25, reviewed SHA 7008b8a

Same-vendor review at the maintainer's direction; not a cross-vendor claim. A fresh reviewer in a separate clone at the PR head; CI green. Condensed to its findings:

1. HIGH (lib/duplication.py:208-213, :239): a copy configured to be caught can pass. latest keeps only the most recent place of each token run; when that place fails the line threshold the pair is dropped and the earlier copies are never compared with each other. Probe: two 13-line, 104-token copies are reported; add a third copy of the same tokens reflowed to 8 lines in a file that sorts between them, and the run reports nothing; rename that file so it sorts last and the pair is reported again. The verdict depends on file names, against README:210. Fix: keep a bounded per-hash list of places and walk back past a place that fails the thresholds; add the three-file fixture.
2. MEDIUM (templates/jpdev.json:21-37, jpdev check --changed: run only the check-list steps the changed paths map to #40 integration): the template opts into three languages but no changed rule selects duplication, so a new copy passes jpdev check --changed (probed for a Go and a TypeScript change). Fix: add duplication to each template rule covering a declared language, and one sentence in adopting.md. The kit's own mapping is fine (an unmapped change runs the whole list).
3. MEDIUM (tests/duplication_test.py:254-298): two mutants survive all 29 tests: an ignore block silencing every pair that names its file (even a copy outside the block), and all barriers sharing one id (code around two blocks can join). The code is correct; nothing pins it. Fix: the two fixtures.
4. MEDIUM (lib/duplication.py:144-146 vs README:208 "a symbolic link is not followed"): only the last path part is checked; a tracked directory replaced by a symlink to outside the root is read and reported under the tracked path (hand-made worktree state only). Fix: require the resolved path to stay under the resolved root.
5. MEDIUM: no time or memory bound; a 23.7 MB generated file took 10.8 s and ~1 GB; a MemoryError ends in a traceback, not a named refusal. Fix: skip or refuse files over a byte limit by name; document that large generated files go in exclude.
   6-12. LOW: a one-line ignore block inside a copy splits it below threshold (by design, undocumented); markers in shell heredoc text are honoured; identical binary files matching a glob are read as code (a NUL-byte skip would avoid it); overlapping sub-pairs on a long copied table; min_tokens: 50.0 passes the schema and fails the command; bin/jpdev (no extension) is not read; CR-only line endings count as one line.

Verified: the full diff; the three test files once (29, 21, 24 OK) under a memory cap; the kit's own run (no pair, 21 files, 0.2 s); CRLF, Latin-1 and cross-language copies caught; renamed identifiers not caught as documented; nested and unbalanced markers reported; paths/exclude cannot escape the root; mutants M1, M5-M8 killed, M2 and M3 survived; both generated_docs.py copies byte-identical; changed_checks accepts the member; CI and evidence list the step; no machine-local path or personal name. merge: after fixes.

Disposition (the coordinator's)

#	Severity	Disposition
1	HIGH	Fixed in this PR by the same builder: per-hash place list, walk back past a rejected place; the three-file fixture as a test
2	MEDIUM	Fixed in this PR: duplication in every template rule that covers a declared language; the adopting.md sentence
3	MEDIUM	Fixed in this PR: the two fixtures that kill mutants M2 and M3
4	MEDIUM	Fixed in this PR: resolved path must stay under the resolved root
5	MEDIUM	Fixed in this PR: a byte limit per file, refused or skipped by name; documented
8, 10	LOW	Ride in the same pass (NUL-byte skip; multipleOf: 1 in the schema)
6, 7, 9, 11, 12	LOW	Ride: documented in README where a sentence suffices (6, 7, 12); 9 and 11 left as noted

The same reviewer re-reads the fixed head.

[kikashy]

Review re-read: 5dd5f44 (fix round)

Same reviewer as the first record, same clean-room clone, moved to 5dd5f44. Same-vendor review, not a cross-vendor claim. Python 3.12.11, every run under a memory cap and a timeout; probes in a scratch copy only, no tracked file changed.

#	Finding (first record)	Disposition at 5dd5f44
1	HIGH: only the latest place per token run, so a third reflowed copy hid a real pair	Fixed for the recorded case: two 13-line copies with an 8-line reflowed copy between them now exit 1 and name both. Both new tests fail when the fix is broken. Residual: the 16-place cap brings the miss back once 16 or more short copies of every run sit between the two real ones (documented bound; not a HIGH; follow-up filed). Cost holds: two 200k-token runs in 1.5 s.
2	MEDIUM: template changed rules did not select the step	Fixed: a change to a declared-language file or docs selects duplication; a change to the detector runs the whole list. Removing the step from either jpdev.json fails the rules test.
3	MEDIUM: two ignore-block mutants survived	Fixed: both breaks (an ignore block hiding every pair in its file; one shared barrier id) now fail the new tests.
4	MEDIUM: symlink check on the last path part only	Fixed: a path is read only when its resolved form equals the path; the parent-directory symlink probe reads only the in-root file; dropping the check fails the test.
5	MEDIUM: no size bound	Fixed per file: max_file_bytes (default 1 MB) with a not read: line, and an out-of-memory refusal with a named exit 2. Total memory across many files is still unbounded; acceptable with the refusal.
6	LOW: NUL bytes, multipleOf, README	Fixed, with one cosmetic exception: multipleOf: 1 does not refuse 50.0 (JSON Schema counts it as an integer); the command still refuses it by name.

Verified: 41 duplication tests and 24 changed-checks tests pass; the kit's own duplication step passes over 21 files. The round-1 probes were re-run and the code broken ten ways (eight in the detector, two in the mappings); each break failed a test.

No new HIGH. merge: yes (5dd5f44).

[github-actions]

🎉 Thank you for your contribution to Judgment Pack!

If you’d like another task, choose a path that interests you:

• Go runtime
• Judgment desk
• Gateway and provenance
• Python and evaluator experiments
• Specification and documentation
• Agent harness and adapters