kikashy · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026
diff --git a/.github/ci-evidence.json b/.github/ci-evidence.json
@@ -8,6 +8,8 @@
     "lib/ci_evidence.py",
     "templates/check.sh",
     "lib/generated_docs.py",
+    "lib/duplication.py",
+    "jpdev.json",
     "scripts/dev/generated_docs.py",
     "tests/generated_docs_test.py",
     "tests/run.sh",
@@ -60,6 +62,7 @@
             "local-paths",
             "agent-docs",
             "bash-n",
+            "duplication",
             "json",
             "shellcheck"
           ]

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -24,12 +24,13 @@ jobs:
         with:
           persist-credentials: false
       # Whitespace; no machine-local path in tracked text; CLAUDE.md is AGENTS.md and the root
-      # copy starts with the template; every script parses; the JSON documents parse; shellcheck
-      # (the program the runner image carries).
+      # copy starts with the template; every script parses; no block of code is duplicated over the
+      # thresholds of jpdev.json; the JSON documents parse; shellcheck (the program the runner image
+      # carries).
       - name: The lint steps of dev.json
         run: |
           set -o pipefail
-          scripts/dev/check.sh --all --only diff-check,local-paths,agent-docs,bash-n,json,shellcheck | tee "${{ runner.temp }}/checks.log"
+          scripts/dev/check.sh --all --only diff-check,local-paths,agent-docs,bash-n,duplication,json,shellcheck | tee "${{ runner.temp }}/checks.log"
 
       - name: Record exact CI inputs and coverage
         if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'

diff --git a/README.md b/README.md
@@ -183,6 +183,67 @@ the same name overrides it. `config/env.example` documents each setting with its
   the same check first and does not start the run on a refusal.
 - `jpdev env`: the effective settings and where each came from.
 
+## The duplication check
+
+Duplicated logic is where two copies drift. `lib/duplication.py` (python3 standard library) is an
+opt-in check-list step: a repository vendors it as `scripts/dev/duplication.py` and lists the step
+`{"name": "duplication", "run": "python3 scripts/dev/duplication.py", "toolchain": "shell",
+"summary": "^duplication:", "timeout": 300}`, which `templates/dev.json` carries. The step is off,
+and its summary line says so, until the repository's `jpdev.json` declares a `duplication` member
+with its thresholds (`templates/jpdev.schema.json` holds the shape):
+
+```json
+"duplication": {
+  "languages": ["go", "python", "typescript"],
+  "exclude": ["**/testdata/**"],
+  "min_tokens": 70,
+  "min_lines": 6
+}
+```
+
+- `languages` (required): any of `go`, `javascript`, `python`, `shell` and `typescript`; each names
+  its file extensions and comment syntax. The files read are the tracked, and the untracked but not
+  ignored, files of those extensions that match a `paths` glob (default `**`) and no `exclude` glob.
+  Globs are repository-relative: `*` and `?` stay within one segment and a `**` segment spans any
+  number of segments. A symbolic link is not followed, whether it is the file or a directory
+  above it. A file larger than `max_file_bytes` (1024 to 104857600, default 1048576) is not read,
+  nor is a file with a NUL byte in its first 8192 bytes (binary); each is named in a `not read:`
+  line, and the check does not fail on it. A large generated file belongs in `exclude`.
+- `min_tokens` (10 to 10000) and `min_lines` (1 to 10000), both required: a pair is reported when
+  two places share at least `min_tokens` tokens and each place spans at least `min_lines` lines.
+  Whitespace, layout and comments are not tokens; identifiers, keywords, numbers, string literals
+  and each punctuation character are. A copy laid out or commented differently is still a copy; a
+  copy whose names were changed is not. A line ends at a line feed: a file with carriage-return
+  line endings alone counts as one line.
+- An intentional copy is marked with an ignore block: the code between a comment that holds
+  `duplication:ignore-start` and a comment that holds `duplication:ignore-end` is not read, in any
+  comment syntax of the language (`//`, `/* */`, `#`). The text in a string is not a marker; the
+  text of a shell here-document is read as code, so a marker after a `#` there counts. An ignore
+  block inside a copy splits it in two, and each part is held to the thresholds on its own. A start
+  that is never closed, an end with no start, and a start inside an open block are reported.
+
+The report names each pair with both places and the shared token count, one line each. Each run of
+`min_tokens` tokens is compared with the places it appeared before (the 16 most recent), nearest
+first, until one makes a reported pair: three copies of a block make two pairs, and a copy is still
+named when a nearer place of the same tokens spans too few lines. Two places in one file never
+overlap, and a run that repeats a pattern shorter than `min_tokens` in one file (a table of one
+short row) is not reported as copies of itself.
+
+```
+duplication: 1 duplicate pair (at least 70 tokens over at least 6 lines, 214 files of go, python, typescript)
+  internal/audit/receipt.go:18-50 and internal/export/receipt.go:40-72: 131 tokens
+```
+
+The kit runs the check on itself with `python3 lib/duplication.py`, over its Python and shell
+files at 100 tokens and 10 lines, and leaves out `scripts/dev/**`: those files are the vendored
+copies of `templates/check.sh`, `lib/generated_docs.py` and `lib/handover_check.sh`, kept the
+same on purpose.
+
+The exit status is 0 when nothing is found or the check is off, 1 when a pair or an ignore marker
+problem is found, and 2 when the declaration is refused or the files in scope do not fit in
+memory. A refusal names the member, for example `duplication: refused: jpdev.json
+duplication.min_tokens: an integer from 10 to 10000 is required (got 5)`.
+
 ## The caps
 
 | Toolchain | Cap |
@@ -203,12 +264,12 @@ Python steps, and for other steps that invoke `python3`, name the interpreter us
 ## What is here
 
 - `bin/jpdev`, `lib/`: the command, the caps, the manifest check, the brief check, the changed-path
-  check selection.
+  check selection, the duplication check.
 - `config/env.example`, `config/codex-words.txt`: the settings by example, and the base word list of
   `jpdev codex check-brief`.
 - `templates/dev.json`, `templates/dev.schema.json`: the manifest by example and its JSON Schema.
-- `templates/jpdev.json`, `templates/jpdev.schema.json`: the generated-document declaration and the
-  changed-path mapping by example, and their JSON Schema.
+- `templates/jpdev.json`, `templates/jpdev.schema.json`: the generated-document declaration, the
+  changed-path mapping and the duplication thresholds by example, and their JSON Schema.
 - `templates/check.sh` and `lib/handover_check.sh`: the public checks a repository vendors as
   `scripts/dev/check.sh` and `scripts/dev/handover_check.sh`.
 - `templates/codex-brief.md`, `templates/codex-review-brief.md`: the build and review brief
@@ -221,8 +282,8 @@ Python steps, and for other steps that invoke `python3`, name the interpreter us
 - `tests/run.sh`: the kit's own tests (`tests/fixture/` holds a manifest of shell echoes and one
   deliberate failure, and the manifest and mapping of the `jpdev check --changed` fixture clones).
 - `dev.json`, `jpdev.json`, `scripts/dev/check.sh`, `scripts/dev/handover_check.sh`, `AGENTS.md`,
-  `CLAUDE.md`: the kit on itself, the list of checks CI runs, its changed-path mapping, and the
-  handover guard. `CONTRIBUTING.md`, `SECURITY.md`, `docs/releasing.md`: contributing, reporting
+  `CLAUDE.md`: the kit on itself, the list of checks CI runs, its changed-path mapping and
+  duplication thresholds, and the handover guard. `CONTRIBUTING.md`, `SECURITY.md`, `docs/releasing.md`: contributing, reporting
   and releasing.
 
 Apache-2.0; see `LICENSE`.
diff --git a/dev.json b/dev.json
@@ -6,6 +6,7 @@
     {"name": "local-paths", "run": "if git grep -nE '(^|[^A-Za-z0-9_])/(home|tmp)/[A-Za-z0-9_.-]+'; then echo 'a machine-local path is in tracked text'; exit 1; fi", "toolchain": "shell"},
     {"name": "agent-docs", "run": "scripts/sync-agent-docs.sh --check && scripts/sync-agent-docs.sh --check . && head -c \"$(wc -c < templates/AGENTS.md)\" AGENTS.md | cmp - templates/AGENTS.md", "toolchain": "shell", "summary": "^sync-agent-docs:"},
     {"name": "bash-n", "run": "set -eu; for file in bin/* lib/*.sh templates/*.sh scripts/*.sh scripts/dev/*.sh tests/*.sh tests/fixture/*.sh; do bash -n \"$file\"; done; echo 'bash -n: every script parses'", "toolchain": "shell", "summary": "^bash -n:"},
+    {"name": "duplication", "run": "python3 lib/duplication.py", "toolchain": "shell", "summary": "^duplication:", "timeout": 300},
     {"name": "json", "run": "python3 -I -c 'import glob, json; files = sorted(glob.glob(\"*.json\") + glob.glob(\"templates/*.json\") + glob.glob(\"examples/*.json\") + glob.glob(\"tests/fixture/*.json\")); [json.load(open(f, encoding=\"utf-8\")) for f in files]; print(\"parsed %d JSON documents\" % len(files))'", "toolchain": "shell", "summary": "^parsed "},
     {"name": "shellcheck", "run": "shellcheck -x -P \".:SCRIPTDIR\" bin/* lib/*.sh templates/*.sh scripts/*.sh scripts/dev/*.sh tests/*.sh tests/fixture/*.sh && echo 'shellcheck: clean'", "toolchain": "shell", "summary": "^shellcheck:"},
     {"name": "tests", "run": "tests/run.sh", "toolchain": "shell", "summary": "^(Failed cases:|[0-9]+ passed, [0-9]+ failed)", "timeout": 900}

diff --git a/docs/adopting.md b/docs/adopting.md
@@ -167,6 +167,19 @@ JPDEV_DESK=path/to/desk-checkout jpdev check --all --manifest path/to/devkit/exa
 
 Copy the example to the repository as `dev.json` when it adopts the kit.
 
+## Duplication check
+
+The `duplication` step is opt-in. Vendor `lib/duplication.py` as `scripts/dev/duplication.py`, keep
+the `duplication` step of `templates/dev.json` in the check list, and declare the repository's
+languages and its token and line thresholds in the `duplication` member of `jpdev.json`
+(`generators` may be left out when `duplication` is present). Until the member is present the step
+passes and its summary line says `duplication: off`. Choose thresholds the current tree passes, or
+remove or mark the copies the first run names. When `jpdev.json` also maps changed paths
+(`changed`), list `duplication` in every rule whose globs can match a file of a declared language,
+or `jpdev check --changed` does not run the step for those changes.
+[The duplication check](../README.md#the-duplication-check) describes the member, the ignore block
+and the report.
+
 ## Generated reference documents
 
 Keep `dev.json` as the checklist manifest. Repositories with checked-in generated documents add

diff --git a/docs/generated-docs.md b/docs/generated-docs.md
@@ -30,7 +30,10 @@ segment. Declare every output file individually; directories/globs are unsupport
 `timeout` is an integer1–3600 seconds (default120). The optional `dependencies` list contains
 explicit existing `node_modules` directories only, such as `node_modules` or
 `packages/example/node_modules`. Omit it when the generator needs no installed npm dependencies.
-The example command/output must be adjusted to the repository's actual generator.
+The example command/output must be adjusted to the repository's actual generator. The optional
+`duplication` member of the same file configures
+[the duplication check](../README.md#the-duplication-check); this checker accepts that member and
+does not read it.
 
 Add the ordinary `dev.json` check (and keep the generator's useful existing coverage):
 

diff --git a/docs/policy.md b/docs/policy.md
@@ -10,6 +10,9 @@ are the ones a brief does not repeat.
 - **CI owns everything deterministic.** Formatting, builds, unit tests, schema checks,
   certification suites, whitespace and path checks run in CI on every pull request. A session
   does not re-run them to reassure itself and does not watch CI run.
+- **Duplication is where two copies drift:** a repository opts in to the `duplication` check with
+  its own token and line thresholds in `jpdev.json`, and marks an intentional copy with an explicit
+  ignore block.
 - **Define and implement a dependency-ordered coherent batch first.** Name the related issues,
   dependency order and acceptance criteria; write acceptance and regression fixtures by hand
   alongside code. During development, run useful focused checks of changed paths, affected

diff --git a/jpdev.json b/jpdev.json
@@ -1,9 +1,15 @@
 {
   "version": 1,
+  "duplication": {
+    "languages": ["python", "shell"],
+    "exclude": ["scripts/dev/**"],
+    "min_tokens": 100,
+    "min_lines": 10
+  },
   "changed": [
     {
       "paths": ["docs/**", "*.md", "LICENSE", "NOTICE", ".github/PULL_REQUEST_TEMPLATE.md"],
-      "checks": ["diff-check", "local-paths"]
+      "checks": ["diff-check", "local-paths", "duplication"]
     },
     {
       "paths": ["AGENTS.md", "CLAUDE.md", "templates/AGENTS.md", "templates/CLAUDE.md"],

diff --git a/lib/changed_checks.py b/lib/changed_checks.py
@@ -31,7 +31,8 @@
 # Every diff names paths from the top, counts both sides of a rename, runs no external diff, and
 # reports a submodule whatever its ignore setting in .gitmodules or the git configuration.
 DIFF = ('diff', '--name-only', '-z', '--no-renames', '--no-relative', '--no-ext-diff', '--ignore-submodules=none')
-MEMBERS = {'version', 'generators', 'changed'}
+# `generators` is read by lib/generated_docs.py and `duplication` by lib/duplication.py; this reads `changed`.
+MEMBERS = {'version', 'generators', 'changed', 'duplication'}
 RULE_MEMBERS = {'paths', 'checks'}
 UNSUPPORTED = '\\[]{}'
 SHOWN = 20