Skip to content

jpdev v0.1.0: setup, check, mutants, record; the manifest, the public runner, the agent docs and the policy - #1

Merged
kikashy merged 3 commits into
mainfrom
scaffold
Oct 20, 2026
Merged

kikashy merged 3 commits into
mainfrom
scaffold

Conversation

@kikashy

@kikashy kikashy commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What it holds

  • bin/jpdev 0.1.0 with setup, check, mutants, record post and env; settings from ${XDG_CONFIG_HOME:-$HOME/.config}/jpdev/env, overridden by the environment (config/env.example documents each with its default).
  • lib/rsscap.sh (the memory-cap wrapper for node, unchanged in behaviour), lib/wrap.sh (the caps by toolchain: node under the RSS cap; Go under the machine lock with ulimit -v, GOFLAGS=-mod=mod, a per-clone GOCACHE and timeout; python under ulimit -v; shell none), lib/common.sh, lib/validate_manifest.py, lib/mutants_report.py.
  • templates/dev.json and templates/dev.schema.json (the manifest and its JSON Schema; jpdev check validates with python3, using the jsonschema package when it is installed and a structural check otherwise).
  • templates/check.sh: the public runner a repository vendors as scripts/dev/check.sh (python3 standard library, no caps; jpdev check delegates to it and adds the caps and lock through JPDEV_WRAP).
  • templates/AGENTS.md and templates/CLAUDE.md (one text, 31 lines; scripts/sync-agent-docs.sh generates the second and --check is a CI step), templates/ci-check.yml.
  • docs/policy.md, docs/adopting.md, docs/releases/v0.1.0.md, examples/judgment-pack-agent.dev.json, README.md.
  • tests/run.sh (80 checks) with tests/fixture/ (a manifest of shell echoes and one deliberate failure), and .github/workflows/ci.yml (shellcheck, bash -n, the agent-docs sync check, JSON parse, the machine-local-path grep, whitespace, the tests).

Checks

Kit tests: tests/run.sh 80 passed, 0 failed (check.sh and jpdev check on the fixture, the three caps and the lock, the manifest schema, mutants on a scratch copy, the record gate, env, setup on a local origin).

shellcheck 0.10.0 (the native binary), in the CI form from the repository root, no check excluded: shellcheck -x -P ".:SCRIPTDIR" bin/* lib/*.sh templates/*.sh scripts/*.sh tests/*.sh tests/fixture/*.sh: clean. bash -n on every script and git diff --check: clean. No machine-local path or personal name in tracked text.

The agent repository's manifest, jpdev setup then jpdev check --all on a copy of a clone of judgment-pack-agent at 61b1331 (the Desk checkout at d9251ec for the parity step):

== check summary (judgment-pack-agent.dev.json) ==
PASS     diff-check               0s
PASS     python-tests             30s
    Ran 44 tests in 30.214s
    OK
PASS     mutation-rows            0s
    163 rows checked, 0 problems; 1 retired
PASS     build                    2s
PASS     unit-tests               69s
    Test Files  48 passed (48)
    Tests  1923 passed | 27 skipped (1950)
PASS     certify-vercel           27s
    certify vercel: passed, 305 of the model engine's own checks among them
PASS     certify-codex            5s
    certify codex: passed, 63 of the agent engine's own checks among them
PASS     certify-deepagents       18s
    certify deepagents: passed, 329 of the model engine's own checks among them
PASS     gofmt                    0s
PASS     go-vet                   0s
PASS     go-generate              0s
PASS     go-test                  13s
    13 lines matched; last: ok  	github.com/Judgment-Pack/judgment-pack-agent/go/relay	0.276s
PASS     harness-check            11s
    170 legs passed, 0 failed
    38 agent checks passed, 0 failed (the package's stand-in only: no host's bridge was certified)
PASS     parity                   0s
    parity: packages/harness against the Desk at d9251ec
    parity: 26 moved files identical, 45 differ for the reasons named above; 68 package files are new
    parity: every difference is named
== 14 passed, 0 failed, 0 skipped, 0 not run ==

jpdev mutants --row "a source's description reaches the model unframed" on the same copy: caught, 1 caught, 0 not caught; the scratch copy removed afterwards.

Review round

Findings of the narrow review of 2182988, all fixed in edc51a8:

  • HIGH, the name gate failed open on an invalid regex: the pattern file is now compiled first (grep -E -f on empty input, status 2 refuses) and any grep status other than 0 or 1 refuses.
  • MEDIUM, a missing name-pattern file only logged: record post now refuses unless --no-name-check is given; an empty file, or one of comments only, is warned about.
  • MEDIUM, requires_env let CI go green on a skipped check: under CI a skip fails unless the step is optional: true (new schema field; the agent example marks parity optional); ci-check.yml shows the shallow checkout that sets the variable; docs/adopting.md says so.
  • MEDIUM, a shell step had no timeout under jpdev check: the shell branch of lib/wrap.sh runs under timeout -k 10, and a test runs the same timing-out step through both runners.
  • MEDIUM, no test covered the node or python branches: tests for the node RSS cap (stopped with 137 and reported as a failure), the python ulimit -v and timeout, and the gate cases above.
  • LOW, the venv name was hard-coded: jpdev_prepare_run exports the manifest's venv name and wrap.sh uses it.
  • LOW, --only with an unknown name exited 0: it exits 2 naming the step (--skip likewise).

Each new safeguard was broken in a scratch copy and the tests failed on it. Earlier, 2182988 fixed the CI lint failures (the source directives and two A && B || C tests).
The shellcheck line of the first version of this description named a build that could not follow source paths and was run with SC1091 excluded; it is replaced above.

Not in this PR

  • A jpdev release command for the prepare, tag and gate chain.
  • The per-repository adoptions (each repository vendors check.sh, writes its dev.json, adds AGENTS.md and CLAUDE.md and the CI workflow in its own pull request).

🤖 Generated with Claude Code

kikashy and others added 3 commits October 20, 2026 10:05
… runner, the agent docs and the policy

Adds the development kit: bin/jpdev with setup, check, mutants, record
and env; the node, Go and python caps and the machine lock; the dev.json
manifest and its JSON Schema; the vendored check.sh; AGENTS.md and
CLAUDE.md templates with a sync check; the CI step list; the operating
policy and the adoption guide; the agent repository's manifest as the
worked example; the kit's own tests and CI.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Source directives name lib/common.sh from the repository root and CI
runs shellcheck with -x -P ".:SCRIPTDIR" so they resolve; the two
"A && B || C" argument checks become if statements.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
- record post refuses an invalid name-pattern regex and any grep failure
  (it no longer fails open), and refuses a missing name-pattern file
  unless --no-name-check is given; an empty file is warned about.
- check.sh: a step skipped for an empty requires_env variable fails when
  CI is set unless it is marked optional; --only and --skip with an
  unknown name fail naming it; ci-check.yml shows how CI sets the
  variable for a parity step.
- wrap.sh: a shell step runs under timeout, as under check.sh; the
  python virtual environment is the manifest's venv name.
- tests: the node memory cap, the python ulimit and timeout, the venv
  name, the runner timeout agreement, the CI skip rules and the gate
  cases above.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
@kikashy

kikashy commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Narrow review record (same vendor, not a cross-vendor claim)

Reviewer: a clean-room Claude Sonnet agent, read-only, four probes and the kit tests once. Verdict on the reviewed head: not mergeable — head 2182988f6e07a61108a82b80bfc63dd5942e8bb1 (one HIGH). Since v0.1.0 was not yet tagged, the pre-release batch is the same builder's one round on this branch (the disposition names the fixing commit). Verbatim report follows, with scratch paths reduced to <scratch>.


Review of judgment-pack-devkit PR 1 at head 2182988. Read-only; nothing posted; no sub-agents; no pkill.

FINDINGS (most severe first)

  1. HIGH — bin/jpdev:212. The name gate fails open on an invalid regex. if [ -s "$patterns" ] && grep -nEif "$patterns" ... treats a grep error (status 2) as "no match". Probe: pattern file (unclosed + zork quux, body with "Zork Quux": grep prints "Unmatched ( or (", then "would post 2 lines". No pattern applied, so the name would be posted. Fix: capture grep's status and refuse on >=2; validate the pattern file first. Confirmed by running.

  2. MEDIUM — bin/jpdev:217-219. A missing JPDEV_NAME_PATTERNS file only logs "the name check did not run" and the post proceeds; an empty file passes with no message. docs/policy.md says jpdev record post "refuses to post otherwise". Fix: refuse unless an explicit --no-name-check; warn on an empty file. Confirmed by running.

  3. MEDIUM — templates/check.sh:24, templates/ci-check.yml:47-50, docs/adopting.md:17-18,71-72. requires_env lets a check be absent in CI. The skip is visible (SKIP row "needs JPDEV_DESK", a skipped count) but does not fail the run; ci-check.yml never sets the variable and adopting.md does not say CI sets it. Parity is therefore always skipped in CI and the run is green. Fix: make a SKIP a failure when CI is set (or add --require-env), or document that the repository's CI sets the variable. Confirmed by reading.

  4. MEDIUM — lib/wrap.sh:42-44 + templates/check.sh:31. Under jpdev check a shell step has no timeout: check.sh leaves the timeout to the wrapper when JPDEV_WRAP is set, and wrap's shell branch is exec "$@". Probe: step sleep 6, timeout 1: check.sh gives FAIL after 1 s ("stopped for time"); jpdev check gives PASS after 6 s. The two runners disagree. Fix: shell branch uses timeout -k 10. Confirmed by running.

  5. MEDIUM — tests/run.sh. No test covers the node or python branches of wrap.sh (rsscap.sh is tested directly; the python fixture step is only python3 --version). Routing a node step through ulimit -v, or dropping the python cap, passes all 55 tests. No test for an invalid or missing pattern file. Fix: add a node step under a small JPDEV_NODE_RSS_MB and a python ulimit -v echo to the fixture, plus the gate cases from findings 1-2.

  6. LOW — lib/wrap.sh:22. The venv path is hard-coded to .venv, while jpdev setup creates the manifest's venv name; any other name is created but never put on PATH. Fix: export the name from jpdev_prepare_run. Confirmed by reading.

  7. LOW — templates/check.sh:22. --only with a name not in the manifest runs nothing and exits 0 ("0 passed"); a typo reads as green. Fix: fail when a named check does not exist. Confirmed by reading.

  8. LOW — public check.sh --setup has no venv step. In the example manifest python-deps runs pip install into whatever python3 is on PATH. The public runner itself needs only bash, git and python3 stdlib. pyenv appears only in jpdev setup and jpdev env, so a contributor without pyenv is not blocked.

WHAT HOLDS

  • Q1: with JPDEV_NODE_RSS_MB=200 a node step allocating 600 MB was killed ("STOPPED for memory ... 642 MB", exit 137) and the run continued. A node step shows no ulimit -v of its own; the python step showed 4000000. Two parallel jpdev check runs on one go step took 3 s and 6 s (serialised). The go ulimit -v is covered by a smoke test.
  • Q2: refusal happens before gh is called (a fake gh was never invoked). With a valid pattern, a path and a name were both refused. The reduced body read <scratch>/x/out, home <home>/notes.
  • Q3: no /home/, /tmp/, user name or e-mail in tracked files apart from test@example.com in the tests. Commit authors and sign-offs are the noreply address. config/env.example carries defaults only.
  • Q4: check.sh and jpdev check outputs on the fixture with --all are identical after normalising timings (56 lines). Same order, same summary block.
  • Q5: AGENTS.md and CLAUDE.md are byte-identical, 31 lines, sync --check clean. All required points are present.
  • Q6: every rule listed is in docs/policy.md, with no names and no contradictions; the only conflict is finding 2.
  • Q7: jpdev --version prints jpdev 0.1.0; bash -n and git diff --check clean. shellcheck 0.11.0 ran via the npm package, with the downloaded native binary called directly. In the CI form -x -P ".:SCRIPTDIR" over the same file set it was clean, so the -e SC1091 workaround was not needed here.

RAN: tests/run.sh once (55 of 55 pass, under ulimit -v 8000000). Four probes: node cap, ulimit -v and the python cap; two parallel go runs; the record gate (path, name, missing, invalid and empty pattern file, reduced body); the runner diff and shell timeout. No lingering processes; no pkill. Scratch files are under the scratchpad, in review-devkit, probe and sc.

not mergeable — head 2182988


Disposition (maintainer's rule: HIGH fixed in-PR before merge; MEDIUM fixed in the one pre-release batch, which for an untagged first version is the same round; LOW rides)

# Severity Disposition
1 HIGH Fixed before the merge (commit edc51a8): the pattern file validated first, grep's status captured, anything but 0 or 1 refuses; a test with an invalid pattern file. The coordinator re-ran the reviewer's probe on the fixed head (an invalid pattern file, a matching pattern and a missing pattern file each refused before any gh call; a clean body reached gh).
2 MEDIUM Same round: a missing pattern file refuses unless --no-name-check; an empty file warns; tests.
3 MEDIUM Same round: a SKIP fails under CI unless the step is optional: true; ci-check.yml sets the variable for parity with a shallow Desk checkout; adopting.md says so.
4 MEDIUM Same round: the shell branch under timeout -k 10; the runner-diff test covers a timing-out step.
5 MEDIUM Same round: node and python branch tests; the gate cases.
6 LOW Rode: the venv name exported from the prepare step.
7 LOW Rode: --only with an unknown name fails.
8 LOW No change: the public runner needs only bash, git and the python3 standard library, as found.

Merged on green CI after the round; tagged v0.1.0 by the coordinator.

@kikashy
kikashy merged commit 5439353 into main Oct 20, 2026
2 checks passed
@kikashy
kikashy deleted the scaffold branch October 20, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant