Repository navigation
Re-vendor the devkit scripts at v0.10.0 - #272
Merged
Merged
Conversation
scripts/dev/check.sh, scripts/dev/handover_check.sh and scripts/dev/changed_checks.py from judgment-pack-devkit v0.10.0 (the release on line 2; no behaviour change in these three). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Contributor
|
🎉 Thank you for your contribution to Judgment Pack! If you’d like another task, choose a path that interests you: |
kikashy
added a commit
that referenced
this pull request
Oct 26, 2026
… tag and Policy validation fresh, fail closed (#257) (#273) Closes #257. Release orchestration only: no normative text, schema, example, conformance case or conformance semantics changes, and nothing a `policy-v` tag validates (its bundle paths and corpus) is touched. ## Summary A `vMAJOR.MINOR.PATCH` release no longer runs the repository suite again. A new `CI evidence at the tag` job runs the development kit's release evidence gate (kit v0.10.0 `lib/release_evidence.py`, vendored as `scripts/dev/release_evidence.py`, line 2 naming the release, byte-identical after it) against the tagged commit, configured by `.github/ci-evidence.json` (schema 2). It reuses the evidence main's `repository-conformance` job recorded at that exact commit, or refuses and fails the release before anything is built. `conformance.yml` records that evidence on a push to `main` only. The release job keeps every release-specific step, unchanged and in the same order; only its repository-suite step is skipped, and only when the gate said `reuse`. | File | Change | | --- | --- | | `.github/ci-evidence.json` | workflow `Repository conformance` by path and `name:`, event `push` on `main` (the repository has no dispatch on it), job `repository-conformance` on `ubuntu-latest`, checks `all` with no permitted skips, setup steps recorded, inputs `requirements-dev.txt` and `scripts/dev/check.sh` (beside `dev.json`, the configuration, the gate and the workflow, always recorded), one Linux profile compared with the release runner | | `.github/workflows/conformance.yml` | the setup and check runs are also written to logs (`pipefail`); on a push to `main` only: `release_evidence.py capture`, then `ci-evidence-conformance-repository-conformance-<attempt>` uploaded with a pinned `actions/upload-artifact` | | `.github/workflows/release.yml` | new `evidence` job (`contents: read`, `actions: read`); `release` needs it; the suite step runs unless the evidence job hands over `true`, which a step after the gate does only when the decision record says reuse (fix round) | | `scripts/dev/release_evidence.py` | the vendored gate | | `tests/test_release_evidence.py` | the configuration held to the workflows, `dev.json` and `requirements-dev.txt`; the route held to the gate's tag grammar; the release steps and their order; the evidence job's setup held to main CI's; the vendored copy held to the kit release | | `jpdev.json`, `AGENTS.md`/`CLAUDE.md`, `web/DEPLOYMENT.md` | the new files mapped; one sentence; a "Release evidence" section (the contract, the decision record, the fresh route, what stays fresh) | ## Inventory | Check | Main CI (`conformance.yml`, push to `main`) | Release before | Release now, `vMAJOR.MINOR.PATCH` | Release now, other tags (`v0.3.0-rc.1`, `policy-v…`) | | --- | --- | --- | --- | --- | | `repository-tests`: `tests/` | run (`check.sh --all`, in `.venv`) | run again (`python -m unittest discover -s tests -v`) | **reused** from main's evidence | run again, unchanged | | `local-paths`, `agent-docs`, `diff-check`, `policy-prototype`, `policy-contract`, `policy-official-draft` | run | not run | required `PASS` in the evidence (not run) | not run | | setup `python-deps`: `requirements-dev.txt` with the `format-nongpl` extras | run (`check.sh --setup`, `.venv`) | installed by the release | required `PASS`; the installed set compared; the format checkers pinned (below) | installed, unchanged | | milestone and tagged-checkout admission (`tools/release_milestone.py`) | not run | fresh | fresh | fresh | | exact Policy format and agreement corpus | as `policy-official-draft` | fresh for `policy-v` tags | not a Policy tag | fresh for `policy-v` tags, unchanged | | bundle build: tag grammar and commit, `HEAD`, clean tree, artifact pins, notes path, Policy tag history (`git tag --list policy-v*`) | not run | fresh | fresh | fresh | | draft prerelease (`--draft --prerelease --verify-tag`, notes from `releases/<tag>.md`), bundle attestation | not run | fresh | fresh | fresh | **Identical, so reused: the whole repository suite.** Shown three ways on main's tree (2639de8): - The suite (436 tests) ran on a branch checkout without the release tag, with main-CI context variables, and on a detached checkout of a new annotated `v0.3.0` tag at the same commit, with release context variables (`GITHUB_REF=refs/tags/v0.3.0`, `GITHUB_REF_TYPE=tag`, workflow `Release`): the same result for every test. - A logging `git` first on `PATH`: 25 git calls in the checkout, the same in both contexts, every one a read of `HEAD`, its tree or its attributes (`archive`, `ls-tree`, `check-attr`, `read-tree HEAD`/`add -A`/`write-tree` into a temporary index). No `tag`, `describe`, `for-each-ref`, `ls-remote`, `log`, branch or remote read; the other git calls run in fixture repositories under temporary directories. Their count differed by one round of `git show` in one fixture: `recover_core_rc.py` checks archive members in set order and a mutation test stops at the first difference, so the count follows the hash seed, not the context. - No test reads `GITHUB_*` without setting it, reaches the network, or reads the clock (`tests/*.py`); the milestone, tag and remote reads of the release tools under test go to stand-in `gh`/`git` commands or patched functions. The intentional differences that remain are compared instead of assumed: the runner image (`ImageOS`, `ImageVersion`, OS release digest), Python, Node and git versions, and the installed requirements. `requirements-dev.txt` pins only its direct requirements, so the resolved set is recorded (`pip freeze --all` of the `.venv` the checks ran in) and must equal what the release runner resolves when it sets up the same way (`dev.json` toolchains with the same action pins, `check.sh --setup`). The kit's rule that an unpinned input is not reusable is met by that comparison: a dependency released between main's run and the tag refuses (`runtime-identity`) and the fresh route makes new evidence. **Fresh, and why.** Everything that reads the tag, the milestone, the Policy tag history or makes the artifacts stays in the release job: the gate decides none of it and caches nothing. Tags outside the gate's grammar reuse nothing: kit v0.10.0's gate admits only `vMAJOR.MINOR.PATCH` (`tag-syntax` otherwise), and this repository's release candidates and `policy-v` tags carry a suffix, so the route step sends them to the whole suite instead of a certain refusal. The route is the gate's own grammar (a test holds the two equal), and the fresh route runs strictly more. The exact Policy corpus validation therefore stays fresh for every `policy-v` tag; no equivalence is claimed. **The extras are part of the evidence's identity.** A run without `jsonschema[format-nongpl]`'s extras passes while checking less. The evidence proves they were installed three ways: the setup log's `PASS python-deps` (the install of the exact `requirements-dev.txt`, whose object id is a recorded input); the installed set (`pip freeze --all`); and `jsonschema-formats`, pinned to `jsonschema==4.23.0 formats: color date date-time duration email hostname idn-email idn-hostname ipv4 ipv6 json-pointer regex relative-json-pointer time uri uri-reference uri-template uuid`. Measured locally: a clean venv from `requirements-dev.txt` prints exactly that; `jsonschema==4.23.0` alone prints `date email idn-email ipv4 ipv6 regex time uuid`, which refuses. A test holds the pin's version to `requirements-dev.txt` and requires `uri` and `date-time` in it. ## The contract (issue #257's evidence gate) | Item | Here | | --- | --- | | exact commit, repository, trusted event/ref, workflow identity, job coverage | the gate: run, head repository, `head_sha`, workflow id + path + `name:`, push on the default branch with no pull request association, the attempt's jobs by display name on hosted `ubuntu-latest`, artifacts bound to the run; capture refuses unless `GITHUB_WORKFLOW_REF`/`GITHUB_WORKFLOW_SHA` are `conformance.yml` at the commit. A PR head, synthetic merge, another workflow, a status name or a tree-equivalent commit is refused | | deterministic selection, no older green past a newer failed/cancelled/running attempt; pagination; reruns | the gate's selection (every page; this repository's trusted-branch runs scanned; newest latest-attempt start selected whatever its result; re-read after collection and once more at the end) | | explicit skips | no permitted skip: every `dev.json` check and setup step must be `PASS` | | the log | the decision record in the job log and as `release-evidence-<attempt>`, kept on refusal too (`!cancelled()`, `if-no-files-found: error`), with run id/attempt, inputs, job results, runtime and coverage | | mutable external state | milestone, tag, notes, bundles, prerelease and attestation stay fresh in the release job; the gate checks the tag before and after | | refusal classes tested | in the kit (`tests/release_evidence_test.py`, 85 tests, kit PR #52). Here: the configuration and the workflows, below | | fresh route | documented in `web/DEPLOYMENT.md`: re-run all jobs of main's run at the commit, then re-run the release run's failed jobs once; a second refusal: fix on `main`, release from a new commit. Nothing re-runs automatically, and no approval gate is added or removed | | branch protection | `repository-conformance` keeps its name and runs on every pull request and push as before | ## Live probe (read-only API, the vendored gate's own code) Every probe runs the vendored gate's own `decide()` over this repository's live API (GET only). Substituted in each: the release checkout context (this branch's configuration read at the probed commit, which does not hold it), the tag object (no `vMAJOR.MINOR.PATCH` tag names these commits) and the runner identity (the probe is not a hosted runner). | Probe | Commit and run | Decision | | --- | --- | --- | | main, as it is | `2639de8` (main's tip), `Repository conformance` run 40396213185, push, success, 0.9 h old | `refuse: artifact-coverage`: 0 artifacts named `ci-evidence-conformance-repository-conformance-1`, because capture lands with this pull request. Everything before it passed live: repository and default branch, the workflow resolved by path, name and state, selection (1 run at the commit, 1 applicable, none newer), the run read again (completed, success, age), and the attempt's jobs (`repository-conformance` success on `ubuntu-latest`, hosted runner group) | | main, evidence substituted | the same run, plus an evidence artifact built with the gate's own `evidence()` and `required_checks()` (every `dev.json` check and the setup step `PASS`), bound to the live run, attempt and job interval | `reuse` (run 40396213185, attempt 1) | | a pull request run | `a3cdb91`, the head of #272, run 40395750281 (`pull_request`, success) | `refuse: run-missing` (1 run at the commit, 0 applicable) | | this pull request | `c0a7aa9`, its run 40405251587 (`pull_request`; probed while running and again after it completed with success) | `refuse: run-missing` both times (1 run at the commit, 0 applicable) | Capture, dry run: on this branch's head with the GitHub context variables set by hand and the real `check.sh` logs of the run below, `capture` exits 0 and records the 6 input identities, `PASS` for the 7 checks and `setup:python-deps`, the 20-line `pip freeze --all` of `.venv` and the pinned format line. It is not evidence (the gate never reads a local file); it shows the summary format, the tool commands and a clean tracked tree after the checks. ## Mutants Each in a scratch copy of the checkout; the new test file fails at the named assertion, and passes unmutated. | # | Safeguard broken | Failing test | | --- | --- | --- | | M01 | configuration job name drifts from the workflow | `test_every_job_trigger_and_capture_exists_in_its_workflow`: `'repository-conformance' != 'conformance'` | | M02 | an input path that does not exist | `test_every_input_is_a_tracked_file` | | M03 | capture names another job key | `test_every_job_trigger_and_capture_exists_in_its_workflow` | | M04 | capture runs outside a push to `main` | same: the `if:` differs | | M05 | the push run checks a `--changed` selection | same: the whole-list log line is missing | | M06 | the vendored gate's body changed (`MAX_AGE_HOURS = 72`) | `test_vendored_copy_is_the_kit_library_after_line_two`: digest differs | | M07 | line 2 names another kit release | same | | M08 | suite skipped unless the route is `fresh` (a missing route would skip it) | `test_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_release` | | M09 | the release job no longer needs the evidence job | same: `None != 'evidence'` | | M10 | the route admits suffixed tags | `test_the_route_admits_exactly_the_gates_tag_grammar` | | M11 | the evidence job's token widened to `contents: write` | `test_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_release` | | M12 | release steps reordered (bundles before the Policy corpus) | `test_release_steps_keep_their_order_and_release_semantics` | | M13 | the formats pin without `uri` | `test_installed_requirements_are_recorded_from_the_checks_environment` | | M14 | the evidence job sets up another Node action | `test_the_evidence_job_sets_up_its_runner_as_main_ci_does` | | M15 | the decision record not kept on a refusal | `test_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_release` | ## Duration (measured; nothing else claimed) | Run | Repository suite step | Job | | --- | --- | --- | | `policy-v1.0.0-draft.1`, release run 39299066788 | 49 s | 67 s | | `v0.3.0-rc.1`, release run 40192247704 | 78 s (failed there) | 87 s | | main CI at 2639de8, run 40396213185 (`repository-conformance`, the whole list) | | 106 s | After: not measured yet. On reuse the suite step is skipped, and a separate job runs first (runner start, checkout, Python and Node setup, `check.sh --setup`, the gate; the kit's own gate step took about 9 s and its job 14–15 s at kit v0.9.0). Whether the release gets faster is measured at `v0.3.0`, not promised. Tags on the fresh route now wait for a one-step job before the release job. Main CI adds the capture and upload steps on a push to `main`. ## Not verifiable before merge - `capture` runs only on a push to `main`, so this pull request's CI does not exercise it; the first main push after merge does, and a capture refusal fails that run visibly. A local dry run of `capture` on this branch's commit, with the GitHub context variables set by hand and the real `check.sh` logs, is described at the end of the live probe section above. - `verify` is first live at the `v0.3.0` release. ## Checks Head `c0a7aa9` (no uncommitted diff). `jpdev check --changed` against `origin/main` (merge base `2639de8`): `jpdev.json` changed, so the whole list. ```text == check summary (dev.json) == PASS local-paths 0s PASS agent-docs 0s PASS diff-check 0s PASS repository-tests 113s Ran 445 tests in 112.695s OK PASS policy-prototype 1s PASS policy-contract 3s PASS policy-official-draft 3s == 7 passed, 0 failed, 0 skipped, 0 not run == ``` `actionlint` (with shellcheck) passes on both workflows. Under an outer `ulimit -v 4000000` five `test_profiles` cases fail only because Node cannot reserve WebAssembly memory (identically on both inventory contexts); `jpdev check` caps Node by resident memory instead and they pass. ## Fix round For the [review record](#273 (comment)) at `c0a7aa9` (0 HIGH, 2 MEDIUM, 3 LOW). One commit: `31cda9c`. | Finding | Fix | Test | Safeguards removed in scratch copies | | --- | --- | --- | --- | | MEDIUM 1: no test held that a refusal stops the release (no job-level `if:` on `release`; the gate's exit status reaches its step) | tests hold both. Defence in depth: the release job no longer reads the route written before the gate; it reads `reuse`, which a new step after the gate (`Hand the reuse to the release job`) writes only when the decision record it reads says `reuse` with no refusal | no job has a job-level `if:`; no `continue-on-error` anywhere; the gate step's `run:` is exactly the command; the evidence job's step order; the hand-over's own script run over reuse, refuse, reuse-with-refusal, no-decision, non-object and missing records (only reuse hands over `true`) | R-5 `if: always()` on `release`, R-6 `\|\| true` on the gate, F-1 `set +e`, F-2 `continue-on-error` on the gate step, F-3 hand-over without the record check, F-4 without the refusal check, F-5 hand-over before the gate, F-6 suite keyed to the route again, F-7 the job output taken from the route: all caught | | MEDIUM 2: the route test compared the grammar text, not the step | the route step's own script runs with `bash --noprofile --norc -eo pipefail` under `LC_ALL=C` over 16 tags; `route=evidence` exactly where the gate's `TAG.fullmatch` admits the tag (`v0.3.0`, `v1.10.0`, `v10.0.0`); everything else goes to the whole suite (`-rc.1`, `-rc.10`, `-draft`, `-0`, `+build`, `policy-v1.0.0`, `policy-v1.0.0-draft.1`, `v1.2`, `v01.0.0`, `v0.3.0.1`, a trailing space, Arabic-Indic digits, the demo tag) | `test_the_route_step_sends_exactly_the_gates_tags_to_the_gate` | R-2 tag matched without its suffix, R-3 outcomes swapped: caught | | LOW 3: prerelease consequence undocumented | `web/DEPLOYMENT.md`: the kit's prerelease grammar (kit v0.11.0) is opt-in; until this repository opts in, release candidates keep the whole suite, and opting in widens the route step and its test in the same change; `policy-v` is outside every kit grammar | | | | LOW 4: a failed-jobs re-run keeps a passed evidence job's decision | documented: re-run all jobs of the release run when main's run at the commit has a newer attempt or the decision is nearly 48 hours old; plus waiting out a hosted-image rollout (the review's note) | | | | LOW 5: stale probe row and "is below" | edited above | | | All 15 earlier mutants were run again on the fix (M08 rewritten for the `reuse` output) and are caught; 26 of 26 in all, and the unmutated tests pass (11 tests). `actionlint` with shellcheck passes. Candidate `31cda9c`, `jpdev check --changed` (whole list): ```text == check summary (dev.json) == PASS local-paths 0s PASS agent-docs 0s PASS diff-check 0s PASS repository-tests 112s Ran 447 tests in 111.529s OK PASS policy-prototype 1s Ran 38 tests in 0.902s OK PASS policy-contract 2s Ran 13 tests in 2.345s OK PASS policy-official-draft 2s Ran 16 tests in 2.294s OK == 7 passed, 0 failed, 0 skipped, 0 not run == ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/dev/check.sh,scripts/dev/handover_check.shandscripts/dev/changed_checks.pycopied whole from judgment-pack-devkit v0.10.0 (the release on line 2). The kit release adds the hardened release evidence gate (release_evidence.py,templates/ci-evidence.json), which this repository adopts in its own row; these three scripts change only their line 2. Tooling only.Diff: 3 files changed, 3 insertions(+), 3 deletions(-)
🤖 Generated with Claude Code