Repository navigation
Release: reuse the identical repository test evidence from main, keep tag and Policy validation fresh, fail closed (#257) - #273
Conversation
A vMAJOR.MINOR.PATCH release no longer runs the repository suite again. A new `CI evidence at the tag` job runs the development kit's release evidence gate (lib/release_evidence.py of kit v0.10.0, vendored as scripts/dev/release_evidence.py, byte for byte after line 2) against the tagged commit, configured by .github/ci-evidence.json: the newest push-to-main `repository-conformance` run at that exact commit, every setup step and check PASS, the commit's input identities, and the runner image, Python, Node, git and installed requirements equal to what the release runner resolves, with the jsonschema format checkers pinned so a run without the format-nongpl extras refuses. A refusal fails the release before anything is built; the fresh route is the maintainer's (web/DEPLOYMENT.md, "Release evidence"). conformance.yml records that evidence on a push to main only (capture and a pinned upload). A tag outside the gate's grammar (a release candidate, a policy-v tag) reuses nothing and runs the whole suite as before. The milestone and tagged-checkout admission, the exact Policy corpus for policy-v tags, the bundle build, notes, draft prerelease and attestation stay fresh, unchanged and in the same order. tests/test_release_evidence.py holds the configuration to both workflows, dev.json and requirements-dev.txt, the route to the gate's tag grammar, and the vendored copy to the kit release. Release orchestration only: no normative text, schema or conformance semantics change. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review record: spec PR #273 (issue #257), head c0a7aa9, base 2639de8The release's reuse of main's evidence is sound as written: the suite is context-independent, RC and Dispositions
Notes (no finding):
Probes (read-only API; scratch clones; caps on every run)
Mutants (test_release_evidence.py; unmutated OK)
Verified unchanged
merge: yes (no HIGH). Disposition per the kit policy: both MEDIUMs and the cheap LOWs are fixed in this PR in one round by the builder and confirmed by this reviewer. |
#257) Review round on #273. The release job takes the reuse from a step after the gate that reads the decision record and hands over `true` only for a reuse, so a refused gate cannot skip the suite even if its exit status were lost. Tests now hold that no job carries a job-level `if:` or `continue-on-error`, that the gate step runs the bare command, the evidence job's step order, the hand-over over reuse, refuse and malformed records, and the route step's own script run with bash over sample tags against the gate's grammar. web/DEPLOYMENT.md: a failed-jobs re-run keeps an earlier passed decision (re-run all jobs for a new one), waiting out a hosted-image rollout, and the kit's opt-in prerelease grammar. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirmation: 31cda9cFix round: one commit on c0a7aa9 (release.yml, test_release_evidence.py, web/DEPLOYMENT.md; +122/−14). Same-vendor review, not a cross-vendor claim.
The reuse flag cannot be satisfied by a stale or forged record. It is read in the same job from that job's Mutants at 31cda9c: 10 of 10 caught. These are my four earlier survivors (R-2, R-3, R-5, R-6) plus six new ones aimed at the flag:
Other checks: test_release_evidence 11/11 OK. Unmutated, test_release_recovery, test_release_milestone, test_release and test_policy_release pass (79 tests). actionlint is clean on both workflows. No local paths or names appear in the diff or the body. The PR's own CI run at 31cda9c was still in progress when I checked. No new HIGH or MEDIUM. merge: yes (31cda9c), once its |
scripts/dev/release_evidence.py from judgment-pack-devkit v0.11.0 (the tag grammar with SemVer prereleases, dispatch validation runs); tag.prerelease is enabled because this repository publishes prerelease tags. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
…ease in the copy test check.sh, handover_check.sh and changed_checks.py from judgment-pack-devkit v0.11.0 (line 2 only); the vendored-copy test records the v0.11.0 gate file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Confirmation: 74a0b77One commit on 31cda9c. It re-vendors the gate at kit v0.11.0 and sets
What I verified:
Note: widening the route for release candidates is a separate, later decision. It would change the route step and its test in the same change (DEPLOYMENT.md says so). Until then the route deliberately admits a subset of the configured grammar, which is the safe direction. merge: no (74a0b77). It becomes yes after the two test constants are updated and Confirmation: 9d8edcdOne commit on 74a0b77, touching four files with one line changed in each:
Same-vendor review, not a cross-vendor claim.
merge: yes (9d8edcd), once its |
|
🎉 Thank you for your contribution to Judgment Pack! If you’d like another task, choose a path that interests you: |
Closes #257. Release orchestration only: no normative text, schema, example, conformance case or conformance semantics changes, and nothing a
policy-vtag validates (its bundle paths and corpus) is touched.Summary
A
vMAJOR.MINOR.PATCHrelease no longer runs the repository suite again. A newCI evidence at the tagjob runs the development kit's release evidence gate (kit v0.10.0lib/release_evidence.py, vendored asscripts/dev/release_evidence.py, line 2 naming the release, byte-identical after it) against the tagged commit, configured by.github/ci-evidence.json(schema 2). It reuses the evidence main'srepository-conformancejob recorded at that exact commit, or refuses and fails the release before anything is built.conformance.ymlrecords that evidence on a push tomainonly. The release job keeps every release-specific step, unchanged and in the same order; only its repository-suite step is skipped, and only when the gate saidreuse..github/ci-evidence.jsonRepository conformanceby path andname:, eventpushonmain(the repository has no dispatch on it), jobrepository-conformanceonubuntu-latest, checksallwith no permitted skips, setup steps recorded, inputsrequirements-dev.txtandscripts/dev/check.sh(besidedev.json, the configuration, the gate and the workflow, always recorded), one Linux profile compared with the release runner.github/workflows/conformance.ymlpipefail); on a push tomainonly:release_evidence.py capture, thenci-evidence-conformance-repository-conformance-<attempt>uploaded with a pinnedactions/upload-artifact.github/workflows/release.ymlevidencejob (contents: read,actions: read);releaseneeds it; the suite step runs unless the evidence job hands overtrue, which a step after the gate does only when the decision record says reuse (fix round)scripts/dev/release_evidence.pytests/test_release_evidence.pydev.jsonandrequirements-dev.txt; the route held to the gate's tag grammar; the release steps and their order; the evidence job's setup held to main CI's; the vendored copy held to the kit releasejpdev.json,AGENTS.md/CLAUDE.md,web/DEPLOYMENT.mdInventory
conformance.yml, push tomain)vMAJOR.MINOR.PATCHv0.3.0-rc.1,policy-v…)repository-tests:tests/check.sh --all, in.venv)python -m unittest discover -s tests -v)local-paths,agent-docs,diff-check,policy-prototype,policy-contract,policy-official-draftPASSin the evidence (not run)python-deps:requirements-dev.txtwith theformat-nongplextrascheck.sh --setup,.venv)PASS; the installed set compared; the format checkers pinned (below)tools/release_milestone.py)policy-official-draftpolicy-vtagspolicy-vtags, unchangedHEAD, clean tree, artifact pins, notes path, Policy tag history (git tag --list policy-v*)--draft --prerelease --verify-tag, notes fromreleases/<tag>.md), bundle attestationIdentical, so reused: the whole repository suite. Shown three ways on main's tree (2639de8):
v0.3.0tag at the same commit, with release context variables (GITHUB_REF=refs/tags/v0.3.0,GITHUB_REF_TYPE=tag, workflowRelease): the same result for every test.gitfirst onPATH: 25 git calls in the checkout, the same in both contexts, every one a read ofHEAD, its tree or its attributes (archive,ls-tree,check-attr,read-tree HEAD/add -A/write-treeinto a temporary index). Notag,describe,for-each-ref,ls-remote,log, branch or remote read; the other git calls run in fixture repositories under temporary directories. Their count differed by one round ofgit showin one fixture:recover_core_rc.pychecks archive members in set order and a mutation test stops at the first difference, so the count follows the hash seed, not the context.GITHUB_*without setting it, reaches the network, or reads the clock (tests/*.py); the milestone, tag and remote reads of the release tools under test go to stand-ingh/gitcommands or patched functions.The intentional differences that remain are compared instead of assumed: the runner image (
ImageOS,ImageVersion, OS release digest), Python, Node and git versions, and the installed requirements.requirements-dev.txtpins only its direct requirements, so the resolved set is recorded (pip freeze --allof the.venvthe checks ran in) and must equal what the release runner resolves when it sets up the same way (dev.jsontoolchains with the same action pins,check.sh --setup). The kit's rule that an unpinned input is not reusable is met by that comparison: a dependency released between main's run and the tag refuses (runtime-identity) and the fresh route makes new evidence.Fresh, and why. Everything that reads the tag, the milestone, the Policy tag history or makes the artifacts stays in the release job: the gate decides none of it and caches nothing. Tags outside the gate's grammar reuse nothing: kit v0.10.0's gate admits only
vMAJOR.MINOR.PATCH(tag-syntaxotherwise), and this repository's release candidates andpolicy-vtags carry a suffix, so the route step sends them to the whole suite instead of a certain refusal. The route is the gate's own grammar (a test holds the two equal), and the fresh route runs strictly more. The exact Policy corpus validation therefore stays fresh for everypolicy-vtag; no equivalence is claimed.The extras are part of the evidence's identity. A run without
jsonschema[format-nongpl]'s extras passes while checking less. The evidence proves they were installed three ways: the setup log'sPASS python-deps(the install of the exactrequirements-dev.txt, whose object id is a recorded input); the installed set (pip freeze --all); andjsonschema-formats, pinned tojsonschema==4.23.0 formats: color date date-time duration email hostname idn-email idn-hostname ipv4 ipv6 json-pointer regex relative-json-pointer time uri uri-reference uri-template uuid. Measured locally: a clean venv fromrequirements-dev.txtprints exactly that;jsonschema==4.23.0alone printsdate email idn-email ipv4 ipv6 regex time uuid, which refuses. A test holds the pin's version torequirements-dev.txtand requiresurianddate-timein it.The contract (issue #257's evidence gate)
head_sha, workflow id + path +name:, push on the default branch with no pull request association, the attempt's jobs by display name on hostedubuntu-latest, artifacts bound to the run; capture refuses unlessGITHUB_WORKFLOW_REF/GITHUB_WORKFLOW_SHAareconformance.ymlat the commit. A PR head, synthetic merge, another workflow, a status name or a tree-equivalent commit is refuseddev.jsoncheck and setup step must bePASSrelease-evidence-<attempt>, kept on refusal too (!cancelled(),if-no-files-found: error), with run id/attempt, inputs, job results, runtime and coveragetests/release_evidence_test.py, 85 tests, kit PR #52). Here: the configuration and the workflows, belowweb/DEPLOYMENT.md: re-run all jobs of main's run at the commit, then re-run the release run's failed jobs once; a second refusal: fix onmain, release from a new commit. Nothing re-runs automatically, and no approval gate is added or removedrepository-conformancekeeps its name and runs on every pull request and push as beforeLive probe (read-only API, the vendored gate's own code)
Every probe runs the vendored gate's own
decide()over this repository's live API (GET only). Substituted in each: the release checkout context (this branch's configuration read at the probed commit, which does not hold it), the tag object (novMAJOR.MINOR.PATCHtag names these commits) and the runner identity (the probe is not a hosted runner).2639de8(main's tip),Repository conformancerun 40396213185, push, success, 0.9 h oldrefuse: artifact-coverage: 0 artifacts namedci-evidence-conformance-repository-conformance-1, because capture lands with this pull request. Everything before it passed live: repository and default branch, the workflow resolved by path, name and state, selection (1 run at the commit, 1 applicable, none newer), the run read again (completed, success, age), and the attempt's jobs (repository-conformancesuccess onubuntu-latest, hosted runner group)evidence()andrequired_checks()(everydev.jsoncheck and the setup stepPASS), bound to the live run, attempt and job intervalreuse(run 40396213185, attempt 1)a3cdb91, the head of #272, run 40395750281 (pull_request, success)refuse: run-missing(1 run at the commit, 0 applicable)c0a7aa9, its run 40405251587 (pull_request; probed while running and again after it completed with success)refuse: run-missingboth times (1 run at the commit, 0 applicable)Capture, dry run: on this branch's head with the GitHub context variables set by hand and the real
check.shlogs of the run below,captureexits 0 and records the 6 input identities,PASSfor the 7 checks andsetup:python-deps, the 20-linepip freeze --allof.venvand the pinned format line. It is not evidence (the gate never reads a local file); it shows the summary format, the tool commands and a clean tracked tree after the checks.Mutants
Each in a scratch copy of the checkout; the new test file fails at the named assertion, and passes unmutated.
test_every_job_trigger_and_capture_exists_in_its_workflow:'repository-conformance' != 'conformance'test_every_input_is_a_tracked_filetest_every_job_trigger_and_capture_exists_in_its_workflowmainif:differs--changedselectionMAX_AGE_HOURS = 72)test_vendored_copy_is_the_kit_library_after_line_two: digest differsfresh(a missing route would skip it)test_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_releaseNone != 'evidence'test_the_route_admits_exactly_the_gates_tag_grammarcontents: writetest_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_releasetest_release_steps_keep_their_order_and_release_semanticsuritest_installed_requirements_are_recorded_from_the_checks_environmenttest_the_evidence_job_sets_up_its_runner_as_main_ci_doestest_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_releaseDuration (measured; nothing else claimed)
policy-v1.0.0-draft.1, release run 39299066788v0.3.0-rc.1, release run 40192247704repository-conformance, the whole list)After: not measured yet. On reuse the suite step is skipped, and a separate job runs first (runner start, checkout, Python and Node setup,
check.sh --setup, the gate; the kit's own gate step took about 9 s and its job 14–15 s at kit v0.9.0). Whether the release gets faster is measured atv0.3.0, not promised. Tags on the fresh route now wait for a one-step job before the release job. Main CI adds the capture and upload steps on a push tomain.Not verifiable before merge
captureruns only on a push tomain, so this pull request's CI does not exercise it; the first main push after merge does, and a capture refusal fails that run visibly. A local dry run ofcaptureon this branch's commit, with the GitHub context variables set by hand and the realcheck.shlogs, is described at the end of the live probe section above.verifyis first live at thev0.3.0release.Checks
Head
c0a7aa9(no uncommitted diff).jpdev check --changedagainstorigin/main(merge base2639de8):jpdev.jsonchanged, so the whole list.actionlint(with shellcheck) passes on both workflows. Under an outerulimit -v 4000000fivetest_profilescases fail only because Node cannot reserve WebAssembly memory (identically on both inventory contexts);jpdev checkcaps Node by resident memory instead and they pass.Fix round
For the review record at
c0a7aa9(0 HIGH, 2 MEDIUM, 3 LOW). One commit:31cda9c.if:onrelease; the gate's exit status reaches its step)reuse, which a new step after the gate (Hand the reuse to the release job) writes only when the decision record it reads saysreusewith no refusalif:; nocontinue-on-erroranywhere; the gate step'srun:is exactly the command; the evidence job's step order; the hand-over's own script run over reuse, refuse, reuse-with-refusal, no-decision, non-object and missing records (only reuse hands overtrue)if: always()onrelease, R-6|| trueon the gate, F-1set +e, F-2continue-on-erroron the gate step, F-3 hand-over without the record check, F-4 without the refusal check, F-5 hand-over before the gate, F-6 suite keyed to the route again, F-7 the job output taken from the route: all caughtbash --noprofile --norc -eo pipefailunderLC_ALL=Cover 16 tags;route=evidenceexactly where the gate'sTAG.fullmatchadmits the tag (v0.3.0,v1.10.0,v10.0.0); everything else goes to the whole suite (-rc.1,-rc.10,-draft,-0,+build,policy-v1.0.0,policy-v1.0.0-draft.1,v1.2,v01.0.0,v0.3.0.1, a trailing space, Arabic-Indic digits, the demo tag)test_the_route_step_sends_exactly_the_gates_tags_to_the_gateweb/DEPLOYMENT.md: the kit's prerelease grammar (kit v0.11.0) is opt-in; until this repository opts in, release candidates keep the whole suite, and opting in widens the route step and its test in the same change;policy-vis outside every kit grammarAll 15 earlier mutants were run again on the fix (M08 rewritten for the
reuseoutput) and are caught; 26 of 26 in all, and the unmutated tests pass (11 tests).actionlintwith shellcheck passes.Candidate
31cda9c,jpdev check --changed(whole list):🤖 Generated with Claude Code