Skip to content

Release: reuse the identical repository test evidence from main, keep tag and Policy validation fresh, fail closed (#257) - #273

Merged
kikashy merged 4 commits into
mainfrom
release-evidence-257
Oct 26, 2026
Merged

kikashy merged 4 commits into
mainfrom
release-evidence-257

Conversation

@kikashy

@kikashy kikashy commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #257. Release orchestration only: no normative text, schema, example, conformance case or conformance semantics changes, and nothing a policy-v tag validates (its bundle paths and corpus) is touched.

Summary

A vMAJOR.MINOR.PATCH release no longer runs the repository suite again. A new CI evidence at the tag job runs the development kit's release evidence gate (kit v0.10.0 lib/release_evidence.py, vendored as scripts/dev/release_evidence.py, line 2 naming the release, byte-identical after it) against the tagged commit, configured by .github/ci-evidence.json (schema 2). It reuses the evidence main's repository-conformance job recorded at that exact commit, or refuses and fails the release before anything is built. conformance.yml records that evidence on a push to main only. The release job keeps every release-specific step, unchanged and in the same order; only its repository-suite step is skipped, and only when the gate said reuse.

File Change
.github/ci-evidence.json workflow Repository conformance by path and name:, event push on main (the repository has no dispatch on it), job repository-conformance on ubuntu-latest, checks all with no permitted skips, setup steps recorded, inputs requirements-dev.txt and scripts/dev/check.sh (beside dev.json, the configuration, the gate and the workflow, always recorded), one Linux profile compared with the release runner
.github/workflows/conformance.yml the setup and check runs are also written to logs (pipefail); on a push to main only: release_evidence.py capture, then ci-evidence-conformance-repository-conformance-<attempt> uploaded with a pinned actions/upload-artifact
.github/workflows/release.yml new evidence job (contents: read, actions: read); release needs it; the suite step runs unless the evidence job hands over true, which a step after the gate does only when the decision record says reuse (fix round)
scripts/dev/release_evidence.py the vendored gate
tests/test_release_evidence.py the configuration held to the workflows, dev.json and requirements-dev.txt; the route held to the gate's tag grammar; the release steps and their order; the evidence job's setup held to main CI's; the vendored copy held to the kit release
jpdev.json, AGENTS.md/CLAUDE.md, web/DEPLOYMENT.md the new files mapped; one sentence; a "Release evidence" section (the contract, the decision record, the fresh route, what stays fresh)

Inventory

Check Main CI (conformance.yml, push to main) Release before Release now, vMAJOR.MINOR.PATCH Release now, other tags (v0.3.0-rc.1, policy-v…)
repository-tests: tests/ run (check.sh --all, in .venv) run again (python -m unittest discover -s tests -v) reused from main's evidence run again, unchanged
local-paths, agent-docs, diff-check, policy-prototype, policy-contract, policy-official-draft run not run required PASS in the evidence (not run) not run
setup python-deps: requirements-dev.txt with the format-nongpl extras run (check.sh --setup, .venv) installed by the release required PASS; the installed set compared; the format checkers pinned (below) installed, unchanged
milestone and tagged-checkout admission (tools/release_milestone.py) not run fresh fresh fresh
exact Policy format and agreement corpus as policy-official-draft fresh for policy-v tags not a Policy tag fresh for policy-v tags, unchanged
bundle build: tag grammar and commit, HEAD, clean tree, artifact pins, notes path, Policy tag history (git tag --list policy-v*) not run fresh fresh fresh
draft prerelease (--draft --prerelease --verify-tag, notes from releases/<tag>.md), bundle attestation not run fresh fresh fresh

Identical, so reused: the whole repository suite. Shown three ways on main's tree (2639de8):

  • The suite (436 tests) ran on a branch checkout without the release tag, with main-CI context variables, and on a detached checkout of a new annotated v0.3.0 tag at the same commit, with release context variables (GITHUB_REF=refs/tags/v0.3.0, GITHUB_REF_TYPE=tag, workflow Release): the same result for every test.
  • A logging git first on PATH: 25 git calls in the checkout, the same in both contexts, every one a read of HEAD, its tree or its attributes (archive, ls-tree, check-attr, read-tree HEAD/add -A/write-tree into a temporary index). No tag, describe, for-each-ref, ls-remote, log, branch or remote read; the other git calls run in fixture repositories under temporary directories. Their count differed by one round of git show in one fixture: recover_core_rc.py checks archive members in set order and a mutation test stops at the first difference, so the count follows the hash seed, not the context.
  • No test reads GITHUB_* without setting it, reaches the network, or reads the clock (tests/*.py); the milestone, tag and remote reads of the release tools under test go to stand-in gh/git commands or patched functions.

The intentional differences that remain are compared instead of assumed: the runner image (ImageOS, ImageVersion, OS release digest), Python, Node and git versions, and the installed requirements. requirements-dev.txt pins only its direct requirements, so the resolved set is recorded (pip freeze --all of the .venv the checks ran in) and must equal what the release runner resolves when it sets up the same way (dev.json toolchains with the same action pins, check.sh --setup). The kit's rule that an unpinned input is not reusable is met by that comparison: a dependency released between main's run and the tag refuses (runtime-identity) and the fresh route makes new evidence.

Fresh, and why. Everything that reads the tag, the milestone, the Policy tag history or makes the artifacts stays in the release job: the gate decides none of it and caches nothing. Tags outside the gate's grammar reuse nothing: kit v0.10.0's gate admits only vMAJOR.MINOR.PATCH (tag-syntax otherwise), and this repository's release candidates and policy-v tags carry a suffix, so the route step sends them to the whole suite instead of a certain refusal. The route is the gate's own grammar (a test holds the two equal), and the fresh route runs strictly more. The exact Policy corpus validation therefore stays fresh for every policy-v tag; no equivalence is claimed.

The extras are part of the evidence's identity. A run without jsonschema[format-nongpl]'s extras passes while checking less. The evidence proves they were installed three ways: the setup log's PASS python-deps (the install of the exact requirements-dev.txt, whose object id is a recorded input); the installed set (pip freeze --all); and jsonschema-formats, pinned to jsonschema==4.23.0 formats: color date date-time duration email hostname idn-email idn-hostname ipv4 ipv6 json-pointer regex relative-json-pointer time uri uri-reference uri-template uuid. Measured locally: a clean venv from requirements-dev.txt prints exactly that; jsonschema==4.23.0 alone prints date email idn-email ipv4 ipv6 regex time uuid, which refuses. A test holds the pin's version to requirements-dev.txt and requires uri and date-time in it.

The contract (issue #257's evidence gate)

Item Here
exact commit, repository, trusted event/ref, workflow identity, job coverage the gate: run, head repository, head_sha, workflow id + path + name:, push on the default branch with no pull request association, the attempt's jobs by display name on hosted ubuntu-latest, artifacts bound to the run; capture refuses unless GITHUB_WORKFLOW_REF/GITHUB_WORKFLOW_SHA are conformance.yml at the commit. A PR head, synthetic merge, another workflow, a status name or a tree-equivalent commit is refused
deterministic selection, no older green past a newer failed/cancelled/running attempt; pagination; reruns the gate's selection (every page; this repository's trusted-branch runs scanned; newest latest-attempt start selected whatever its result; re-read after collection and once more at the end)
explicit skips no permitted skip: every dev.json check and setup step must be PASS
the log the decision record in the job log and as release-evidence-<attempt>, kept on refusal too (!cancelled(), if-no-files-found: error), with run id/attempt, inputs, job results, runtime and coverage
mutable external state milestone, tag, notes, bundles, prerelease and attestation stay fresh in the release job; the gate checks the tag before and after
refusal classes tested in the kit (tests/release_evidence_test.py, 85 tests, kit PR #52). Here: the configuration and the workflows, below
fresh route documented in web/DEPLOYMENT.md: re-run all jobs of main's run at the commit, then re-run the release run's failed jobs once; a second refusal: fix on main, release from a new commit. Nothing re-runs automatically, and no approval gate is added or removed
branch protection repository-conformance keeps its name and runs on every pull request and push as before

Live probe (read-only API, the vendored gate's own code)

Every probe runs the vendored gate's own decide() over this repository's live API (GET only). Substituted in each: the release checkout context (this branch's configuration read at the probed commit, which does not hold it), the tag object (no vMAJOR.MINOR.PATCH tag names these commits) and the runner identity (the probe is not a hosted runner).

Probe Commit and run Decision
main, as it is 2639de8 (main's tip), Repository conformance run 40396213185, push, success, 0.9 h old refuse: artifact-coverage: 0 artifacts named ci-evidence-conformance-repository-conformance-1, because capture lands with this pull request. Everything before it passed live: repository and default branch, the workflow resolved by path, name and state, selection (1 run at the commit, 1 applicable, none newer), the run read again (completed, success, age), and the attempt's jobs (repository-conformance success on ubuntu-latest, hosted runner group)
main, evidence substituted the same run, plus an evidence artifact built with the gate's own evidence() and required_checks() (every dev.json check and the setup step PASS), bound to the live run, attempt and job interval reuse (run 40396213185, attempt 1)
a pull request run a3cdb91, the head of #272, run 40395750281 (pull_request, success) refuse: run-missing (1 run at the commit, 0 applicable)
this pull request c0a7aa9, its run 40405251587 (pull_request; probed while running and again after it completed with success) refuse: run-missing both times (1 run at the commit, 0 applicable)

Capture, dry run: on this branch's head with the GitHub context variables set by hand and the real check.sh logs of the run below, capture exits 0 and records the 6 input identities, PASS for the 7 checks and setup:python-deps, the 20-line pip freeze --all of .venv and the pinned format line. It is not evidence (the gate never reads a local file); it shows the summary format, the tool commands and a clean tracked tree after the checks.

Mutants

Each in a scratch copy of the checkout; the new test file fails at the named assertion, and passes unmutated.

# Safeguard broken Failing test
M01 configuration job name drifts from the workflow test_every_job_trigger_and_capture_exists_in_its_workflow: 'repository-conformance' != 'conformance'
M02 an input path that does not exist test_every_input_is_a_tracked_file
M03 capture names another job key test_every_job_trigger_and_capture_exists_in_its_workflow
M04 capture runs outside a push to main same: the if: differs
M05 the push run checks a --changed selection same: the whole-list log line is missing
M06 the vendored gate's body changed (MAX_AGE_HOURS = 72) test_vendored_copy_is_the_kit_library_after_line_two: digest differs
M07 line 2 names another kit release same
M08 suite skipped unless the route is fresh (a missing route would skip it) test_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_release
M09 the release job no longer needs the evidence job same: None != 'evidence'
M10 the route admits suffixed tags test_the_route_admits_exactly_the_gates_tag_grammar
M11 the evidence job's token widened to contents: write test_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_release
M12 release steps reordered (bundles before the Policy corpus) test_release_steps_keep_their_order_and_release_semantics
M13 the formats pin without uri test_installed_requirements_are_recorded_from_the_checks_environment
M14 the evidence job sets up another Node action test_the_evidence_job_sets_up_its_runner_as_main_ci_does
M15 the decision record not kept on a refusal test_reuse_skips_only_the_repository_suite_and_a_refusal_stops_the_release

Duration (measured; nothing else claimed)

Run Repository suite step Job
policy-v1.0.0-draft.1, release run 39299066788 49 s 67 s
v0.3.0-rc.1, release run 40192247704 78 s (failed there) 87 s
main CI at 2639de8, run 40396213185 (repository-conformance, the whole list) 106 s

After: not measured yet. On reuse the suite step is skipped, and a separate job runs first (runner start, checkout, Python and Node setup, check.sh --setup, the gate; the kit's own gate step took about 9 s and its job 14–15 s at kit v0.9.0). Whether the release gets faster is measured at v0.3.0, not promised. Tags on the fresh route now wait for a one-step job before the release job. Main CI adds the capture and upload steps on a push to main.

Not verifiable before merge

  • capture runs only on a push to main, so this pull request's CI does not exercise it; the first main push after merge does, and a capture refusal fails that run visibly. A local dry run of capture on this branch's commit, with the GitHub context variables set by hand and the real check.sh logs, is described at the end of the live probe section above.
  • verify is first live at the v0.3.0 release.

Checks

Head c0a7aa9 (no uncommitted diff). jpdev check --changed against origin/main (merge base 2639de8): jpdev.json changed, so the whole list.

== check summary (dev.json) ==
PASS     local-paths              0s
PASS     agent-docs               0s
PASS     diff-check               0s
PASS     repository-tests         113s
    Ran 445 tests in 112.695s
    OK
PASS     policy-prototype         1s
PASS     policy-contract          3s
PASS     policy-official-draft    3s
== 7 passed, 0 failed, 0 skipped, 0 not run ==

actionlint (with shellcheck) passes on both workflows. Under an outer ulimit -v 4000000 five test_profiles cases fail only because Node cannot reserve WebAssembly memory (identically on both inventory contexts); jpdev check caps Node by resident memory instead and they pass.

Fix round

For the review record at c0a7aa9 (0 HIGH, 2 MEDIUM, 3 LOW). One commit: 31cda9c.

Finding Fix Test Safeguards removed in scratch copies
MEDIUM 1: no test held that a refusal stops the release (no job-level if: on release; the gate's exit status reaches its step) tests hold both. Defence in depth: the release job no longer reads the route written before the gate; it reads reuse, which a new step after the gate (Hand the reuse to the release job) writes only when the decision record it reads says reuse with no refusal no job has a job-level if:; no continue-on-error anywhere; the gate step's run: is exactly the command; the evidence job's step order; the hand-over's own script run over reuse, refuse, reuse-with-refusal, no-decision, non-object and missing records (only reuse hands over true) R-5 if: always() on release, R-6 || true on the gate, F-1 set +e, F-2 continue-on-error on the gate step, F-3 hand-over without the record check, F-4 without the refusal check, F-5 hand-over before the gate, F-6 suite keyed to the route again, F-7 the job output taken from the route: all caught
MEDIUM 2: the route test compared the grammar text, not the step the route step's own script runs with bash --noprofile --norc -eo pipefail under LC_ALL=C over 16 tags; route=evidence exactly where the gate's TAG.fullmatch admits the tag (v0.3.0, v1.10.0, v10.0.0); everything else goes to the whole suite (-rc.1, -rc.10, -draft, -0, +build, policy-v1.0.0, policy-v1.0.0-draft.1, v1.2, v01.0.0, v0.3.0.1, a trailing space, Arabic-Indic digits, the demo tag) test_the_route_step_sends_exactly_the_gates_tags_to_the_gate R-2 tag matched without its suffix, R-3 outcomes swapped: caught
LOW 3: prerelease consequence undocumented web/DEPLOYMENT.md: the kit's prerelease grammar (kit v0.11.0) is opt-in; until this repository opts in, release candidates keep the whole suite, and opting in widens the route step and its test in the same change; policy-v is outside every kit grammar
LOW 4: a failed-jobs re-run keeps a passed evidence job's decision documented: re-run all jobs of the release run when main's run at the commit has a newer attempt or the decision is nearly 48 hours old; plus waiting out a hosted-image rollout (the review's note)
LOW 5: stale probe row and "is below" edited above

All 15 earlier mutants were run again on the fix (M08 rewritten for the reuse output) and are caught; 26 of 26 in all, and the unmutated tests pass (11 tests). actionlint with shellcheck passes.

Candidate 31cda9c, jpdev check --changed (whole list):

== check summary (dev.json) ==
PASS     local-paths              0s
PASS     agent-docs               0s
PASS     diff-check               0s
PASS     repository-tests         112s
    Ran 447 tests in 111.529s
    OK
PASS     policy-prototype         1s
    Ran 38 tests in 0.902s
    OK
PASS     policy-contract          2s
    Ran 13 tests in 2.345s
    OK
PASS     policy-official-draft    2s
    Ran 16 tests in 2.294s
    OK
== 7 passed, 0 failed, 0 skipped, 0 not run ==

🤖 Generated with Claude Code

A vMAJOR.MINOR.PATCH release no longer runs the repository suite again.
A new `CI evidence at the tag` job runs the development kit's release
evidence gate (lib/release_evidence.py of kit v0.10.0, vendored as
scripts/dev/release_evidence.py, byte for byte after line 2) against
the tagged commit, configured by .github/ci-evidence.json: the newest
push-to-main `repository-conformance` run at that exact commit, every
setup step and check PASS, the commit's input identities, and the
runner image, Python, Node, git and installed requirements equal to
what the release runner resolves, with the jsonschema format checkers
pinned so a run without the format-nongpl extras refuses. A refusal
fails the release before anything is built; the fresh route is the
maintainer's (web/DEPLOYMENT.md, "Release evidence").

conformance.yml records that evidence on a push to main only (capture
and a pinned upload). A tag outside the gate's grammar (a release
candidate, a policy-v tag) reuses nothing and runs the whole suite as
before. The milestone and tagged-checkout admission, the exact Policy
corpus for policy-v tags, the bundle build, notes, draft prerelease and
attestation stay fresh, unchanged and in the same order.

tests/test_release_evidence.py holds the configuration to both
workflows, dev.json and requirements-dev.txt, the route to the gate's
tag grammar, and the vendored copy to the kit release. Release
orchestration only: no normative text, schema or conformance semantics
change.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikashy kikashy added this to the v0.3.0 milestone Oct 26, 2026
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Review record: spec PR #273 (issue #257), head c0a7aa9, base 2639de8

The release's reuse of main's evidence is sound as written: the suite is context-independent, RC and policy-v tags never reach the gate, and a refusal stops the release. No HIGH.
Two fail-closed properties are not held by tests (4 of my 6 mutants survive). Clean-room review of a trust boundary; same-vendor review, not a cross-vendor claim.

Dispositions

# Finding Sev Evidence What to do
1 "A refusal stops the release" rests on two properties no test holds: the release job has no job-level if:, and the verify step's exit status reaches the step MEDIUM R-5 (if: always() on release) and R-6 (verify ... || true) both survive all 9 tests. Either one would let a refused gate reach the build with the suite skipped, because route=evidence is written before the gate runs Assert field(release, "if", 4) is None and that the verify step's run: is exactly the command. Optional defence in depth: emit the reuse output from a step after verify
2 The route test compares the grammar strings, not what the route does MEDIUM R-2 (tag matched with ${RELEASE_TAG%%-*}) and R-3 (outcomes swapped) survive. The harm is bounded: RC and policy-v tags would reach the gate and refuse tag-syntax, which blocks those releases but never reuses evidence Run the route step's script with bash over the sample tags and assert route=evidence exactly when gate.TAG.fullmatch
3 The docs do not give the prerelease consequence LOW Kit main (devkit #55, v0.11.0) keeps TAG and adds TAG_PRERELEASE behind "tag": {"prerelease": true}. The re-vendor stays green and RC tags stay fresh. Opting in needs the route widened and the test compared with tag_grammar(config). policy-v is outside every kit grammar, so it always runs fresh One sentence in web/DEPLOYMENT.md "Release evidence" (the repository has no docs/releasing.md)
4 A failed-jobs re-run of a release run whose evidence job passed reuses that attempt's reuse LOW GitHub keeps successful jobs and their outputs, so the gate is not asked again. A newer failed main attempt or the 48 h limit goes unchecked Document "re-run all jobs of the release run", or accept and state it
5 PR body: two stale details LOW The "this pull request" probe row is filled, but says "in progress": run 40405251587 has since completed with success, and my re-probe still gives run-missing. Also, "a local dry run of capture ... is below" refers to text that sits above it Edit the body

Notes (no finding):

  • runtime-identity refusals are documented with the fresh route. During a staggered hosted-image rollout, the re-run can refuse again with nothing on main to fix; the docs could mention waiting.
  • The 18-format list is enforced at verify. The test holds only the version plus uri and date-time.

Probes (read-only API; scratch clones; caps on every run)

  • Identity (1). I ran a branch checkout main (5 tags, main-CI env) against a detached checkout of a new annotated v0.3.0 (6 tags, release env: GITHUB_REF=refs/tags/v0.3.0, workflow Release).
    • Five targeted modules: test_release, test_policy_release, test_release_milestone, test_historical_release and test_release_recovery. I added test_release_evidence and test_site, for 153 tests with identical per-test results.
    • Whole suite: 445 tests, identical per-test results. Five test_profiles Node/Wasm cases fail in both contexts under ulimit -v; this is the known cap artifact, and they read no git or environment.
    • A logging git saw 27 calls in the checkout, identical in both contexts: add, archive, check-attr, ls-files, ls-tree, read-tree and write-tree. There were no tag, describe, for-each-ref or log calls. validate_policy_history (git tag --list) is patched in its test.
  • Route (2). I executed the route step under LC_ALL=C over 13 tags (v0.3.0, v1.10.0, -rc.1, -rc.10, policy-v1.0.0, policy-v1.0.0-draft.1, v01.0.0, v0.3, +build, v1.0.0-0, Arabic-Indic digit, trailing space, demo/.../v0.2.0). It sent evidence exactly where the gate's TAG.fullmatch admits the tag; every other tag went fresh.
  • Fail closed (3). release has needs: evidence and no if, so the job runs only if evidence succeeded.
    • verify exits 1 on every non-reuse decision. Every evidence-path step carries an implicit success(), and there is no continue-on-error.
    • An empty or absent route gives '' != 'evidence', so the suite runs.
    • The record is uploaded under !cancelled() with if-no-files-found: error, so it is kept on refusal. The fresh route has no record; the decision is in the step log.
  • Extras (4). In a clean venv, requirements-dev.txt gives the 18 formats, equal to the pin. jsonschema==4.23.0 alone gives 8 (date email idn-email ipv4 ipv6 regex time uuid), which refuses.
    • pip freeze --all lists 20 entries: 2 pinned and 18 unpinned, pip among them (pip comes from the venv, not PyPI).
    • Resolution is identical on the same day unless PyPI moves. It does move: 29 releases of those packages in 180 days, 6 of them jsonpointer in October. Drift refuses with runtime-identity, documented with its fresh route.
  • Inputs (5). Run head_sha == GITHUB_SHA == checkout HEAD, with the tracked tree clean. That binds requirements-dev.txt, dev.json, the workflows, tests/fixtures, schema/ and conformance/ by commit; six of them are also recorded by object id. Runner image, OS release, Python, Node, git and the resolved packages are compared. The suite reads no clock, network or GITHUB_*.
  • Live (6). These calls use the PR's configuration and fake only the tag:
    • Main tip 2639de8, pure: config-source, because the commit does not hold the configuration yet.
    • Main tip 2639de8 with the configuration substituted: run 40396213185 (push, success, 1.2 h, 1 applicable) gives artifact-coverage (0 ci-evidence-...-1).
    • PR head c0a7aa9 with the real source context: run 40405251587 (pull_request) gives run-missing.
    • Re-vendor the devkit scripts at v0.10.0 #272 head a3cdb91 gives run-missing.
    • PR head with no tag substitution gives api-unavailable, because the tag is absent.

Mutants (test_release_evidence.py; unmutated OK)

  • Builder's, all 8 re-run and caught: M04 capture outside push to main, M05 --changed on push, M06 gate body changed, M08 suite == 'fresh', M11 contents: write, M12 bundles before Policy, M13 pin without uri, M15 record not kept on refusal.
  • Mine, caught: R-1 grammar loses $, R-4 needs dropped.
  • Mine, survived: R-2 and R-3 (finding 2); R-5 and R-6 (finding 1).

Verified unchanged

  • Release job: 12 steps in the same order. Milestone admission, the Policy corpus if: startsWith(github.ref_name, 'policy-v'), the bundle build (tag, HEAD, clean tree, pins, Policy tag history), notes, --draft --prerelease --verify-tag and attestation are all unchanged. The trigger and concurrency are unchanged, and the only new if is on the suite step.
  • Vendored gate: byte-identical to kit v0.10.0 lib/release_evidence.py after line 2 (sha256 d44af3df…).
  • Evidence job: contents: read and actions: read under workflow permissions: {}. Tags are passed through env, never inline. Artifact retention is 7 days against a 48 h limit.
  • Main CI: repository-conformance keeps its name and job id. pipefail is set on both tee'd steps.
  • Public text: the diff and the PR body have no local paths or personal names.

merge: yes (no HIGH). Disposition per the kit policy: both MEDIUMs and the cheap LOWs are fixed in this PR in one round by the builder and confirmed by this reviewer.

#257)

Review round on #273. The release job takes the reuse from a step after
the gate that reads the decision record and hands over `true` only for
a reuse, so a refused gate cannot skip the suite even if its exit status
were lost. Tests now hold that no job carries a job-level `if:` or
`continue-on-error`, that the gate step runs the bare command, the
evidence job's step order, the hand-over over reuse, refuse and
malformed records, and the route step's own script run with bash over
sample tags against the gate's grammar.

web/DEPLOYMENT.md: a failed-jobs re-run keeps an earlier passed
decision (re-run all jobs for a new one), waiting out a hosted-image
rollout, and the kit's opt-in prerelease grammar.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Confirmation: 31cda9c

Fix round: one commit on c0a7aa9 (release.yml, test_release_evidence.py, web/DEPLOYMENT.md; +122/−14). Same-vendor review, not a cross-vendor claim.

# Finding Status Evidence
1 Fail-closed path not test-held fixed R-5 (if: always()) and R-6 (|| true) are now caught. Every job is held to having no job-level if, the verify run: is held exactly, and continue-on-error is refused. The design is also stronger: the suite now waits on needs.evidence.outputs.reuse != 'true', and reuse is written only by a step placed after the gate
2 Route held by text, not by behaviour fixed The test now runs the route step's script under bash over 16 tags against TAG.fullmatch. R-2 and R-3 are caught
3 Prerelease consequence not documented fixed DEPLOYMENT.md: tag.prerelease is opt-in from kit v0.11.0; opting in widens the route and its test in the same change; policy-v always runs the whole suite
4 Failed-jobs re-run keeps the old decision fixed (documented) DEPLOYMENT.md says this, and says to re-run all jobs when main's run has a newer attempt or the decision is near 48 h. It also adds the image-rollout wait
5 PR body rows fixed The probe row says "while running and again after it completed with success"; the dry-run sentence now points to the right place

The reuse flag cannot be satisfied by a stale or forged record. It is read in the same job from that job's runner.temp, by a step whose if: carries the implicit success(). The step therefore runs only after the gate exited 0. The gate exits 0 only after it has overwritten that path with a reuse record, so a file planted by an earlier step is replaced first. A missing, refused, malformed or non-object record makes the step fail. The test also covers a reuse record that carries a refusal. If the release job somehow ran after a refusal, the empty reuse would still run the suite.

Mutants at 31cda9c: 10 of 10 caught. These are my four earlier survivors (R-2, R-3, R-5, R-6) plus six new ones aimed at the flag:

  • N-1: the decision check removed.
  • N-2: the flag step runs after a failed gate.
  • N-3: the job output taken from the route instead of the flag.
  • N-4: the flag reads another record path.
  • N-5: continue-on-error on verify.
  • N-6: the check accepts any decision except refuse.

Other checks: test_release_evidence 11/11 OK. Unmutated, test_release_recovery, test_release_milestone, test_release and test_policy_release pass (79 tests). actionlint is clean on both workflows. No local paths or names appear in the diff or the body. The PR's own CI run at 31cda9c was still in progress when I checked.

No new HIGH or MEDIUM.

merge: yes (31cda9c), once its repository-conformance run is green.

kikashy and others added 2 commits October 26, 2026 18:06
scripts/dev/release_evidence.py from judgment-pack-devkit v0.11.0 (the
tag grammar with SemVer prereleases, dispatch validation runs); tag.prerelease is
enabled because this repository publishes prerelease tags.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
…ease in the copy test

check.sh, handover_check.sh and changed_checks.py from judgment-pack-devkit
v0.11.0 (line 2 only); the vendored-copy test records the v0.11.0 gate
file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Confirmation: 74a0b77

One commit on 31cda9c. It re-vendors the gate at kit v0.11.0 and sets "tag": {"prerelease": true}; the .github/ci-evidence.json arrays are reformatted with no change in meaning. release.yml and the tests are unchanged. Same-vendor review, not a cross-vendor claim.

Finding Severity Evidence What to do
The vendored-copy test still pins the previous kit release, so the repository suite fails BLOCKER (red required check) The gate is byte-identical after line 2 to the kit's lib/release_evidence.py at tag v0.11.0 (sha256 f272b7a4…). But KIT_RELEASE = "v0.10.0" and KIT_LIBRARY_SHA256 = d44af3df… were not updated, so test_vendored_copy_is_the_kit_library_after_line_two FAILS (10 of 11 pass). The PR's repository-conformance run on this commit was still in progress when checked Set KIT_RELEASE = "v0.11.0" and KIT_LIBRARY_SHA256 = "f272b7a43272bac31bb9b0c37a583d9c40d376972b04c812e576bd82c1ef18e3"

What I verified:

  • Tests. The route tests (pattern text and behaviour), the fail-closed tests and the reuse-flag tests pass unchanged. The gate accepts the configuration, and tag_grammar(config) is now the prerelease grammar.
  • Release-candidate tags. The route still sends v0.3.0-rc.1 to the whole suite before the gate is reached. The gate's widened grammar would admit it, but nothing routes it there.
  • policy-v tags. policy-v1.0.0 and policy-v1.0.0-draft.1 take the whole-suite route. They also fail every kit grammar (tag-syntax).
  • The v0.3.0 walk is unchanged. It runs as a publication with the prerelease grammar passed, against main run 40396213185, and is refused with artifact-coverage as before.
  • New dispatch mode. A workflow_dispatch run is refused with dispatch-not-gated, because release.yml has no dispatch trigger and dispatch keeps its default. The new validated decision can never pass the reuse flag, which requires reuse.

Note: widening the route for release candidates is a separate, later decision. It would change the route step and its test in the same change (DEPLOYMENT.md says so). Until then the route deliberately admits a subset of the configured grammar, which is the safe direction.

merge: no (74a0b77). It becomes yes after the two test constants are updated and repository-conformance is green. Nothing else changes.

Confirmation: 9d8edcd

One commit on 74a0b77, touching four files with one line changed in each:

  • KIT_RELEASE = "v0.11.0" and KIT_LIBRARY_SHA256 = f272b7a4… in the copy test, which fixes the 74a0b77 blocker.
  • Line 2 only of check.sh, changed_checks.py and handover_check.sh, re-vendored at v0.11.0.

Same-vendor review, not a cross-vendor claim.

  • Tests: test_release_evidence passes 11 of 11.
  • Vendored scripts, against the kit's v0.11.0 tag:
    • check.sh is byte-identical to the kit's templates/check.sh, whose own line 2 names the release.
    • changed_checks.py and handover_check.sh are byte-identical after line 2 to the kit's lib/ files.
    • release_evidence.py is still identical after line 2.
    • Between kit v0.10.0 and v0.11.0 the template changes only its line 2.
  • Route: the workflows have not changed since 31cda9c. The route still sends v0.3.0 to the evidence path and v0.3.0-rc.1, policy-v1.0.0 and policy-v1.0.0-draft.1 to the whole suite.
  • CI: the PR's repository-conformance run on this commit was still in progress when I checked.

merge: yes (9d8edcd), once its repository-conformance run is green. No open findings remain.

@kikashy
kikashy merged commit 24b1f05 into main Oct 26, 2026
3 checks passed
@kikashy
kikashy deleted the release-evidence-257 branch October 26, 2026 22:12
@github-actions

Copy link
Copy Markdown
Contributor

🎉 Thank you for your contribution to Judgment Pack!

If you’d like another task, choose a path that interests you:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reuse equivalent repository test evidence in releases while retaining tag and Policy validation

1 participant