Repository navigation
Release: reuse main's exact-commit CI evidence for the history-independent checks, keep the tag-context run for the rest, fail closed (#38) - #44
Conversation
…ndent checks, keep the tag-context run for the rest, fail closed (#38) The release no longer runs the whole check list twice more at the tag. It takes the evidence main's push CI records at the exact release commit for the checks that read the commit's tree (local-paths, agent-docs, diff-check), through the development kit's release evidence gate (v0.10.0, vendored as scripts/dev/release_evidence.py, configured by .github/ci-evidence.json), and refuses when that evidence is missing, partial, stale or different. The entries tests read the tags and the release history, which the release's tag changes, so the release still runs them at the tag: release.yml calls ci.yml with tag-context: true, after the gate. That call is the one run in the tag's context; ci.yml's push trigger is held to branches, so a tag push no longer starts a duplicate. ci.yml runs a push's list in two parts (the tree checks, whose summary is the evidence log, then HISTORY_CHECKS) and, on a push to main, records and uploads the evidence. Every release-specific check is kept as it was and in the same order. tests/test_release_evidence.py holds the configuration to ci.yml, dev.json and the tree, the release's wiring, and the vendored copy to the kit's file. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review record: exchange PR #44 (issue #38), head 134100a, base 6398ed8The PR is correct on every point in the brief. No HIGH: no reused check reads tags or history, the tag-context run keeps full history and the pinned specification, and a refusal stops everything downstream. Two MEDIUMs are mutants the new tests do not catch: a reused check's command is never held to tree-only, and the downstream jobs are never held free of a job-level
(1)-(5) Checks
(6) Live probe (read-only; only the tag stubbed)
(7) Mutants (a scratch clone for each)
(8) Docs and the PR body
Verified unchanged
merge: yes (no HIGH). Disposition per the kit policy: both MEDIUMs and the two LOWs are fixed in this PR in one round by the builder and confirmed by this reviewer. |
… gate (#38 review) The review of #44 found two safeguards no test held. The three checks the release takes from main's evidence (local-paths, agent-docs, diff-check) are now pinned by their dev.json commands, and each command and the repository script it runs (scripts/sync-agent-docs.sh) is scanned for a read of a tag, the run's ref, the history or the specification checkout: editing one means classifying it again. Every release job after the gate now must need it, directly or through the job it needs, and none may carry a job-level condition or continue-on-error, so no job can run past a refusal. The release_being_made docstring no longer names a tag push. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirmation: 15ff65eOne commit on top of 134100a: 2 files, +99/−2, tests and a docstring only (no workflow, config or gate change). In a scratch clone with
merge: yes (15ff65e). No HIGH; no MEDIUM open. |
scripts/dev/release_evidence.py from judgment-pack-devkit v0.11.0 (the tag grammar with SemVer prereleases, dispatch validation runs). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
…ease in the copy test check.sh, handover_check.sh and changed_checks.py from judgment-pack-devkit v0.11.0 (line 2 only); the vendored-copy test records the v0.11.0 gate file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Confirmation: fbd0e86
merge: no (fbd0e86) until the copy test is moved and Confirmation: 510aa06
merge: yes (510aa06). |
|
🎉 Thank you for your contribution to Judgment Pack! If you’d like another task, choose a path that interests you: |
Release notes for Exchange v0.2.2, a tooling-only release: the development kit's checks and re-vendored scripts, the changed-path mapping on pull requests, and the release evidence gate (#44). The presets, the manifest and the specification verifier pin are unchanged since v0.2.1. No release is published by this pull request; the signed tag on the merged commit starts the release workflow, which waits at the `release` gate. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Closes #38. Its "Required evidence gate" list is the acceptance contract. Milestone v0.3.0 (tooling). One review round: release admission is a trust boundary.
Summary
A release no longer runs the whole check list twice more at the tag (once from the tag push's own CI run, once from the release's call). It now does two things in order:
local-paths,agent-docsanddiff-check. It does this through the development kit's release evidence gate (devkit v0.10.0, vendored asscripts/dev/release_evidence.py, configured by.github/ci-evidence.json) in a new job,CI evidence at the tag. Any refusal fails that job, names the code and stops the release.entriestests again in the tag's context. These tests read the tags and the release history (release_being_made()and the tag listing intests/test_entries.py), and the release's tag changes both. So the release still runs them at the tag, after the gate:CI at the tagcalls the commit'sci.ymlwithtag-context: true. The checkout still has the whole history and tags, the specification is still at the pin, and no verifier test is skipped. That call is now the only run in the tag's context.ci.yml'spushtrigger is limited to branches, so a tag push no longer starts a duplicate run that gates nothing.Every release-specific check stays exactly as it was and in the same order: tag grammar (shell and
release_tag_ok), tag names the commit, commit on main, notes, environment reviewer, manifest blob bytes, checksums, attestation, no existing release, draft identity, notes and asset digests, published digests. The requiredentriesstatus keeps its name (the job now saysname: entriesexplicitly, which gives the same check name). Branch protection, review and publication requirements are unchanged.Public surface changes
ci.ymlgains aworkflow_callinputtag-context(boolean, defaultfalse). A push now runs its list in two parts: the tree checks, whose summary block is the evidence log, thenHISTORY_CHECKS(entries). A failure in the first part does not stop the second. On a push to main it then runsrelease_evidence.py captureand uploadsci-evidence-ci-entries-<attempt>(7 days).ci.ymlno longer runs on a tag push (push: branches: ["**"]). Branch pushes and pull requests run as before.release.ymlgains theevidencejob (CI evidence at the tag,contents: read,actions: read).verify(CI at the tag) needs it, andpackageneeds both. The decision record is in the step log and in therelease-evidence-<attempt>artifact, uploaded on reuse and on refusal (!cancelled()).run-unfinishedotherwise), must be under 48 h old, and must have run on the runner image the release job resolves. The tag must be annotated;git tag -s, whichdocs/releasing.mdalready prescribes, makes an annotated tag, and v0.1.0 to v0.2.1 are all annotated.Inventory: three executions at a release commit
local-paths,agent-docs,diff-check(read the commit's tree only)entries(the unittest suite: readsGITHUB_REF's release tag, the tag listing below it, the earlier releases' blobs, and the specification at the pin)CI at the tag,tag-context: true)python-deps, an empty requirements file)entriesMain's evidence and the release's tag-context run do not overlap. A test holds the configuration's reused checks plus
HISTORY_CHECKSto exactlydev.json's checks, with no check in both. The tag-push run was equivalent to the release's call: same commit, samerefs/tags/<tag>ref (so the samerelease_being_made()), same full-history checkout, same pinned specification, started by the same push. Its result gated nothing. The release's call now covers every check it ran, which is the issue's condition for suppressing tag CI.Branch CI is not changed. A branch push (whole list, at the head commit) and a pull request run (the
--changedselection, at the synthetic merge commit) have different commits and selections, so they are not equivalent executions. A branch without a pull request has only the push run.Toolchain and inputs. There is one profile,
linux, undercompare: release-runner. The release job resolves the image (ImageOS,ImageVersion,/etc/os-release) andbash,git,cmpandpython3itself, and they must equal main's. CI runs the checks with setup-python's 3.12 (3.12.15 from the runner's tool cache, not the image's ownpython3), so the evidence job sets Python up with the same two steps asci.yml, and a test holds the two to the same text. Recorded inputs:dev.json, the configuration, the gate,ci.yml(which also holds the specification pin),requirements-dev.txt,scripts/dev/check.shandscripts/sync-agent-docs.sh. Permitted skips: none (no conditional job, no optional check skip). Trusted event:pushto main only; this repository does not dispatch CI.Fresh route (documented in
docs/releasing.md)A refusal stops the run, and nothing runs CI again on its own. The operator picks one route, once:
run-ageand for a hosted image that changed (runtime-identity). Forrun-unfinished, wait for main's run instead.Live probe (read-only API)
probe.pywalks real runs through the vendored gate'sdecide()with this configuration, using GET calls only. Three things are stubbed:ci.ymldoes not capture until this PR merges, so the probe builds the artifact with the gate's ownevidence(). It uses the real run's ids, the real job interval, and the realPASSrows of that job's log for the three tree checks.6398ed8ci.ymlresolved (id 384594220, nameCI, active); runs at the commit: 1, applicable 1; run identity; attempt-1 jobs (entries, success,ubuntu-latest, hosted group 0); re-read of the run; re-selection twicereuse(run 40395182182 attempt 1, age 0.92 h)6398ed8refuse: artifact-coverage(0 artifacts namedci-evidence-ci-entries-1: main records no evidence before this merges)88cbf53refuse: run-missing88cbf53run-triggerfor bothThe first live
captureis the first main push after merge. A capture refusal there fails main'sentriesjob visibly. The first liveverifyis the next release. Locally,capturewith this configuration, given the realcheck.sh --all --skip entriessummary of this head, records{agent-docs, diff-check, local-paths: PASS}and the seven input ids.Tests and mutants
tests/test_release_evidence.pyadds 15 tests. They run inside theentriescheck, both at main and at the tag:check_config.ci.ymlby path andname:, withpushonly and no skipped jobs.ci.yml, on its labels, with no matrix and no job-levelif.HISTORY_CHECKSequaldev.json's checks, with none in both.check.shsummary block is read by the gate's owncoverage(). The three rows are accepted. Addingentries, dropping a check, or aSKIPeach refuses.ci.yml's capture and upload run on a push to main after the checks, from the tee'd log.release.yml's gate needsadmit, hasactions: read, has nocontinue-on-error, and keeps its record on refusal.ci.ymldoes.CI at the tagneeds the gate and passestag-context: true, andpackageneeds both.ci.ymlhas no tag trigger.lib/release_evidence.pyat v0.10.0 (d00e4d48…, checked against the kit's tag), and the four vendored scripts name one kit release.The kit's 85 gate tests are not copied: the copy is held byte for byte instead.
ci.ymltest_every_required_job_is_a_job_of_ci_yml_on_its_labelstest_every_input_path_is_a_file_of_the_treeentriesadded to the reused checks…_dev_json_split_in_two,test_the_evidence_log_is_read_as_the_gate_reads_it"all"test_the_rest_is_the_kits_file_byte_for_bytetest_line_two_names_the_kit_release,test_the_vendored_scripts_name_one_kit_releaseci.ymlruns on a tag push againtest_a_tag_push_starts_no_second_run_at_the_tagci.yml's job display name driftstest_every_required_job_is_a_job_of_ci_yml_on_its_labelstest_ci_records_the_evidence_on_a_push_to_main_after_the_checksHISTORY_CHECKSemptied…_dev_json_split_in_twoci.ymlfor the whole listtest_the_tag_context_run_follows_the_gate_and_the_package_waits_for_bothtest_the_gate_runs_after_admission_and_keeps_its_recordpackageno longer needs the gatetest_the_tag_context_run_follows_…continue-on-erroron the gate steptest_the_gate_runs_after_admission_and_keeps_its_recordtest_the_gate_sets_up_the_python_ci_yml_ran_withtest_ci_records_the_evidence_on_a_push_to_main_after_the_checksAll 18 were caught, and the unmutated head passes. actionlint 1.7.12 on both workflows reports one finding, SC2034 (
attemptunused) in the unchanged draft step, which is the same on main.Measured before, expected after
Durations are from the API, job start to completion.
entriesentriesCI at the tag / entriesOn the latest main run (40395182182), the three tree checks report
0seach,entries3 s, and the job takes 13 s, mostly checkout and setup.After this PR, a release has one fewer full CI run (the tag-push run, about 14 s), and the release's call drops three checks that each take under a second. In exchange it adds the gate job, which runs before the tag-context run. The kit measured its earlier gate job at 14–15 s (v0.8.0, v0.9.0). This gate reads one more repository object and sets up Python, so a similar cost is expected, but that is not measured here.
No saving is claimed. Total runner time at a release is expected to stay about the same. The release's own critical path is expected to grow by about the gate job's duration, because the gate runs first, as the brief orders. What changes is that two tag-context runs become one, and an exact-commit decision record now exists. The first real numbers come at the next release. A refusal adds the fresh route's re-run.
Not changed / not verifiable before merge
capture, because it runs only on a push to main. The first push after merge will.verifyis first live at the next release.update-branches.ymlis not changed.Checks
Head
134100a,jpdev check --changedagainstorigin/main(merge base6398ed8;jpdev.jsonchanged, so the whole list ran), with.spec/at the pin:The handover self-check against
origin/mainpassed: the boundary pathscriptsgains only a new file.Fix round
This round answers the review record at
134100a(0 HIGH, 2 MEDIUM, 2 LOW) in one commit,15ff65e.ReusedChecksReadTheTree: thedev.jsoncommands oflocal-paths,agent-docsanddiff-checkare pinned in the test (shell toolchain, nocwd/requires_env), so editing one means classifying it again. Each command, and the repository script it runs (scripts/sync-agent-docs.sh, which runs no other), is scanned for a read of a tag, the run's ref or the history or.spec(git tag/describe/log/rev-list/for-each-ref/ls-remote/show-ref/cat-file/show/merge-base/fetch...,GITHUB_,refs/tags/,HEAD~). The scan has its own handwritten vectorsgit tag --points-at), R1b (local-paths branches on$GITHUB_REF), R1c (pin and command both moved togit describe), R1d (git loginsync-agent-docs.sh), R1e (the script runs another repository script): all caughtneeds:test_no_job_after_the_gate_can_run_past_a_refusal: the release's jobs are exactly admit, evidence, verify, package, attest-and-draft and publish. None has a job-levelif:orcontinue-on-error, and each after the gate needsevidencedirectly or through the jobs it needspackageif: ${{ always() }}), R3b (verifyif: ${{ !cancelled() }}), R3c (publishcontinue-on-error), R3d (attest-and-draftneedsadmitonly): all caughtrelease_being_madedocstring named a tag pushMutants this round: 9 of 9 caught, and the unmutated head passes.
tests/test_release_evidence.pynow has 19 tests.Head
15ff65e,jpdev check --changedagainstorigin/main, which ran the whole list becausejpdev.jsonchanged:🤖 Generated with Claude Code