Repository navigation
Release: reuse main's exact-commit CI evidence at the tag, fail closed; both Runner jobs, release contents and the pinned integration inputs kept (#175) - #182
Conversation
#175) The release's "CI at the tag" job runs the development kit's release evidence gate (scripts/dev/release_evidence.py, vendored from v0.10.0) against .github/ci-evidence.json instead of calling ci.yml again. Main CI records each job's evidence on a push to main: both Runner legs and Release contents, every check passed, the input identities, the image and the toolchain. The Runner jobs now set up the exact Go toolchain release.yml names, so the integration tests a release takes ran under it. tools/test_ci_evidence.py holds the configuration to the workflows and the manifest, and the vendored gate to the kit's. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te (#175) What the gate holds this repository's release to, the inventory of the checks before and after, the measured durations of the last two releases' CI at the tag, the fresh route after a refusal, and the tag forms the gate takes. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review record: runner PR #182 (issue #175), release evidence gate, head 14c30f4 on base 7f544dcClean room, same-vendor review, not a cross-vendor claim. No HIGH: skipped named tests, a dropped leg, changed pins and refusals all fail closed, and the old path cannot be reached.
Probes (scratch copy; private caches; read-only
|
…ripts named (#175) - The macOS profile pins ImageOS macos26: a re-pointed macos-latest refuses at capture (runner-platform) and at the release (runtime-identity) instead of being taken as the same leg. - tools/test_ci_evidence.py names manual-gate and unit-tests beside build-pinned and chain-tests, so a script that loses its must-run assertions fails it, and holds every pins-compared profile to an ImageOS. - The Release contents scripts and .goreleaser.yml are inputs too, and jpdev.json maps them to ci-evidence. - docs/releasing.md names the Linux-only must-run tests, says how to move the Go and macOS pins, and gives the rollout case of runtime-identity its own answer in the fresh route. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirmation: eef5b29One fix commit on top of 14c30f4: 4 files, +51/−13. The diff was read in full. Its public text is clean.
The
If the real value differs, the first push to main after merge fails No new finding. Nothing outside the four files changed. At review time merge: yes for eef5b29, once both Runner legs at eef5b29 are green |
scripts/dev/release_evidence.py from judgment-pack-devkit v0.11.0 (the tag grammar with SemVer prereleases, dispatch validation runs); tag.prerelease is enabled because this repository publishes prerelease tags. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
…ease in the copy test check.sh, handover_check.sh and changed_checks.py from judgment-pack-devkit v0.11.0 (line 2 only); the vendored-copy test records the v0.11.0 gate file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Confirmation: fcd15edHead fcd15ed is one commit on top of eef5b29, touching 2 files. What checks out:
What blocks the merge:
merge: no for fcd15ed. Fix finding 6 (tests red), with finding 7 in the same commit. Then a re-read of that commit is enough. Confirmation: 4b901b3Head 4b901b3 is one commit on top of fcd15ed, touching 5 files (+9/−9). Its public text is clean.
merge: yes for 4b901b3, once its four required contexts are green (finding 8 is LOW and does not block) |
Issue #175: its "Required evidence gate" list is the acceptance contract. Release admission is a trust boundary, so this PR gets one clean-room review.
The release's
CI at the tagjob no longer callsci.ymlagain. It runs the development kit's release evidence gate (scripts/dev/release_evidence.py, vendored from v0.10.0: line 2 names the release, and the rest is byte-identical to the kit'slib/release_evidence.py) against.github/ci-evidence.json. Onreuseit records the decision in the log and keeps therelease-evidence-<attempt>artifact. On any refusal it fails closed and names the code. The fresh route is manual, documented, and taken once. Every release-specific check afterCI at the tagis unchanged and runs in the same order.Inventory (reused vs fresh)
main)Runner (ubuntu-latest): local-paths, agent-docs, diff-check, go-fmt, go-vet, build-pinned, manual-gate, unit-tests, chain-tests, openapi, build, ci-evidence (new)ci.ymlcalled)PASS, no skip permitted; image, OS release, Go, Python, git and bash equal to the release job's runner; Go pinned 1.26.5Runner (macos-latest): the same listPASS; macOS ARM64, image familymacos26(fix round); Go pinned 1.26.5; image version within the family recorded and not comparedRelease contents: release-toolchain, third-party-notices, release-archives, release-programsRunnerjobs1.26.x, which resolved to 1.26.9 in run 40395171085)09206e7, earlier Runtime7a2fcfe, old Runner1cc6b29, Gateway5b49e1e, built from sourcebuild-pinnedbuild-pinnedPASS.scripts/checks/build-pinned.sh, which holds the pins, is an input whose git object id the evidence carries and the release re-derivesmanual-gate,chain-testsandunit-tests(store conformance) fail when a named test did not run and pass (--- PASS:grep, Linux-only extras on Linux)PASSof all three on both Runner legs is the proof. Their scripts are evidence inputs. ASKIP,FAILorNOT RUNrefuses (check-result)fresh_at_release)No CI leg stays fresh, for two reasons. The check results prove that the named tests ran, through each script's own assertions. And the toolchain is now exact on every leg. A leg whose evidence cannot prove that would refuse the release; it would not fall back to running CI.
Input identities (git object ids; the gate compares the captured map with its own derivation):
go.mod,go.sum,scripts/checks/{build-pinned,manual-gate,chain-tests,unit-tests,release-toolchain}.sh,scripts/dev/check.sh,release.yml, plusci.yml,dev.json, the configuration and the gate, which the gate always records. The fix round adds theRelease contentsscripts and.goreleaser.yml.What changed
.github/ci-evidence.json(schema 2) contains:CIby path and name;pushevent onmain, with no dispatch becauseci.ymlhas none;linuxprofile (release-runnerplus a Go 1.26.5 pin) and amacosprofile (pins: macOS/ARM64, Go 1.26.5).ci.yml:1.26.5, which isrelease.yml's toolchain. Before this, they ran1.26.x, so the release's integration proof was not run under the toolchain that links the archives.check.shsummary underset -o pipefail.mainonly, each job runsrelease_evidence.py captureand uploadsci-evidence-ci-<job>-<attempt>with the already-pinnedupload-artifactv7.0.1.workflow_callis kept, and nothing calls it.release.yml:verify(CI at the tag) runs onubuntu-latestwithcontents: readandactions: read. It checks outgithub.sha, sets up Go 1.26.5 (the same pinnedsetup-goasPackage, no cache), runsverify --output, and keeps the decision record underif: ${{ !cancelled() }}withif-no-files-found: error.Packagestill needsverify. Nothing else changed apart from the header comment.scripts/checks/release-toolchain.sh:release.ymlnow names Go in two jobs. The check requires everygo-versionthere to name one version, and requires this job to run it. The old script fails on the two lines (M20 below).tools/test_ci_evidence.py(newci-evidencecheck, run in the Runner legs). It holds the configuration to the realci.yml,release.ymlanddev.json:job_idand labels, with no leg missing or extra;pushonmain;pipefailplusteeon eachcheck.sh --allcall;--skip/--onlyagainstdev.json), with everydev.jsoncheck covered;go-versioninci.ymlandrelease.ymlis one version, and each profile's Go pin isgo version go<that> <os>/<arch>;ci.yml;check.shnames, and the SHA-256 of the rest is the kit's v0.10.0lib/release_evidence.py(d44af3df…).jpdev.jsonmaps the three new files toci-evidenceand addsci-evidenceto the rules for the input scripts andgo.mod/go.sum. The workflows stay unmapped, so a change to them still runs the whole list.docs/releasing.mdgains a new section, "CI at the tag", covering the conditions, the inventory, the measurements, the fresh route and what goes live first after merge. It also updates the Tag, Prepare, workflow-steps and failure-table rows.AGENTS.md/CLAUDE.mdgain one sentence.Live probe (read-only API, this branch's configuration, the vendored gate's own
decide)7f544dc, real artifacts. The tag is stubbed because none exists at the commitartifact-coverage: "0 artifacts named ci-evidence-ci-runner-ubuntu-latest-1". Main has not captured yet. Before that, repository, default branch, workflow id/path/name/state, selection (1 run at the commit, 1 applicable), run identity, result, age, and the attempt's 3 jobs with labels and hosted group all passed liveruntime-identity: "runner-ubuntu-latest: go is 'go version go1.26.9 linux/amd64', pinned 'go version go1.26.5 linux/amd64'". Today's main Runner legs are not the release toolchain, which is whyci.ymlnow pins itci-evidencerow, Go 1.26.5)reuse: run 40395171085 attempt 1, all three jobs. Every run, job, selection and re-read is live. The tag, the artifact listing and archives, and the release runner's tool resolution are stubbed604c1b7, main run 40368774529, real annotated tag, real artifactsartifact-coverage. The tag read passed live (both reads)reuse: run 40368774529 attempt 1. The tag is live before and afterc7de9af, tag stubbedrun-missing: "no push run of .github/workflows/ci.yml at c7de9af…" (1 run at the commit, 0 applicable)v0.25.0tag-commit: "v0.25.0 names commit 604c1b7…"Capture was also run locally in a fixture checkout of this head, with the CI environment set by hand. The
runner-ubuntu-latestevidence was written with 12 checks, 13 input ids andgo version go1.26.5 linux/amd64. With another job id it refusescapture-job; with apull_requestevent it refusescapture-event. The kit's 85 gate tests pass on the vendored copy (RELEASE_EVIDENCE_HELPER).Mutants (scratch copies, one safeguard broken each; the unmutated tree passes)
Runner (macos-15)test_every_job_and_leg_of_ci_is_required_by_its_display_name--job runner(no matrix value)test_each_leg_records_its_evidence_under_the_name_the_gate_reads(both legs)set -o pipefailremovedrelease-contents)chain-testsdropped from the macOS legtest_each_leg_lists_the_checks_its_push_run_reports,test_the_pins_and_the_must_run_tests_are_in_every_runner_legbuild-pinned.shnot an inputtest_the_pins_and_the_must_run_tests_are_in_every_runner_legtest_every_input_is_a_tracked_file1.26.xtest_every_leg_ran_the_release_toolchaingo1.26.9ci.ymlagaintest_the_release_takes_the_evidence_and_does_not_call_ci_again!cancelled()MAX_AGE_HOURS = 72)test_the_gate_is_the_kits_of_the_release_line_2_namestest_each_leg_records_its_evidence_under_the_name_the_gate_readsPackageno longer needsverifytest_the_release_takes_the_evidence_and_does_not_call_ci_againrelease-archivesdropped from Release contentstest_each_leg_lists_the_checks_its_push_run_reports1.26.9test_every_leg_ran_the_release_toolchainrelease-toolchain.shrelease-toolchain.shon thisrelease.ymlMeasured before; the gate not yet
CI at the tag(job start to completion)ci.ymlcalledci.ymlcalledMain CI keeps running every check: run 40368774529 took 76 s, 553 s and 513 s, and run 40395171085 took 78 s, 598 s and 740 s. In the kit's own release, the gate job took 14–15 s for two jobs' evidence. This job also sets up Go (the
Release contentssetup-go step for 1.26.5 took 11 s with its cache restore) and reads a third artifact. Its cost is first measured at the next release; no saving is claimed until then. A refusal adds the fresh route: a full main CI re-run (about 10–12 min) plus a re-run of the release's failed jobs.Decisions for the review
tools/test_ci_evidence.pykeepsci.yml,release.ymland the pins together.vX.Y.Zas an annotated tag only, so a prerelease tag refuses (tag-syntax) with no fresh route, and a lightweight tag refuses (tag-annotated).Admit the tagis unchanged;docs/releasing.mdsays so. The only prerelease so far wasv0.1.0-rc.1. v0.24.0 and v0.25.0 were annotated.Runner (ubuntu-latest)andRelease contentson different image versions, one of them refusesruntime-identity. The answer is the fresh route.actions/checkout@v4andactions/setup-go@v5references inci.ymlremain tags, as before. The gate binds what they set up (exact Go, and the image on Linux), not the action code.Not verifiable before merge
main, so this PR's CI does not exercise it. The first main push after merge does. A capture refusal there fails that main job visibly, for example if a hosted macOS image did not setImageOS/ImageVersion, which the gate requires.verifyis first live at the next release. The probe above walked every read except the artifacts, which do not exist yet.Checks
Head
14c30f4.jpdev check --changedselects the whole list, becausedev.jsonandjpdev.jsonchanged. Per the brief, the long Go steps, which this PR does not touch, were skipped locally and run in CI:--skip go-fmt,go-vet,build-pinned,manual-gate,unit-tests,chain-tests,build,third-party-notices,release-archives.Fix round
This round answers the review record at
14c30f4: 0 HIGH, 2 MEDIUM, 3 LOW. It is one commit,eef5b29.profiles.macos.platform.ImageOSis now"macos26", the family ofmacos-26-arm64, whichmacos-latestruns today. The first capture after merge records the real value. A different value fails that main job by name (ImageOS is <value>, not macos26), and the pin then moves in one line (docs, "Moving a pin")macos27withrunner-platform("ImageOS is macos27, not macos26"). Live walk of run 40395171085 with the macOS evidence from amacos27image:runtime-identity("runner-macos-latest: ImageOS is macos27, not macos26"). Withmacos26:reusetest_every_leg_ran_the_release_toolchainfails, because everypinsprofile must name an ImageOSmanual-gateandunit-testsare asserted by name besidebuild-pinnedandchain-testsmanual-gate.shwithout its loop) and N02 (unit-tests.shwithout its loop):test_the_pins_and_the_must_run_tests_are_in_every_runner_legfails. O6 (one name dropped from a loop) stays a review matterTestManualGovernedActualRuntimeHasNoAuthorityFD, andTestExecutorOwnedRealRuntimeSurvivesOwnerDeathwith/closeand/kill. They are required on Linux and may skip on macOS by designgo-versioninci.ymlandrelease.ymland bothgopins move in one pull request;tools/test_ci_evidence.pyandrelease-toolchainfail until they agree, and CI stays on that patch. macOS family: main turns red at capture withrunner-platform; move the pin to the value named. Fresh route step 3: aruntime-identityduring a hosted-image rollout means waiting and taking the route again, not a new versioninputsnow also listsscripts/checks/release-programs.sh,tools/third_party_notices.py,tools/release_archives.py,tools/test_release_archives.pyand.goreleaser.yml, andjpdev.jsonmaps them toci-evidencetest_every_input_is_a_tracked_filefailsM01–M18 were re-run on the fix and all are still caught. The no-op control passes.
Head
eef5b29,jpdev check --changed, with the same skip set as above:🤖 Generated with Claude Code