Skip to content

Release: reuse main's exact-commit CI evidence at the tag, fail closed; both Runner jobs, release contents and the pinned integration inputs kept (#175) - #182

Merged
kikashy merged 5 commits into
mainfrom
release-evidence-175
Oct 26, 2026
Merged

kikashy merged 5 commits into
mainfrom
release-evidence-175

Conversation

@kikashy

@kikashy kikashy commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Issue #175: its "Required evidence gate" list is the acceptance contract. Release admission is a trust boundary, so this PR gets one clean-room review.

The release's CI at the tag job no longer calls ci.yml again. It runs the development kit's release evidence gate (scripts/dev/release_evidence.py, vendored from v0.10.0: line 2 names the release, and the rest is byte-identical to the kit's lib/release_evidence.py) against .github/ci-evidence.json. On reuse it records the decision in the log and keeps the release-evidence-<attempt> artifact. On any refusal it fails closed and names the code. The fresh route is manual, documented, and taken once. Every release-specific check after CI at the tag is unchanged and runs in the same order.

Inventory (reused vs fresh)

Check Main CI (push to main) Release before (v0.24.0, v0.25.0) Release with the gate
Runner (ubuntu-latest): local-paths, agent-docs, diff-check, go-fmt, go-vet, build-pinned, manual-gate, unit-tests, chain-tests, openapi, build, ci-evidence (new) run run again (ci.yml called) reused: every check PASS, no skip permitted; image, OS release, Go, Python, git and bash equal to the release job's runner; Go pinned 1.26.5
Runner (macos-latest): the same list run run again reused: every check PASS; macOS ARM64, image family macos26 (fix round); Go pinned 1.26.5; image version within the family recorded and not compared
Release contents: release-toolchain, third-party-notices, release-archives, release-programs run, Go 1.26.5 run again reused: same as the Linux runner
Go in the Runner jobs 1.26.5 (was 1.26.x, which resolved to 1.26.9 in run 40395171085) 1.26.x 1.26.5, compared
Pinned integration inputs: Runtime 09206e7, earlier Runtime 7a2fcfe, old Runner 1cc6b29, Gateway 5b49e1e, built from source build-pinned again build-pinned PASS. scripts/checks/build-pinned.sh, which holds the pins, is an input whose git object id the evidence carries and the release re-derives
Named real-companion tests manual-gate, chain-tests and unit-tests (store conformance) fail when a named test did not run and pass (--- PASS: grep, Linux-only extras on Linux) again PASS of all three on both Runner legs is the proof. Their scripts are evidence inputs. A SKIP, FAIL or NOT RUN refuses (check-result)
Tag, notes, reviewer rule, notices with the release toolchain, packaging, checksums, archive contents, native smoke (3 targets), attestation, draft, publication not run run fresh, unchanged (listed in fresh_at_release)

No CI leg stays fresh, for two reasons. The check results prove that the named tests ran, through each script's own assertions. And the toolchain is now exact on every leg. A leg whose evidence cannot prove that would refuse the release; it would not fall back to running CI.

Input identities (git object ids; the gate compares the captured map with its own derivation): go.mod, go.sum, scripts/checks/{build-pinned,manual-gate,chain-tests,unit-tests,release-toolchain}.sh, scripts/dev/check.sh, release.yml, plus ci.yml, dev.json, the configuration and the gate, which the gate always records. The fix round adds the Release contents scripts and .goreleaser.yml.

What changed

  • .github/ci-evidence.json (schema 2) contains:
    • workflow CI by path and name;
    • the push event on main, with no dispatch because ci.yml has none;
    • three required jobs/legs with their labels and check lists;
    • no skipped jobs or optional skips;
    • the inputs above;
    • a linux profile (release-runner plus a Go 1.26.5 pin) and a macos profile (pins: macOS/ARM64, Go 1.26.5).
  • ci.yml:
    • Both Runner legs set up Go 1.26.5, which is release.yml's toolchain. Before this, they ran 1.26.x, so the release's integration proof was not run under the toolchain that links the archives.
    • Each job tees its check.sh summary under set -o pipefail.
    • On a push to main only, each job runs release_evidence.py capture and uploads ci-evidence-ci-<job>-<attempt> with the already-pinned upload-artifact v7.0.1.
    • Pull request CI is otherwise unchanged. workflow_call is kept, and nothing calls it.
  • release.yml: verify (CI at the tag) runs on ubuntu-latest with contents: read and actions: read. It checks out github.sha, sets up Go 1.26.5 (the same pinned setup-go as Package, no cache), runs verify --output, and keeps the decision record under if: ${{ !cancelled() }} with if-no-files-found: error. Package still needs verify. Nothing else changed apart from the header comment.
  • scripts/checks/release-toolchain.sh: release.yml now names Go in two jobs. The check requires every go-version there to name one version, and requires this job to run it. The old script fails on the two lines (M20 below).
  • tools/test_ci_evidence.py (new ci-evidence check, run in the Runner legs). It holds the configuration to the real ci.yml, release.yml and dev.json:
    • every job and matrix leg by display name, job_id and labels, with no leg missing or extra;
    • push on main;
    • each leg's capture command and artifact name exactly as the gate reads them, with the main-push condition, a SHA-pinned upload and pipefail plus tee on each check.sh --all call;
    • each leg's check list equal to what its push run selects (--skip / --only against dev.json), with every dev.json check covered;
    • every step script that fetches a pinned companion or asserts must-run tests (derived from the scripts' content) is an input and runs in both Runner legs;
    • every input is a tracked file;
    • every go-version in ci.yml and release.yml is one version, and each profile's Go pin is go version go<that> <os>/<arch>;
    • the release takes the evidence and does not call ci.yml;
    • the vendored gate equals the kit's: line 2 names the release check.sh names, and the SHA-256 of the rest is the kit's v0.10.0 lib/release_evidence.py (d44af3df…).
  • jpdev.json maps the three new files to ci-evidence and adds ci-evidence to the rules for the input scripts and go.mod/go.sum. The workflows stay unmapped, so a change to them still runs the whole list.
  • docs/releasing.md gains a new section, "CI at the tag", covering the conditions, the inventory, the measurements, the fresh route and what goes live first after merge. It also updates the Tag, Prepare, workflow-steps and failure-table rows. AGENTS.md/CLAUDE.md gain one sentence.

Live probe (read-only API, this branch's configuration, the vendored gate's own decide)

Walk Decision
1a. The latest main CI run, 40395171085 at 7f544dc, real artifacts. The tag is stubbed because none exists at the commit refuse artifact-coverage: "0 artifacts named ci-evidence-ci-runner-ubuntu-latest-1". Main has not captured yet. Before that, repository, default branch, workflow id/path/name/state, selection (1 run at the commit, 1 applicable), run identity, result, age, and the attempt's 3 jobs with labels and hosted group all passed live
1b. The same run, with the evidence it would have recorded under its own toolchain (coverage parsed from the real job logs, Go from the setup-go log) refuse runtime-identity: "runner-ubuntu-latest: go is 'go version go1.26.9 linux/amd64', pinned 'go version go1.26.5 linux/amd64'". Today's main Runner legs are not the release toolchain, which is why ci.yml now pins it
1c. The same run, with evidence as this configuration records it (the real summary rows plus the new ci-evidence row, Go 1.26.5) reuse: run 40395171085 attempt 1, all three jobs. Every run, job, selection and re-read is live. The tag, the artifact listing and archives, and the release runner's tool resolution are stubbed
2a. v0.25.0 commit 604c1b7, main run 40368774529, real annotated tag, real artifacts refuse artifact-coverage. The tag read passed live (both reads)
2b. The same with evidence as 1c reuse: run 40368774529 attempt 1. The tag is live before and after
3a. Pull request run 40390687338 at its head c7de9af, tag stubbed refuse run-missing: "no push run of .github/workflows/ci.yml at c7de9af…" (1 run at the commit, 0 applicable)
3b. The same head, real tag v0.25.0 refuse tag-commit: "v0.25.0 names commit 604c1b7…"

Capture was also run locally in a fixture checkout of this head, with the CI environment set by hand. The runner-ubuntu-latest evidence was written with 12 checks, 13 input ids and go version go1.26.5 linux/amd64. With another job id it refuses capture-job; with a pull_request event it refuses capture-event. The kit's 85 gate tests pass on the vendored copy (RELEASE_EVIDENCE_HELPER).

Mutants (scratch copies, one safeguard broken each; the unmutated tree passes)

# Broken Caught by
M01 config leg renamed Runner (macos-15) test_every_job_and_leg_of_ci_is_required_by_its_display_name
M02 macOS leg dropped from the matrix the same
M03 capture --job runner (no matrix value) test_each_leg_records_its_evidence_under_the_name_the_gate_reads (both legs)
M04 set -o pipefail removed the same
M05 capture on any main event the same (release-contents)
M06 chain-tests dropped from the macOS leg test_each_leg_lists_the_checks_its_push_run_reports, test_the_pins_and_the_must_run_tests_are_in_every_runner_leg
M07 build-pinned.sh not an input test_the_pins_and_the_must_run_tests_are_in_every_runner_leg
M08 an input that does not exist test_every_input_is_a_tracked_file
M09 Runner Go back to 1.26.x test_every_leg_ran_the_release_toolchain
M10 macOS pin go1.26.9 the same
M11 release calls ci.yml again test_the_release_takes_the_evidence_and_does_not_call_ci_again
M12 decision record upload without !cancelled() the same
M13 one byte of the vendored gate changed (MAX_AGE_HOURS = 72) test_the_gate_is_the_kits_of_the_release_line_2_names
M14 line 2 names v0.9.0 the same
M15 push call not teed test_each_leg_records_its_evidence_under_the_name_the_gate_reads
M16 Package no longer needs verify test_the_release_takes_the_evidence_and_does_not_call_ci_again
M17 release-archives dropped from Release contents test_each_leg_lists_the_checks_its_push_run_reports
M18 the verify job's Go 1.26.9 test_every_leg_ran_the_release_toolchain
M19 the same, through release-toolchain.sh exits 1: "release.yml names Go 1.26.5 1.26.9; its jobs must name one version"
M20 the old release-toolchain.sh on this release.yml exits 1 (two lines): why the script changed

Measured before; the gate not yet

Release CI at the tag (job start to completion) Admit end to Package start
v0.24.0 (run 39287501642), ci.yml called Release contents 92 s, Runner (ubuntu-latest) 581 s, Runner (macos-latest) 620 s 651 s
v0.25.0 (run 40371047612), ci.yml called 74 s, 589 s, 488 s 594 s

Main CI keeps running every check: run 40368774529 took 76 s, 553 s and 513 s, and run 40395171085 took 78 s, 598 s and 740 s. In the kit's own release, the gate job took 14–15 s for two jobs' evidence. This job also sets up Go (the Release contents setup-go step for 1.26.5 took 11 s with its cache restore) and reads a third artifact. Its cost is first measured at the next release; no saving is claimed until then. A refusal adds the fresh route: a full main CI re-run (about 10–12 min) plus a re-run of the release's failed jobs.

Decisions for the review

  • Go 1.26.5 in the Runner jobs. This is the issue's "exact release toolchain". PR and main CI now test with 1.26.5, the patch the archives are linked with, not the hosted toolcache's newest 1.26 (1.26.9 today). Moving the release toolchain is a separate one-line decision; tools/test_ci_evidence.py keeps ci.yml, release.yml and the pins together.
  • Prerelease and lightweight tags. The kit's gate takes vX.Y.Z as an annotated tag only, so a prerelease tag refuses (tag-syntax) with no fresh route, and a lightweight tag refuses (tag-annotated). Admit the tag is unchanged; docs/releasing.md says so. The only prerelease so far was v0.1.0-rc.1. v0.24.0 and v0.25.0 were annotated.
  • macOS image family: pinned in the fix round (below).
  • One Linux profile for two jobs. If a hosted-image rollout puts Runner (ubuntu-latest) and Release contents on different image versions, one of them refuses runtime-identity. The answer is the fresh route.
  • The actions/checkout@v4 and actions/setup-go@v5 references in ci.yml remain tags, as before. The gate binds what they set up (exact Go, and the image on Linux), not the action code.

Not verifiable before merge

  • Capture runs only on a push to main, so this PR's CI does not exercise it. The first main push after merge does. A capture refusal there fails that main job visibly, for example if a hosted macOS image did not set ImageOS/ImageVersion, which the gate requires.
  • verify is first live at the next release. The probe above walked every read except the artifacts, which do not exist yet.

Checks

Head 14c30f4. jpdev check --changed selects the whole list, because dev.json and jpdev.json changed. Per the brief, the long Go steps, which this PR does not touch, were skipped locally and run in CI: --skip go-fmt,go-vet,build-pinned,manual-gate,unit-tests,chain-tests,build,third-party-notices,release-archives.

== check summary (dev.json) ==
PASS     local-paths              0s
PASS     agent-docs               0s
PASS     diff-check               0s
PASS     openapi                  0s
PASS     ci-evidence              0s
    Ran 10 tests in 0.060s
    OK
PASS     release-toolchain        0s
PASS     release-programs         0s
== 7 passed, 0 failed, 0 skipped, 0 not run ==

Fix round

This round answers the review record at 14c30f4: 0 HIGH, 2 MEDIUM, 3 LOW. It is one commit, eef5b29.

Finding Fix Probe Safeguard removed in scratch
MEDIUM 1: a change to the macOS image passed profiles.macos.platform.ImageOS is now "macos26", the family of macos-26-arm64, which macos-latest runs today. The first capture after merge records the real value. A different value fails that main job by name (ImageOS is <value>, not macos26), and the pin then moves in one line (docs, "Moving a pin") Capture's runtime check refuses ImageOS macos27 with runner-platform ("ImageOS is macos27, not macos26"). Live walk of run 40395171085 with the macOS evidence from a macos27 image: runtime-identity ("runner-macos-latest: ImageOS is macos27, not macos26"). With macos26: reuse N03, ImageOS removed: test_every_leg_ran_the_release_toolchain fails, because every pins profile must name an ImageOS
MEDIUM 2: a script that lost its must-run loop survived manual-gate and unit-tests are asserted by name beside build-pinned and chain-tests N01 (manual-gate.sh without its loop) and N02 (unit-tests.sh without its loop): test_the_pins_and_the_must_run_tests_are_in_every_runner_leg fails. O6 (one name dropped from a loop) stays a review matter
LOW 3: the docs did not say which named tests are Linux-only "CI at the tag" names them: TestManualGovernedActualRuntimeHasNoAuthorityFD, and TestExecutorOwnedRealRuntimeSurvivesOwnerDeath with /close and /kill. They are required on Linux and may skip on macOS by design
LOW 4: the docs did not say how to move the toolchain, and gave the wrong remedy for an image rollout New section "Moving a pin". Go: every go-version in ci.yml and release.yml and both go pins move in one pull request; tools/test_ci_evidence.py and release-toolchain fail until they agree, and CI stays on that patch. macOS family: main turns red at capture with runner-platform; move the pin to the value named. Fresh route step 3: a runtime-identity during a hosted-image rollout means waiting and taking the route again, not a new version
LOW 5: the Release contents scripts were not inputs inputs now also lists scripts/checks/release-programs.sh, tools/third_party_notices.py, tools/release_archives.py, tools/test_release_archives.py and .goreleaser.yml, and jpdev.json maps them to ci-evidence N04, a misspelt input: test_every_input_is_a_tracked_file fails

M01–M18 were re-run on the fix and all are still caught. The no-op control passes.

Head eef5b29, jpdev check --changed, with the same skip set as above:

PASS     local-paths              0s
PASS     agent-docs               0s
PASS     diff-check               0s
PASS     openapi                  0s
PASS     ci-evidence              0s
    Ran 10 tests in 0.046s
    OK
PASS     release-toolchain        0s
PASS     release-programs         0s
== 7 passed, 0 failed, 0 skipped, 0 not run ==

🤖 Generated with Claude Code

kikashy and others added 2 commits October 26, 2026 17:18
#175)

The release's "CI at the tag" job runs the development kit's release
evidence gate (scripts/dev/release_evidence.py, vendored from v0.10.0)
against .github/ci-evidence.json instead of calling ci.yml again. Main
CI records each job's evidence on a push to main: both Runner legs and
Release contents, every check passed, the input identities, the image
and the toolchain. The Runner jobs now set up the exact Go toolchain
release.yml names, so the integration tests a release takes ran under
it. tools/test_ci_evidence.py holds the configuration to the workflows
and the manifest, and the vendored gate to the kit's.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te (#175)

What the gate holds this repository's release to, the inventory of the
checks before and after, the measured durations of the last two
releases' CI at the tag, the fresh route after a refusal, and the tag
forms the gate takes.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Review record: runner PR #182 (issue #175), release evidence gate, head 14c30f4 on base 7f544dc

Clean room, same-vendor review, not a cross-vendor claim. No HIGH: skipped named tests, a dropped leg, changed pins and refusals all fail closed, and the old path cannot be reached.
2 MEDIUM: a change to the macOS image passes without a refusal, and a mutant that removes a script's must-run loop survives. 3 LOW. PR CI is green on all four required contexts.

# Finding Sev Evidence What to do
1 A change to the macOS image passes without a refusal. The macos profile compares only RUNNER_OS, RUNNER_ARCH and the Go string. ImageOS and ImageVersion are recorded but not compared, so a re-pointed macos-latest within 48 h is accepted MEDIUM S7: evidence with ImageOS macos27 and a new ImageVersion gives reuse. The kit allows ImageOS under platform Pin ImageOS in profiles.macos.platform to the value the first capture records. A family change then refuses: capture runner-platform turns main red, and verify gives runtime-identity. State in the docs that drift within the family is accepted for up to 48 h
2 Surviving mutant. If manual-gate.sh or unit-tests.sh loses its must-run loop, test_ci_evidence.py still passes, because it derives the must-run scripts from their content and asserts only build-pinned and chain-tests by name. The gate would then take a PASS that no longer proves the named tests ran. This is not a regression: the old path ran the same scripts MEDIUM O4 and O5: OK (survived). O6, one name dropped from a loop, also survives; only review can guard that Assert manual-gate and unit-tests by name next to chain-tests (one line)
3 The docs say "a PASS of each, on both Runner legs, is the proof that those tests ran". Linux requires TestManualGovernedActualRuntimeHasNoAuthorityFD and TestExecutorOwnedRealRuntimeSurvivesOwnerDeath (with /close and /kill). They may skip on macOS by design LOW Stubbed go: those tests SKIP on Darwin and the script exits 0. On Linux it exits 1 Name the Linux-only tests in "CI at the tag"
4 releasing.md leaves out two maintenance consequences. Moving the toolchain is one change across 3 setup-go steps in ci.yml, 2 in release.yml and 2 pins, and CI now stays on 1.26.5 instead of the newest 1.26 patch. In fresh-route step 3, "release a new version" is the wrong remedy for a repeated runtime-identity during a hosted-image rollout LOW AGENTS.md and CLAUDE.md state the coupling, but releasing.md does not One sentence each: how to move the toolchain, and for a rollout, wait and repeat the route
5 inputs leaves out the Release contents scripts (tools/release_archives.py, test_release_archives.py, third_party_notices.py, release-programs.sh, .goreleaser.yml). This is not a binding gap. Every input id is git rev-parse <commit>:<path>, which depends only on the commit the gate already binds (run-sha, checkout-sha), and none of these files fetches anything at run time LOW Only build-pinned.sh fetches, by commit SHA, and it is an input. The goreleaser version is in release.yml, which is an input Optional: list them for a complete record

Probes (scratch copy; private caches; read-only gh api GETs)

  • Integration proof (1). The evidence holds each check's status from the check.sh summary, not the PASS lines. The scripts exit 1 unless each named test printed --- PASS:. A stubbed go printed a named test as SKIP or dropped a subtest's PASS line, and manual-gate and chain-tests exited 1 ("did not run and pass"). The real check.sh then printed FAIL manual-gate, and capture on that log refused with check-result (an all-PASS log captured 12 checks, 13 inputs, go1.26.5). The env-unset skips cannot fire in CI because the scripts always set every companion variable. A missing binary fails the test; it does not skip.
  • Verify over a fake API with the PR's configuration. Each scenario ran the vendored decide().
    • S0 control: reuse.
    • Evidence and coverage: macOS manual-gate SKIP gives check-result; chain-tests row missing gives check-coverage; Linux FAIL or NOT RUN gives check-result.
    • Missing or bad jobs: macOS artifact missing gives artifact-coverage; macOS job missing gives job-coverage; skipped gives job-skipped; failed gives job-result; self-hosted group gives job-runner.
    • Changed inputs: Linux image differs gives runtime-identity; macOS Go 1.26.9 gives runtime-identity; build-pinned.sh id changed or dropped gives input-identity.
    • Run selection and age: a newer failed push gives run-result; an unfinished schedule run gives run-unfinished; 49 h gives run-age; a PR-only run gives run-missing.
    • Tags: lightweight gives tag-annotated; prerelease gives tag-syntax.
    • Image: S7, the macOS image change, gives reuse (finding 1).
  • Live walk (6). The vendored gate ran against the real API with only the tag stubbed. All results are as expected before the first capture:
    • Main run 40395171085 at 7f544dc: artifact-coverage (1 run, 1 applicable, everything before it passed live).
    • v0.25.0 commit with the real annotated tag: the tag read passed, then artifact-coverage.
    • PR run at c7de9af and this PR's head: run-missing (0 applicable).
    • Real v0.25.0 tag against 7f544dc: tag-commit.
  • Toolchain (4). release-toolchain.sh passes on the control. It refuses with a 1.26.9 verify job (M19: "must name one version"). The old script refuses the two-line release.yml (M20).

Mutants: 9 of the builder's re-run, all caught; 5 of mine

  • Builder's: M02, M06, M07, M09, M11, M12, M13, M16 (test_ci_evidence.py) and M19, M20.
  • O2, macOS leg dropped from the config's required list: the test fails, and the gate gives job-coverage (unexpected Runner (macos-latest)).
  • O3, an optional_skips of manual-gate on macOS: the test fails. Without that test the gate would accept the SKIP.
  • Companion PASS line removed from the evidence (S1, S2), or from the go output: refused, or the script exits 1.
  • O4, O5, O6: survived (finding 2).

Verified unchanged or correct

  • Vendored gate: byte-identical to kit v0.10.0 lib/release_evidence.py after line 2 (sha256 d44af3df…).
  • Fail closed (3): verify exits 1 on any refusal, and an uncaught exception also fails the job. package needs verify. smoke → attest-and-draft → publish follow the needs chain with no if: override. The record upload runs under !cancelled() with if-no-files-found: error.
  • Old path: release.yml never names ci.yml (M11 caught). capture refuses a workflow_call (GITHUB_WORKFLOW_REF), and verify selects only push runs of ci.yml's id. A tag on a commit before the merge runs that commit's own release.yml, which runs CI fresh: the stricter path, not a bypass.
  • Branch protection: required contexts Runner (ubuntu-latest), Runner (macos-latest), Release contents and DCO, strict, admins enforced. Job names unchanged.
  • Everything after "CI at the tag" (package, notices, archives, smoke, tag re-checks, attestation, draft, publication) is byte-for-byte as before.
  • Doc durations match the API: v0.25.0 at 74, 589 and 488 s, and 594 s from admit to package.
  • The docs state the fresh route, the 48 h precondition, and the prerelease and lightweight-tag refusal. Public text is clean.
  • macOS ImageOS/ImageVersion: the hosted macOS image build exports both. If they were absent, capture refuses with runner-identity (probed), which is loud, not silent.
  • ci.yml's checkout@v4 and setup-go@v5 tags were already there before this PR; the gate binds the Go they install.

merge: yes (no HIGH). Disposition per the kit policy: both MEDIUMs and the cheap LOWs are fixed in this PR in one round by the builder and confirmed by this reviewer.

…ripts named (#175)

- The macOS profile pins ImageOS macos26: a re-pointed macos-latest
  refuses at capture (runner-platform) and at the release
  (runtime-identity) instead of being taken as the same leg.
- tools/test_ci_evidence.py names manual-gate and unit-tests beside
  build-pinned and chain-tests, so a script that loses its must-run
  assertions fails it, and holds every pins-compared profile to an
  ImageOS.
- The Release contents scripts and .goreleaser.yml are inputs too, and
  jpdev.json maps them to ci-evidence.
- docs/releasing.md names the Linux-only must-run tests, says how to move
  the Go and macOS pins, and gives the rollout case of runtime-identity
  its own answer in the fresh route.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Confirmation: eef5b29

One fix commit on top of 14c30f4: 4 files, +51/−13. The diff was read in full. Its public text is clean.

# Disposition Evidence at eef5b29
1 Fixed profiles.macos.platform.ImageOS is macos26. In the fake-API decide probe, macos26 at today's image version gives reuse, and macos26 at a newer version also gives reuse (stated in the docs as accepted for 48 h). macos27 refuses with runtime-identity ("ImageOS is macos27, not macos26"). Capture on a macos27 runner refuses with runner-platform. Removing the pin fails test_every_leg_ran_the_release_toolchain
2 Fixed O4 and O5 now fail test_the_pins_and_the_must_run_tests_are_in_every_runner_leg. O6, one name dropped from a loop, still survives as stated: only review can guard it, and it is not a finding
3 Fixed releasing.md names the Linux-only tests: TestManualGovernedActualRuntimeHasNoAuthorityFD, and TestExecutorOwnedRealRuntimeSurvivesOwnerDeath with /close and /kill
4 Fixed The new "Moving a pin" section lists every Go place, says CI stays on the exact patch, and covers the macOS family move. Fresh-route step 3 now says to wait out a hosted-image rollout instead of releasing a new version
5 Fixed The Release contents scripts and .goreleaser.yml are in inputs, and jpdev.json maps them to ci-evidence. Dropping one is not tested; it is audit-only, as before

The ImageOS value was derived from source, not observed. No log we can read prints the variable. The evidence that the value is right:

  • The runner-images README maps macos-latest to macOS 26 Arm64.
  • That image's template, macOS-26.arm64.anka.pkr.hcl, defaults image_os to macos26 and passes it as IMAGE_OS to configure-preimagedata.sh. That script exports ImageOS=$IMAGE_OS.
  • The same script writes the image data that this PR's own macOS job printed: "Image: macos-26-arm64, Version: 20261021.0445.8".

If the real value differs, the first push to main after merge fails Runner (macos-latest) at capture with runner-platform, naming the real value. That failure is loud: pull request CI stays green, and a release refuses until it is fixed. The fix is one line, as docs/releasing.md "Moving a pin" says.

No new finding. Nothing outside the four files changed. At review time Release contents and DCO were green, and both Runner legs at eef5b29 were still running.

merge: yes for eef5b29, once both Runner legs at eef5b29 are green

kikashy and others added 2 commits October 26, 2026 18:06
scripts/dev/release_evidence.py from judgment-pack-devkit v0.11.0 (the
tag grammar with SemVer prereleases, dispatch validation runs); tag.prerelease is
enabled because this repository publishes prerelease tags.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
…ease in the copy test

check.sh, handover_check.sh and changed_checks.py from judgment-pack-devkit
v0.11.0 (line 2 only); the vendored-copy test records the v0.11.0 gate
file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Confirmation: fcd15ed

Head fcd15ed is one commit on top of eef5b29, touching 2 files.

What checks out:

  • The vendored gate. It is byte-identical after line 2 to the kit's lib/release_evidence.py at v0.11.0 (sha256 f272b7a4…).
  • The configuration. Compared as JSON, the only change is the new "tag": {"prerelease": true}. Everything else is reformatting.
  • Tag grammar. The gate's prerelease pattern is the same grammar as the regex in Admit the tag, and the v* trigger covers it. Both accept v9.9.9, v9.9.9-rc.1, v0.1.0-rc.1 and v9.9.9-alpha.0.x-y. Both refuse +build, -rc.01, v09.9.9 and a trailing -. A lightweight prerelease tag refuses with tag-annotated. With prerelease: false, -rc.1 refuses with tag-syntax. The draft and publish jobs already handle prereleases, so nothing earlier in the workflow blocks one now.
  • Dispatch mode. release.yml has no workflow_dispatch trigger and the configuration names no dispatch, so every run is a publication. The new validated decision cannot be reached.
  • A plain vX.Y.Z walk is unchanged. It reuses on the fake API. The read-only live walks give the same results as before: artifact-coverage for main and for v0.25.0 with its real tag, and run-missing for a PR run.

What blocks the merge:

# Finding Sev What to do
6 The repo's tests fail. test_the_gate_is_the_kits_of_the_release_line_2_names requires scripts/dev/check.sh line 2 to name the same kit release as the gate, and check.sh is still at v0.10.0. The kit's templates/check.sh changed in v0.11.0. KIT_GATE has no v0.11.0 digest. So ci-evidence will fail both Runner legs, which are required contexts blocks merge Re-vendor check.sh (and, for one kit release, changed_checks.py and handover_check.sh, which are unchanged in v0.11.0 apart from line 2) at v0.11.0. Add 'v0.11.0': 'f272b7a43272bac31bb9b0c37a583d9c40d376972b04c812e576bd82c1ef18e3' to KIT_GATE
7 docs/releasing.md is now wrong about prereleases. Lines 112–117 say the gate refuses a prerelease tag and that none passes "until the kit's gate takes prereleases"; line 293 says "a prerelease ... tag cannot pass". The gate now takes annotated SemVer prereleases LOW Say that CI at the tag takes an annotated vX.Y.Z or vX.Y.Z-<prerelease>, and still refuses a lightweight tag

merge: no for fcd15ed. Fix finding 6 (tests red), with finding 7 in the same commit. Then a re-read of that commit is enough.

Confirmation: 4b901b3

Head 4b901b3 is one commit on top of fcd15ed, touching 5 files (+9/−9). Its public text is clean.

  • Finding 6: fixed.
    • tools/test_ci_evidence.py passes: 10 tests, OK.
    • KIT_GATE gains v0.11.0 (f272b7a4…).
    • scripts/dev/check.sh is byte-identical to the kit's v0.11.0 templates/check.sh. The kit's only change there in v0.11.0 is its own line 2.
    • changed_checks.py, handover_check.sh and release_evidence.py each equal the kit's v0.11.0 lib/ file once line 2 is set aside.
  • Finding 7: fixed. Line 293 now names only the lightweight tag. Lines 112–116 say the gate accepts the admit grammar through tag.prerelease: true, and that a prerelease is published as one, never marked latest.
  • New LOW (8): two errors in the new sentence at line 112.
    • It says Admit the tag "requires an annotated tag". It does not: its ls-remote check accepts a lightweight tag. The gate in CI at the tag refuses one (tag-annotated).
    • It drops the explicit refusal of build metadata (+...). SemVer allows build metadata, so "as SemVer 2.0.0 writes them" alone does not say it is refused.
    • Do: give the annotated-tag requirement to CI at the tag, and restore "refuses build metadata". This can follow in a later commit.
  • At review time DCO was green. Both Runner legs and Release contents were still running at 4b901b3.

merge: yes for 4b901b3, once its four required contexts are green (finding 8 is LOW and does not block)

@kikashy
kikashy merged commit 530afd5 into main Oct 26, 2026
5 checks passed
@kikashy
kikashy deleted the release-evidence-175 branch October 26, 2026 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant