fix(pr): exit status must agree with the failure the command just printed - #922
Merged
Conversation
…nted run_pr_create collected per-repo failures, printed "Failed to create N PR(s):" with every one of them, and returned Ok(()). Anything gating on the exit status -- a shell script, CI, an agent deciding whether to continue -- read that as success while the command was saying the opposite on stdout. The predicate was never missing. Four lines above the return, the --json branch already computes `success: !created.is_empty() && failed.is_empty()` and serializes it. The truth was computed and then discarded, which is the same shape as the merge base in #917: the right value in hand, one line from where it was needed, dropped. Scope is deliberately the ruled half of grip#886 and no wider. A run that creates nothing and fails nothing still exits 0; that is the zero-match no-op question in #804/#836/#839, and folding it in would change the exit status of runs no witness here covers. Witness: tests/test_pr_create_exit_status.rs. The failure case asserts is_err and carries a control that the creation was actually ATTEMPTED, so a command that skipped the repo entirely cannot satisfy it for the wrong reason. The second test is the discriminating control -- a run where every PR is created must still return Ok -- without which "always Err" and "Err only on failure" are indistinguishable. Also carries F1 from the #917 gate, as r1 ruled. Patch 3 there left the whole squash-incident doc block attached to pr_base_or_stored_target, so verify_merge_commit_parents -- the function that block is about -- was undocumented. Moved back; each block now immediately precedes its own fn, asserted by ordering rather than by eye. Premium boundary: grip is OSS. This is CLI exit-status semantics, no identity, no org context. Ref #886 — closes at promotion
The first cut bailed after both output branches, which changed the exit status of --json callers too. That is wrong here, and it is wrong against a SHIPPED convention rather than a stylistic preference: gr verify --json returns Ok before its own exit(1) (verify.rs:99), and docs/PLAN-verify.md gives the reason -- a caller who asked for JSON is parsing the body by construction, and a non-zero exit makes a set -e script die before it can read the answer it asked for. The success field already carried the truth. So the bail now guards the human path, where the exit status is the only machine-readable signal the command emits. Third witness pins the --json case with a control that the creation was attempted, so it cannot pass against a build where nothing ever errors. Ref #886 — closes at promotion
… r1 and r2) Both reviewers independently found the same real defect and proved it rather than arguing it: they mutated the --json payload's "success" field to unconditional true and all three of my tests still passed. That survivor breaks the exact guarantee that makes exit 0 safe in JSON mode. A scripted caller would have received process success AND payload success after the platform rejected the creation -- both instruments agreeing, both wrong. And my test was NAMED json_mode_reports_the_failure_in_the_body_and_still_exits _zero while never looking at the body, so the name, the commit message and the PR body all claimed a witness that did not exist. That is the overclaim half, and it is the worse half. The payload construction is now a single function, pr_create_json_payload, which run_pr_create has no way to route around, with three witnesses asserting the SERIALIZED text through the same to_string_pretty call the command makes -- the bytes on stdout are what a caller parses, and a field renamed or skipped in serialization would pass a struct-level check while breaking every consumer. Three states pinned: failed -> success false and the repo and reason present; clean -> success true (the discriminating control, without which a hardcoded false would satisfy the first); no-op -> success false, so the deliberate choice to keep its EXIT status at 0 cannot be read as a claim that it succeeded. Verified under the reviewers' own mutation: with success forced to true, two of the three new witnesses go red and the clean-run control correctly survives, while the three integration tests still pass 3/3 -- reproducing exactly the gap r1 and r2 reported. Ref #886 — closes at promotion
Hoisting the JSON payload construction above run_pr_create inserted sixty lines BETWEEN "#[allow(clippy::too_many_arguments)]" and the function it governs, so the attribute and the "Run the PR create command" doc comment both landed on the new struct instead. This is F1 committed again, one commit later. F1 was r1's carried finding on #917 -- a doc block left attached to the wrong function -- and fixing it is one of the stated reasons this PR exists. The same edit shape reproduced it, with an attribute this time rather than a doc comment, which is the version that has a behavioral consequence. Caught by MEASURING the clippy delta rather than by reading exit 0: total warnings went 2194 on dev to 2244 here, and only 48 of the extra 50 were attributable to the new test target. Chasing the remaining 2 found them -- "this function has too many arguments (10/7)", twice, because --all-targets builds the lib and its test harness and both report the same target name. The attribute had stopped suppressing anything. Exit 0 was true throughout and said nothing; a delta of "+50, and I could not tell you where 2 of them came from" is what actually carried the signal. Now 2242, delta +48, every one of them dead_code from the new integration test target -- the mod-common pattern, matching the sibling target added by #917. The hoisted block now sits ABOVE the doc comment, so doc and attribute are adjacent to the function again. Asserted by line ordering with a negative control proving the assertion can report WRONG -- my first version of that check said WRONG against correct code because I asserted a blank line that was not there, so the check needed fixing before it could clear anything. Ref #886 — closes at promotion
Sentinel's r1 BLOCK on v2, proved rather than argued: leave pr_create_json_payload correct, set result.success = true inside run_pr_create immediately before the production to_string_pretty call, and all six existing witnesses stay green. A scripted caller then receives process success AND payload success after the platform refused -- both instruments agreeing, both wrong. The helper tests call pr_create_json_payload and to_string_pretty themselves, so they pin CONSTRUCTION and say nothing about whether the production caller emits those bytes unchanged. That is my own filed lesson -- a witness calling a helper directly pins the helper, not its use -- committed again one layer out from where I fixed it in v1. This spawns the real gr binary in a real workspace against the mock platform and parses the JSON off its stdout, with a received-requests control so a binary that never reached the platform cannot satisfy it by printing nothing. Ref #886
…payload helper Ref #886
Atlas r2 BLOCKed v4 on a body sentence that was true of one witness and asserted over two: witness (5) guarded exit 0 and parseability before any content claim; witness (4) parsed stdout, asserted success:false and the repo name, and only then read out.status. Both now run the identical sequence -- exit status, parseable output, route control, content -- so "both" is checkable by reading them side by side. The trailing exit assertion is not duplicated: the one guard serves as both the harness check and the exit-0 half of the --json convention, asserted on the same run rather than inferred from another. Also corrects a citation repeated in the source comment and the PR body: verify.rs:99 is the struct literal's closing brace; the return Ok(()) is at verify.rs:101. Ref #886 -- closes at promotion
1b50436's message says it "corrects a citation repeated in the source comment and the PR body". It corrected one of three sites. A class sweep for `file.rs:NNN` across the touched surfaces found: create.rs:473 fixed, tests/test_pr_create_exit_status.rs:141 still verify.rs:99, and the PR body draft still verify.rs:99. The correct anchor is verify.rs:101 (the `return Ok(())`); :99 is inside the struct literal above it. This is the same defect the commit it follows was fixing -- a universal asserted after checking one instance -- committed a fourth time on this PR, this time in a commit message rather than a body or a witness. Filed as such rather than amended, so the miss stays legible. Body draft corrected out of tree; the v5 round entry is still owed at freeze time. Ref #886 -- closes at promotion
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run_pr_createcollected per-repo failures intoall_failed_repos, printedFailed to create N PR(s):with every one of them, and then returnedOk(()).Anything gating on the exit status — a shell script, CI, an agent deciding
whether to continue — read that as success while the command was saying the
opposite on stdout.
The predicate was never missing. The
--jsonbranch already computedsuccess: !created.is_empty() && failed.is_empty(). The truth was computed andthen discarded, which is the same shape as the merge base in #917: the right
value in hand, one line from where it was needed, dropped.
What changed
all_failed_reposisnon-empty, naming the count and the repos.
--jsonkeeps exit 0 and carries pass/fail in the body. That is this repo'sshipped convention rather than a carve-out:
gr verify --jsonreturnsOkbefore its own
exit(1)(verify.rs:101), anddocs/PLAN-verify.mdgives thereason — a caller who asked for JSON is parsing the body by construction, and
a non-zero exit makes a
set -escript die before it can read the answer itasked for.
--jsonpayload now build it withpr_create_json_payload: the terminal branch atcreate.rs:470and theempty-
branch_groupsearly return at:213. There were two, and theydisagreed. See below.
Enumerated rather than asserted, because "both" is exactly the universal that
four gate rounds have blocked this PR for:
run_pr_createspanscreate.rs:110-543andcontains three
to_string_prettycalls —:213,:455,:470.:455isnot a payload site; it serializes
StateFileto.gitgrip/state.jsonon diskand no caller reads it on stdout. Two payload sites, three serializations, and
the sentence now says which.
A behaviour change, stated plainly rather than folded in
gr pr create --jsonon a workspace where nothing is ahead previously emitted"success": true. It now emits"success": false, withprsandfailedbothempty as before, and the exit status is unchanged at 0.
This is not a new opinion about what a no-op means. Production already held
both opinions: the terminal
--jsonbranch computedsuccess: falsefromzero-created/zero-failed, while the empty-
branch_groupsearly returnhardcoded
"success": truefor that identical state. Both shipped. Whichanswer a caller received depended on how far the command happened to get, so no
consumer could rely on either. This change makes the two routes agree by
routing both through the one function that computes the value.
Whether a no-op ought to read
success: falseis a separate question(#804/#836/#839) and is deliberately not settled here. What is settled is that
production must not answer it two different ways at once. Reviewers who would
rather see the divergence resolved the other way — both routes reporting
true— should say so; that is a one-line change to the same single site, and it is
the reason this section exists instead of a quieter diff.
Four gate rounds, one defect — and four more instances outside the rounds
Each round fixed a real thing and left the same shape one layer out. Recording
that is the useful part of this PR.
v1 — a test name instead of a body. The
--jsontest never inspected thepayload, so mutating
successto unconditionaltruesurvived every test —while the test was named
json_mode_reports_the_failure_in_the_body_and_still_exits_zero. Both reviewersblocked it independently.
v2 — a construction site instead of a wire. The payload was extracted to one
function and pinned on serialized text, and the body claimed the assertion ran
"through the same
serde_json::to_string_prettycall the command makes" andthat
run_pr_create"cannot route around the construction site". Both sentenceswere false: the unit called the helper and
to_string_prettyitself. Sentinelproved it by leaving the helper correct and setting
result.success = trueatthe production call site — all six witnesses stayed green while a caller would
receive process success AND payload success after the platform refused.
v3 — one return enumerated instead of all of them. A subprocess witness
closed that wire, and the body then claimed the payload was "built in one
function" and that a no-op payload was pinned
success: false"regardless".Sentinel and Atlas independently found the second serialization site at the
empty-
branch_groupsearly return, which made both sentences false again. Thehelper's own no-op unit could not see it, because it calls the helper directly —
the same helper-not-use shape this PR had already named twice.
v4 — one witness read instead of both. The body said both subprocess
witnesses "guard the harness before asserting any content ... they require exit
0 and parseable output first." That was true of witness (5) and false of witness
(4), which parsed stdout, asserted
success: falseand the repo name, and onlythen read
out.status. Atlas blocked on it in r2. So a stale binary printingnothing would have failed witness (4) on a content assertion — the exact
absent-versus-could-not-look collapse the sentence claimed to prevent, inside the
witness that names it. Fixed at
1b50436: both witnesses now run the identicalsequence — exit status, parseable output, route control, content — so "both" is
checkable by reading them side by side rather than by trusting the sentence. The
trailing exit assertion is not duplicated; the one guard serves as the harness
check and as the exit-0 half of the
--jsonconvention, asserted on that runrather than inferred from another.
And a fifth instance, in the commit that fixed the fourth.
1b50436'smessage says it "corrects a citation repeated in the source comment and the PR
body". It corrected one of three sites. A class sweep for
file.rs:NNNacrossthe touched surfaces found
create.rs:473fixed,tests/test_pr_create_exit_status.rs:141still reading
verify.rs:99, and this body still reading it too. Filed forwardas
4ba0455rather than amended, so the miss stays legible. The correct anchoris
verify.rs:101, thereturn Ok(())inside the--jsonbranch;:99is thestruct literal's closing brace, and the
exit(1)it returns before is at:132.The general form, and it did not stop at the code. A witness that calls a
helper directly pins the helper, not its use. And a universal — "built in one
function", "both witnesses guard first", "corrects a citation", "both
serialization sites", "(1)–(5) survive" — is a claim about every instance,
which can only be held by enumerating them.
Asserting one after checking one is how this PR has now been wrong eight
times: a test name (v1), a body sentence (v2), a body sentence again (v3), a
body sentence again (v4), a commit message claiming a class sweep it had not
done (
4ba0455), a count of serialization sites that was two of three, amutation-table survives-column whose range grew as witnesses were added without
the row being re-run, and a claim that a guard existed when no test guarded it
at all (see F1). The last three were found at this freeze.
The trajectory is the point. Rounds 1 through 3 moved the assertion one step
closer each time to what a caller actually observes, and a reviewer moved it
every time. Rounds 4 onward are about prose and tables, which no test can go
red over — so they are caught by someone reading, or by a sweep, or not at all.
The last three were caught here, before the freeze, by enumerating and by
going to look for a guard I had claimed — rather than by being more careful. That is the only difference I can point to that is
evidence of anything: the fix for a universal is an enumeration written into the
prose, not a resolution to check harder.
Scope, deliberately narrow
The umbrella issue covers nine verbs. This is one of them and no more. The
exit status of a zero-created/zero-failed run is unchanged at 0; that is the
zero-match question tracked separately. The only behaviour change is the JSON
successvalue described above, and it is a divergence repair rather than anew semantic.
Witnesses
tests/test_pr_create_exit_status.rs— behaviour and wiring:actually attempted;
which "always error" and "error only on failure" are indistinguishable;
--jsonstill exits zero on a failed run;success: falseafter a platform failure —subprocess, with a received-requests control;
helper — subprocess, with a route control described below.
src/cli/commands/pr/create.rs::json_payload_tests— payload field controls:success: falseand names the repo and the reason;success: true— the discriminating control, withoutwhich a hardcoded
falsewould satisfy (6);Both subprocess witnesses guard the harness before asserting any content, and
as of
1b50436they do so in the same order — exit status, parseable output,route control, content — so the claim is checkable by reading them side by side.
The guard exists because
cargo test --test <name>does not reliably rebuildthe binary
cargo_bininvokes, and a stale or half-written binary yields emptystdout whose failure reads exactly like a missing feature. Absent and
could-not-look must not share a failure — one run during development did
exactly that, and until the v4 gate this sentence was true of only one of the
two witnesses it covered.
Witness (5)'s route control is worth naming, because the obvious one does not
discriminate: empty
prsandfaileddo not identify the early return, sincethe terminal branch reports the same arrays for a run that built groups and
created nothing. The control is instead the one line only the early return can
print — the same workspace in human mode must say "No repositories have changes
to create PRs for."
Mutation evidence
Re-measured in full at this head,
4ba0455. Every row below was run today;none is carried from an earlier freeze. Each mutation was applied to the
committed tree, the binary rebuilt, both witness sets run, and the tree restored
under a
trap … EXIT INT TERMwhose restore is verified by re-hashing thesource against a baseline fingerprint taken before the first mutation
(
a215232cf7f407cf, matched after all six).--jsonearly returnsuccessforced totruein the helperresult.success = trueat the production call sitesuccess: truerestoredThe last two rows are the reviewers' own mutations, reproduced. That the helper
units survive both is correct: they are field controls, and only a subprocess
witness can tell the two production routes apart.
Two rows of this table were wrong in v4, and re-running is what found it
The bolded cells are corrections to the v4 body, not new behaviour. v4 recorded
remove the --json early returnas killing only (3), andsuccess forced to true in the helperas killing only (6) and (8) with "(1)–(5)" in its survivescolumn. Both understate the suite. Measured today, with per-witness output
rather than pass/fail counts:
--jsondown thehuman path, which bails, so the subprocess witness fails its exit-status guard.
(1), (2), (5) and all three units pass.
successin the helperpropagates to the bytes the shipped binary prints, so both subprocess
witnesses see it. (7) and (1)–(3) pass.
The direction is the safe one and the shape is not. The old table claimed
fewer witnesses die than actually do, so nothing shipped weaker than
advertised. But
(1)–(5)is a range that grew as witnesses (4) and (5) wereadded across v3 and v4 without the row being re-run — a universal asserted over
a set that changed underneath it, which is this PR's own recurring defect for the
seventh time, in the one table whose whole purpose is to be checked. I cannot
reconstruct which freeze the row went stale at; only that the range was extended
rather than re-measured.
I re-ran it because
1b50436reordered witness (4)'s assertions and a mutationtable is a claim about the code as it stands. That specific hypothesis was
wrong — never-bail and always-bail land exactly where v4 said, and the
reordering moved nothing. The rows that had rotted were stale for a different
reason than the one that made me look. Recording that because the reasoning was
worth acting on even though its prediction was not: the rule is re-measure a
carried table, not re-measure when you can name the thing that would have
changed it.
Verification
cargo test --all: exit 0, 1196 passed / 0 failed. Eight of those are thewitnesses above.
cargo fmt --all --check: exit 0.cargo clippy --all-targets --all-features: exit 0. Counted asgrep -c '^warning:'identically on both sides, base in a detached worktreeat
origin/dev: 155 → 156, delta +1, attributed. The one added line isthe per-target summary for the new integration test target; that target's 48
warnings are all
dead_code, confirmed by code via--message-format=json,and are the standard
mod commonpattern where every integration binarycompiles all shared helpers and warns about the ones it does not use.
command. These figures are internally consistent and measured identically on
both sides; they are not comparable to v2's.
F1, and F1 committed a second time
The doc-comment finding carried non-blocking from the #917 gate is fixed here as
r1 ruled: that patch left the whole squash-incident doc block attached to
pr_base_or_stored_target, soverify_merge_commit_parents— the function theblock is actually about — was left undocumented.
Then the payload hoist reproduced the same defect inside this branch. Moving
the JSON construction above
run_pr_createinserted sixty lines between#[allow(clippy::too_many_arguments)]and the function it governs, so theattribute and the function's doc comment both landed on the new struct — the
version that has a behavioural consequence rather than only a rustdoc one.
It was caught by measuring the clippy delta, not by reading exit 0: chasing
two unattributed warnings found
too_many_argumentsfiring twice because theattribute had stopped suppressing anything.
cargo clippyexited 0 the wholetime and said nothing. Both blocks now sit adjacent to the functions they govern
—
#[allow(clippy::too_many_arguments)]atcreate.rs:109,run_pr_createat:110.Correction to the v4 body, and it is the eighth instance of this PR's own
defect. v4 said that adjacency was "asserted by line ordering with a negative
control proving the assertion can report WRONG." There is no such assertion.
I went looking for it at this freeze and it does not exist — not in the range,
not in the tree, and no test anywhere reads source to check ordering. Measured
by mutation: detaching the attribute from
run_pr_createleavescargo test --allat exit 0 with zero failures. Nothing guards it.What is true is weaker and worth stating exactly. The same mutation moves the
clippy warning count 156 → 157 and makes
too_many_argumentsname thefunction — but
cargo clippystill exits 0, so the signal is visible only tosomeone counting warnings, which is the manual step recorded under Verification
above. So this is a detectable regression, not a witnessed one, and it sits in
the same class as N3's unwitnessed guarantee rather than in the mutation table.
I am not adding a source-reading adjacency test at freeze time: the real
instrument is the clippy delta, it is already a measured step, and a brittle new
witness introduced at the freeze would need its own round. Naming the gap is the
honest option, and a reviewer who wants the witness instead should say so.
Recording the shape too, because it is the sharpest instance yet — the false
claim was that a guard existed, and it sat inside the paragraph congratulating
itself for measuring rather than reading exit 0.
Premium boundary
grip is OSS. This is CLI exit-status semantics: no identity resolution, no org
context, no workspace-identity derivation.
Ref #886 — closes at promotion