[clubanderson]

Fixes #14500

Problem

Adopted contributor PRs #14430, #14440, #14456 (stage/074-adopted-by-hive, in-place on the fork branch) sat red only because of TestPollTmuxOutputForAgent_WiresThrashBreaker — the shared-CI flake in #14406, fixed on v5 by #14415. Their heads were 15–22 commits behind v5 and never ran CI again, so the 074 escape read the stale red as nothing the hive can push and escalated them to 085-needs-decision. An update-branch or a re-run would have made them green.

Fix

escapeStalledAdoption now runs a recovery step before escalating a red adopted head (automerge_adoption_recovery.go):

1. Candidate: hive-owned in-place adoption, head unchanged, lane reason CI-red, no open shared-CI incident / base-red / fork-guard / hold label / blocking verdict / workflow-path refusal.
2. Base clears the failure: the check blamed by required-check-failing:<name> is completed green on the base tip; when the base does not run that check (PR-only workflows) or no name is known, the base tip must have completed checks and none red.
3. Action: head behind base → executeUpdateBranchEffect (GitHub update-branch; works on a maintainer-editable fork, errors classified as before). Head current (or GitHub says up to date) → RerunFailedJobsByID on the latest failed run per workflow.
4. Rate limit: one attempt per (head, base tip) (Record.AdoptRecoveryAttempt) and adoptionRecoveryMaxAttempts = 3 per adoption (AdoptRecoveryCount), both reset with the adoption. Marked before the API call, like every adoption path.
5. Escalation: an attempt holds 074 while CI runs on the result and releases an escalation already in place for the same head; 085 happens only after an attempt (or a skip) and the head is still red. API errors are logged and skipped without consuming the attempt.

One structured line per decision: msg="adopted PR recovery" reason=update-branch|rerun-failed-jobs|skipped:<why>.

Tests

Table-driven TestAdoptedStaleRedRecovery over the adopt harness (update-branch, rerun, base-red, base-unknown, listing failure, fork-not-editable, up-to-date fallthrough, no failed runs), plus the one-attempt-then-escalate / release-on-base-move sequence, the lifetime budget, and unit tables for the candidate predicate, baseClearsFailure and the reason parser. No sleeps.

Docs

src/docs/hive-merge.md, src/docs/labels-and-control-signals.md; changelog fragment changelog.d/fixed-14500-adopted-stale-red-recovery.md.

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

[hivecommons-hive]

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this repository have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

[hivecommons-hive]

Evidence bundle `#1450341f12af` sha256:fdd9357ec7fef4014476d8bb64b38f65b44e55e9c1dd7dfcc04e6fc98ca33051 — download from the dashboard or `hivectl review evidence #14503`.

[hivecommons-hive]

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 41f12af.

[github-actions]

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

[hivecommons-hive]

Read automerge_adoption_recovery.go in full and traced the wiring against v5. The design holds together; one small comment/behaviour mismatch worth a look, no blocking defect found.

What I verified:

• Escalation genuinely waits. escapeStalledAdoption only calls recoverStaleRedAdoption when adoptionStalledReason is non-empty, and adoptionStale (automerge_contributor_adopt.go:1230-1237) returns false while adoptionCIPending(obs) or when obs.HeadSHA != rec.AdoptedHead. So after a re-run the stall is suppressed until CI completes, and after an update-branch the new head is no longer the adopted head — in neither case can the 074 escape fire in the next sweep on the pre-recovery red. On the sweep after a failed recovery, AdoptionRecoveryAttempted(head, baseSHA) short-circuits to skipped:already-attempted and the escape proceeds as before, which is the intended steady state.
• Candidate gate is conservative. adoptionRecoveryCandidate defers to SharedCIIncident, BaseRedChecks, ForkGuardChecks (all populated at automerge_stage_labels.go:417-419), hold labels, blocking/needs-human verdicts and the workflow-path refusal, and requires classifyCIReason == ciReasonRed. baseClearsFailure refuses on base-red:<check> and on an unknown base (base-checks-unknown), so a red base can never trigger an update-branch.
• Bounds. MarkAdoptionRecovery is idempotent per (head, base) pair and increments AdoptRecoveryCount, capped at 3 by adoptionRecoveryMaxAttempts; both reset in MarkYielded and ClearAdoption. Update-branch errors reuse classifyUpdateBranchError (automerge_sweep.go:2263-2285) and the up-to-date case falls through to the re-run rather than consuming a second attempt. MarkHivePushedHead is already gated on AdoptedInPlace() (store.go:954), so the merge commit is attributed to the hive, not the author.

The one thing to reconcile (low): the doc comment at automerge_adoption_recovery.go:172-173 says API failures are "treated as a skip … without consuming the attempt", but store.MarkAdoptionRecovery runs at :216, before the action, so update-branch-failed (:242), list-runs-failed (:265) and rerun-refused (:281) do consume the (head, base) attempt and the lifetime count. That matches the store comment at store.go:1060 ("recorded before the API call … so a refused call is not retried each sweep") and is the safer choice for a 403, but a transient ListRepositoryWorkflowRuns error will spend one of the three attempts and let escalation proceed on the next sweep. Either fix the comment at :172 or move the mark to after the compare/base-check reads for the transient cases — your call; just flagging the two comments disagree.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88