Skip to content

🐛 Stage tracker keeps a PR the hive merged itself until the webhook or sweep catches up #14420

Closed

Description

@clubanderson

Summary

A pull request the hive merges itself (automerge sweep, queue, relay or forge path) stays listed as open on the stage tracker and in the governor's cached actionable snapshot until GitHub's pull_request closed webhook comes back round through the hub, the closed-row sweep runs on the next status publish, or the next full enumeration replaces the snapshot. When those are delayed or lost the row lingers for an hour or more.

Observed

Hosted hive hosted-available-oke-11-placeholder-r05x, 2026-10-26, hivecommons/docs#469 (dependabot PR):

  • /data/audit.jsonl: pr_merged at 18:25:46Z, detail path=sweep, agent=governor - the hive merged it.
  • Same file: intent_evaluated lane=merge-eligible for docs#469 at 18:31, 18:42, 18:47, 18:52, 18:59, 19:04 and 19:10Z - the governor kept evaluating the merged PR from its cached snapshot for 45+ minutes.
  • Tracker "Needs human" view at 19:25Z still showed docs#469 as "Sentinel alert - needs human", "no linked issue".
  • Spoke log 18:11-19:40Z: one closed-sweep eviction at 18:13Z, none after; eval ran in webhook cadence (interval 900s, "planning from cached snapshot", snapshot_age up to 57m).
  • Hub pod rolled three times (18:59, 19:19, 19:41Z) and hub relays to this spoke failed 921/2314 with context deadline exceeded while the spoke thrashed before an OOM kill (16Gi) at 19:40Z. The relay is single-shot with a 30s timeout and no retry, so the closed delivery for fix: lowercase display names, fix config dialog blank on open #469 was most likely lost.
  • After the 19:42Z restart (fresh enumeration) the row was gone.

Why

Every merge path ends in Client.RecordPRMergedAudit, which writes the audit record, review evidence and clears merge-failure alerts - but nothing evicts the PR from Server.status or the cached ActionableResult. Eviction only exists on the webhook path (evictClosedWebhookItem -> applyWebhookItemSnapshots), on the closed sweep (maybeSweepClosedRows, only from UpdateStatusIfFresh, 5 min interval, moving since) and on re-enumeration. The hive's own write is the one close it never needs to be told about, yet it is the only one not applied locally.

Fix

Apply the same closed snapshot the webhook path would apply, from the attribution audit sink, when the record is pr_merged or pr_closed. This patches the governor snapshot (Dependencies.ApplyWebhookItems) and the published status, is remembered for replay so an in-flight enumeration cannot resurrect the row, and needs no GitHub call.

Separate observations worth their own issues: the spoke memory growth (16Gi in 90 min, hive:pr_terminal_merged_by made 412 PR GETs in 6 min) and the hub relay's single-shot delivery with no retry.

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Oct 26, 2026
  2. hivecommons-hive commented on Oct 26, 2026

    @hivecommons-hive
    Contributor

    🔒 Claimed by @thatguywob (contributor on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 30m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.

  3. hivecommons-hive commented on Oct 26, 2026

    @hivecommons-hive
    Contributor

    🔓 Claim by @thatguywob released (lease revoked).

  4. github-actions commented on Oct 26, 2026

    @github-actions
    Contributor

    @clubanderson this issue was closed by @hivecommons-hive[bot] as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions