[clubanderson]

Fixes #14681

Root cause

A lock convoy in the dashboard stage tracker, fed by the webhook stream.

where	what
src/pkg/dashboard/stage_tracker.go stageTrackerPayload	held stageTrackerCacheMu for the entire buildStageTracker
stage_tracker.go buildStageTracker → stageTrackerSentinel → github.LoadVerifiedReviewEvidence	one evidence read per open PR per build
src/pkg/github/review_evidence.go reviewEvidenceDirs	each read did filepath.Glob(<repo-dir>/<pr>.uid*) = a full ReadDir of the repo's evidence directory (thousands of entries, NFSv3)
stage_tracker.go invalidateStageTrackerCache	took the same mutex; called from every status publish (server.go UpdateStatusIfFresh) and from every webhook that touched a tracked item (webhook_tracker_refresh.go invalidateStageTrackerForWebhook, both the sync handler path and the debounced re-read's applyWebhookItemSnapshots)

Evidence from the live hive (read-only)

• GET /api/stage-tracker through a port-forward took 40 s to answer (91 KB body); the first attempt timed out at 60 s.
• ingress-nginx access log, 10 min window, upstream hive:3002:
  • POST /api/webhook/github: 1766 (≈177/min), avg upstream time 16 s, 931 × 499 (hub relay gave up at its 30 s timeout). Each one blocked in invalidateStageTrackerCache behind a build.
  • GET /api/stage-tracker: 72 × 504, each with nginx retrying the upstream 3 × (60 s each); the SPA re-requests every ~2 s, so the queue of GET handlers never drained.
• Hive logs: overview export encode failed … broken pipe 54 per 70 s — handlers finishing a build after the client/ingress was gone.
• /proc/<hive>/status: Threads 224→278 in 10 min, fds 911→1190, CLOSE_WAIT sockets 79→216 (/proc/net/tcp) while RSS grew ~350 MiB/min. With mutex FIFO hand-off, after each 40 s build a webhook invalidator typically runs next, re-emptying the cache, and the next GET rebuilds — so blocked GET handlers, their connections (each pinning a 13 KB request body + bufio + the *StatusPayload it captured), the builders' transient garbage and the NFS-blocked threads all accumulate, and the rate grows with the number of open PRs / evidence dirs, matching the "worse across today's builds" observation.
• Ruled out along the way: actionable snapshot cache (30 issues / 28 PRs), REST accounting (1 h window), forgestate deliveries (bounded), nudger/SSE (non-blocking, chan 16), contribute activity ring, evidence heartbeat (5 min refresh).

Fix

• Invalidation never waits. invalidateStageTrackerCache only flips stale and bumps a generation counter under the cache mutex. Webhook handlers and the governor's publish return immediately.
• Single-flight, stale-while-revalidate. The build runs outside the mutex; at most one is in flight (Server.stageTrackerBuild). Polls that land during a build get the previous body with source: stale, X-Hive-Stage-Tracker-Source: stale, max-age=0 instead of queueing. A build that was invalidated while running stores its result already marked stale, so the next poll rebuilds from newer status but nobody blocks.
• Disconnects stop waiting. A poll with no body yet joins the in-flight build and honours r.Context() (503 if the client is gone).
• The per-PR directory re-read is gone. reviewEvidenceDirs memoises the repo directory's .uid* sibling names for 60 s (evidenceSiblingListTTL, bounded to 512 parents) and drops the entry when this process writes a fallback sibling. A build is now one ReadDir per repo plus one stat/read per PR.

Tests (new, no time.Sleep)

• TestStageTrackerInvalidationNeverWaitsForABuild — holds a build open via an injected build func; invalidateStageTrackerCache and invalidateStageTrackerForWebhook must return.
• TestStageTrackerBuildsAreSingleFlightAndServeStale — 200 concurrent polls (each preceded by a webhook invalidation) during a held build: all return 200 stale at once, exactly one build, runtime.NumGoroutine() returns to baseline; cancelled-context waiter gets 503; post-build staleness and quiet-state cache hits asserted.
• TestStageTrackerFirstWaitersJoinTheOneBuild — 50 first polls, one build.
• TestReviewEvidenceDirsMemoisesTheRepoListing — memo hit/TTL/forget/cap semantics with an injected clock.

Docs: api-reference.md (source: stale), operator-reference.md (webhook/publish section). Changelog changelog.d/fixed-memory-growth-dashboard-stage-tracker.md.

Caveat

The heap-retention mechanism is inferred from the convoy (blocked handlers + connections + repeated builds), not yet from a heap profile — #14682 adds the loopback pprof listener and periodic memstats so it can be confirmed on the next deploy (kubectl port-forward deploy/hive 6060:6060, go tool pprof -top http://127.0.0.1:6060/debug/pprof/heap). Remaining small unbounded maps noted during the audit (automerge engine maps, StageTracker.checks, stageengine anomalies, certCache, HivePushedHeads) are orders of magnitude too small to explain 350 MiB/min and are left for a follow-up.

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR	Title	Relation	Files
#14695	✨ review: reviewer lane scales concurrency with backlog	CONFLICT	src/docs/api-reference.md
#14682	🌱 runtime: loopback pprof, periodic memstats and cgroup-aware GOMEMLIMIT	CONFLICT	src/docs/api-reference.md
#14630	🐛 fix(dashboard): enrich soft-linked stage-tracker issues with title and state	CONFLICT	src/docs/api-reference.md src/pkg/dashboard/server.go src/pkg/dashboard/stage_tracker.go
#14573	✨ feat(dashboard): show a draft pill in the activity tracker	same files	src/pkg/dashboard/stage_tracker.go
#14651	🌱 dashboard: defer merge-gate pills until the PR reaches the gate	same files	src/pkg/dashboard/stage_tracker.go
#14657	🐛 dashboard: anchor tracker rows on the open PR when the issue has closed	same files	src/pkg/dashboard/stage_tracker.go
#14524	✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity	same files	src/docs/api-reference.md

[hivecommons-hive]

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

[github-actions]

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

[hivecommons-hive]

Evidence bundle `#1469977cbb9b` sha256:b40cb3d877d6c4a62c2b4fc66dd52203dc2238c3610917e85c253245fe806c00 — download from the dashboard or `hivectl review evidence #14699`.

[hivecommons-hive]

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 77cbb9b.