[clubanderson]

Why

Refs #14681 — the hive Go process on the hosted hive grows ~360 MiB/min and OOM-kills, and there was no net/http/pprof, no memstats log line and no GOMEMLIMIT, so the leak had to be chased from /proc alone.

What

• pkg/runtimeobs (new):
  • StartPprof — net/http/pprof on a loopback-only listener. Config debug.pprof_addr, default 127.0.0.1:6060, off disables. A non-loopback host is rejected (ErrNotLoopback); bootConfig logs a warning and falls back to the default so a typo can never expose profiles on the pod network. /debug/runtime prints the snapshot below as text.
  • StartLogger — mem: runtime snapshot INFO line every 60 s: heap_inuse heap_live heap_objects total_sys next_gc gc_cycles goroutines threads gomemlimit. Read via runtime/metrics (no STW); threads from /proc/self/status.
  • ApplyMemoryLimit — honours GOMEMLIMIT if set; otherwise sets debug.SetMemoryLimit to MemoryLimitPercent (85%) of the cgroup limit (/sys/fs/cgroup/memory.max, v1 fallback). Unlimited cgroup → untouched. Logged at startup as runtime: memory limit source=env|cgroup|none.
• cmd/hive/runtime_observability.go wires the three into bootConfig right after the root context exists; pprof stop is on the cleanup stack.
• GET /api/health now carries the snapshot under runtime (goroutines, heap, threads) so the dashboard health probe / in-pod curl sees it without the pprof port.
• docs/operator-reference.md → new "Runtime diagnostics" section with port-forward + go tool pprof recipe.
• Changelog: changelog.d/added-runtime-pprof-memstats.md.

Tests

pkg/runtimeobs/runtimeobs_test.go: thread parsing, snapshot population, logger emits per injected tick (no sleeps in the hot path), loopback validation, pprof serves /debug/pprof/ + /debug/runtime, cgroup v1/v2/unlimited/garbage parsing, and the three ApplyMemoryLimit branches (env wins, cgroup derives 85%, none leaves the runtime default).

Operator use on the leaking pod once deployed

kubectl -n <ns> port-forward deploy/hive 6060:6060 &
go tool pprof -top http://127.0.0.1:6060/debug/pprof/heap
curl -s 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' | head

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR	Title	Relation	Files
#14790	🐛 fix: restore App reads for opt-in shared-UID agents	same files	src/cmd/hive/main.go src/pkg/config/config.go
#14524	✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity	same files	src/cmd/hive/main.go src/docs/api-reference.md
#14701	✨ feat(hub): add S256 PKCE verifier cookie to the Hub GitHub login	same files	src/docs/api-reference.md
#14702	✨ review: scale reviewer concurrency up and down with backlog	same files	src/cmd/hive/main.go src/pkg/config/config.go

[hivecommons-hive]

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

[hivecommons-hive]

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

• changelog-fragment-guard

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

[hivecommons-hive]

Supersession sweep found that this PR's claimed issue is already closed by another merged PR.

• Claimed issue: hive#14681
• Closing PR: #14699

Decision needed (@hivecommons): #14699 already merged and closed the claimed issue, but this PR touches files the merged PR did not. Recommended: keep open until the extra changes are reviewed, or close as superseded if they are not wanted. Another option: merge anyway.

Files the merged PR did not touch: changelog.d/added-runtime-pprof-memstats.md, src/cmd/hive/main.go, src/cmd/hive/runtime_observability.go, src/internal/testutil/dashboard_import_allowlist.txt, src/pkg/config/config.go and 5 more.

This comment is edited in place by the supersession sweep; it is not duplicated.

[hivecommons-hive]

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

• the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

[hivecommons-hive]

Read pkg/runtimeobs/runtimeobs.go and cmd/hive/runtime_observability.go in full against v5. The code looks correct to me; the only thing blocking this is the merge conflict.

Verified:

• Loopback enforcement is belt-and-braces: IsLoopbackAddr (runtimeobs.go:171-181) rejects an empty host (":6060" → net.ParseIP("") is nil → false) as well as non-loopback IPs, StartPprof re-checks it (:189) so a caller bypassing bootConfig still cannot bind wide, and bootConfig's fallback to config.DefaultPprofAddr (runtime_observability.go:22-26) means a typo degrades to the default rather than to "off" or to the pod network.
• ApplyMemoryLimit ordering is right: GOMEMLIMIT set → report only, never override (:233-235); max/empty/≥2^62 cgroup values all return ok=false and leave the runtime default (:264-276); DeriveMemoryLimit divides before multiplying so it cannot overflow on large limits.
• StartLogger owns and stops its ticker only when it created it (:156-166), and StartPprof is non-fatal at boot with stop pushed onto b.cleanup.
• All checks are green at c5112475; mergeable=DIRTY is the only blocker.

Conflicts — v5 has moved under 6 of the 13 files since your merge-base (18 commits behind): cmd/hive/main.go, pkg/config/config.go, pkg/dashboard/server.go, pkg/dashboard/settings_parity_allowlist.txt, docs/api-reference.md, docs/operator-reference.md. After rebasing, the 16 new server.go:NNN citations this PR adds to api-reference.md will almost certainly have drifted; bash src/scripts/check-api-reference-citations.sh --fix rewrites them.

Info, not blocking: Snapshot.HeapInuse is sampled from /memory/classes/heap/objects:bytes (:76, :98), which is live+unswept object bytes, not MemStats.HeapInuse (objects plus unused in-use spans). The value is the right one for chasing a leak; only the field/log name heap_inuse is slightly misleading to anyone comparing against runtime.MemStats.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

[hivecommons-hive]

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

• overlayfs-exec-guard

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

[hivecommons-hive]

Evidence bundle `#1468253a9fc6` sha256:3892783f638748882c85435f946c21f32c0bee7b1fddafc05e2c9902c7cd9ad7 — download from the dashboard or `hivectl review evidence #14682`.

[github-actions]

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

[hivecommons-hive]

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 53a9fc6.