[clubanderson]

Fixes #14685

Why

Live evidence 2026-10-27 ~19:20Z on hive's own pod (8 CPU / 40Gi): 23–25 open PRs, merge-skip reasons in 15 min review-gate:no-review-bundle ×41 and contributor-lane:no-verdict ×17, stage labels: PR waits on the hive past the stage dwell — reviewer lane needs attention. Only reviewer and reviewer-2 exist; the pod had headroom. Review throughput is the chokepoint and a fixed reviewer count cannot follow it.

What

review.reviewers.autoscale sizes the reviewer lane to its backlog. Off by default.

review:
  reviewers:
    autoscale:
      enabled: false
      agent: ""                      # default: first role:reviewer agent, else reviewer_agents[0]
      min: 1
      max: 4                         # ≤ replicas cap (5)
      target_backlog_per_reviewer: 2
      scale_up_cycles: 2
      scale_down_cycles: 5
      cooldown: 5m
      cpu_per_reviewer: 1.0
      mem_per_reviewer_mb: 2048

Controller (pkg/review/autoscale)

• Demand — the dispatch planner now records DispatchPlan.Waiting: PRs with perspectives still to review on their current head that it could not dispatch because every eligible reviewer was busy (no-free-reviewer) or max_parallel_reviews was spent (no-parallel-slot). Drafts/holds/already-dispatched are excluded by construction. Waits are aged across cycles keyed on repo#number@head.
• Scale-up — behind = backlog > target × active, or oldest wait > stage/050-review-pending dwell. Sustained scale_up_cycles → +1 replica. Guards, in order, each logged by name: at-max, cooldown, saturated (≥ half of active reviewers' last kicks were CLI did not reach input prompt — saturation, not demand), provider-budget (inference budget latch), provider-headroom (< 10 % on any provider when rotation reports), max-parallel-reviews, cgroup cpu / memory (v2 cpu.max, cpu.stat, memory.max, memory.current; unknown cgroup never blocks). A refused scale-up keeps its streak.
• Scale-down — idle = nothing waiting and ≥ 1 reviewer with no kick in flight. Sustained scale_down_cycles (and > min, cooldown passed) → highest replica marked draining: planner and cadence kick skip it; it is retired (tmux session ended, config entry dropped) only when its turn has ended and it has no pending review. Never killed mid-turn. Demand during drain un-drains instead of adding.
• All thresholds are named constants in pkg/config/reviewer_autoscale.go / pkg/review/autoscale/controller.go.

Materialisation

The controller only sets Config.ReviewerAutoscaleActive; ExpandAgentReplicas uses it as the lane's effective count, so reviewer-3…N are ordinary replicas — same role, prompt, beads dir, scoped token, tmux/pluk wiring and cadence key as a configured reviewer-2. Planner, kick governor, dashboard agent list and token cache cannot tell them apart. ReconcileAgents adds/removes the processes. The live count survives a config reload and is never persisted (MarshalYAML already strips replicas). entrypoint.sh pre-creates per-agent users/token caches up to max.

Observability

• Log: reviewer autoscale: 2→3 reason=backlog=7 oldest=41m0s, … 3→2 drain reviewer-3 reason=idle for 5 cycles, … retire reviewer-3, … blocked by cpu.
• /api/status.reviewerAutoscale: enabled/agent/min/max/current/target/draining/backlog/oldestWait/streaks/lastDecision.
• Dashboard: lane agent card shows N reviewers · auto badge (replicas · auto / · draining) with last decision in the tooltip.
• Settings → Review Gate → Reviewers → Scale reviewers with the backlog (owner-only, PUT /api/config/review reviewer_autoscale).

Docs / changelog

docs/review-swarm.md (new section, help-mark anchor), docs/operator-reference.md, docs/agent-configuration.md, hive.yaml.example, changelog.d/added-reviewer-autoscale.md.

Tests

Controller decision table (sustained backlog up, single cycle holds, at-max, cpu/memory guards, unknown resources don't block, saturated, provider budget/headroom, max-parallel, dwell-age trigger, cooldown, drain → busy never retired → retire when idle, never below min, undrain on demand, wait ageing, cgroup prober fixture), config tests (defaults, validate, agent resolution, expand/clamp/scale-down), planner Waiting/Draining tests, entrypoint shell tests. No time.Sleep; injected now.

Validated locally with go build ./..., gofmt -l, go vet, dashboard inline-JS lint, entrypoint shell tests. go test left to CI.

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR	Title	Relation	Files
#14630	🐛 fix(dashboard): enrich soft-linked stage-tracker issues with title and state	CONFLICT	src/docs/api-reference.md
#14682	🌱 runtime: loopback pprof, periodic memstats and cgroup-aware GOMEMLIMIT	same files	src/cmd/hive/main.go src/docs/api-reference.md src/docs/operator-reference.md src/pkg/config/config.go src/pkg/dashboard/server.go
#14524	✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity	same files	src/cmd/hive/main.go src/docs/api-reference.md
#14688	🐛 fix(adoption): re-sign inherited commits when re-homing a dco-missing contributor PR	same files	src/cmd/hive/main.go
#14702	✨ review: scale reviewer concurrency up and down with backlog	CONFLICT	src/cmd/hive/main.go src/pkg/config/config.go src/pkg/review/dispatch.go
#14628	🐛 fix(review): require sign-off on review-fix follow-up commits	same files	src/pkg/review/dispatch.go
#14651	🌱 dashboard: defer merge-gate pills until the PR reaches the gate	same files	src/pkg/dashboard/static/index.html
#14701	✨ feat(hub): add S256 PKCE verifier cookie to the Hub GitHub login	same files	src/docs/api-reference.md
#14573	✨ feat(dashboard): show a draft pill in the activity tracker	same files	src/pkg/dashboard/static/index.html
#14687	🐛 fix(agent): log a repeated hub project-config refusal once per distinct misconfiguration	same files	src/cmd/hive/main.go
#14656	🐛 fix: keep the header's right chips from overlapping the title at 769-900px	same files	src/pkg/dashboard/static/index.html

[hivecommons-hive]

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

• coverage

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

[hivecommons-hive]

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

[hivecommons-hive]

4c1e01f

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 4c1e01f43d31) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

• sensitive_path — changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
  • src/deploy/entrypoint.sh
  • src/deploy/test_entrypoint_replica_names.sh

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

[hivecommons-hive]

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

• the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

[hivecommons-hive]

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

• docker (hub)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

[hivecommons-hive]

112c27f

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 112c27f0afac) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

• sensitive_path — changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
  • src/deploy/entrypoint.sh
  • src/deploy/test_entrypoint_replica_names.sh

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

[hivecommons-hive]

Evidence bundle `#14695112c27f` sha256:9a9e0f8ba7b0ee17cedb28ff094972f9c7b16c31ee2937701ae14ed31126bcd4 — download from the dashboard or `hivectl review evidence #14695`.

[hivecommons-hive]

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 112c27f.

[github-actions]

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

[hivecommons-hive]

Decision needed: @clubanderson should decide which of #14695 and #14702 lands for #14685 — both add review.reviewers.autoscale and both edit pkg/config/config.go, validate.go, pkg/review/dispatch.go and cmd/hive/main.go, so they conflict and cannot both merge.

Conclusion: the controller and the config/replica wiring read correct to me; one medium defect in the retire path, and this PR conflicts with #14702 (same feature, same issue, same config key, different package layout), so a maintainer has to pick one.

Conflict, not duplicate

#14702 (@Karibhou) implements review.reviewers.autoscale for the same issue. I compared both file sets: they share config.go, validate.go, review/dispatch.go, cmd/hive/main.go, hive.yaml.example, docs/review-swarm.md but diverge everywhere else (pkg/review/autoscale/ package + drain/retire + cgroup prober + dashboard settings here vs. a single pkg/review/autoscale.go + ReplicaLimits there). Different content, same intent — whichever lands second will not apply. Not recommending either closure; that is the decision above.

Finding (medium) — a drained replica that was never kicked, or whose turn-end signal was lost, is never retired

• src/cmd/hive/reviewer_autoscale.go:145 sets Busy = proc.State == crux.StateRunning && !proc.KickOutcome.Settled(proc.LastKick).
• src/crux/kick.go:72 — Settled returns false when kickAt == nil. So a running replica that has not yet received any kick (scale-up started it, demand dropped before the planner handed it a PR; cadence kicks are withheld once draining) is Busy == true indefinitely.
• src/pkg/review/autoscale/controller.go:426 — retirable skips any Busy draining reviewer, and controller.go:307 refuses a further scale-down while len(c.draining) != 0.

Consequence: the lane gets stuck one above where it should be, with an idle process still holding its CPU/memory reservation, until demand returns and un-drains it. The same hold applies to a wedged turn (lost turn-end), which is exactly the case src/cmd/hive/self_upgrade_drain.go:190-201 already handles for the upgrade drain: it skips proc.LastKick == nil outright and treats an unsettled kick older than inFlightKickCeiling as wedged rather than in flight. Reusing that predicate (or agentKicksInFlight itself) for rv.Busy would close both gaps. Note the controller tests cannot see this because Busy is an input; reviewer_autoscale.go's Run has no test of its own.

What I verified and found sound

• KicksUndeliverable is a consecutive counter reset on delivery/restart (src/crux/agent/manager.go:214, busy_visibility.go:281), so Saturated = KicksUndeliverable > 0 means what the body says.
• ExpandAgentReplicas strips and re-materialises derived agents, so re-expanding on SetReviewerAutoscaleActive and after reload is idempotent; main.go:3961 copies ReviewerAutoscaleActive onto newCfg before the reload's ExpandAgentReplicas at main.go:4057, so the live count does survive a reload as claimed.
• Draining excludes a reviewer from reviewCapableAgents (dispatch.go:1736), so a PR left waiting while a replica drains triggers the un-drain path rather than a fresh replica — matches the design.
• The settings PUT validates the band before mutating and keeps pointer-field "absent means unchanged" semantics; entrypoint.sh caps pre-created users at min(max, MAX_REPLICAS).

Body matches the diff; nothing claimed that is not implemented.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88