Skip to content

πŸ› fix(convergence): scope the mutation writer slot per target instead of per repo - #14671

Merged
hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-14645
Oct 27, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-14645

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Why

#14645 found that in 4 hours on one hive, 110 of 111 shadow fence would have denied external mutation warnings said repo hivecommons/hive writers exhausted (max 1). Most were label-only writes: 85 label_mutation, 11 issue_comment, and a handful of other kinds. The cause is DefaultMaxWritersPerRepo = 1 together with activeWritersLocked(repo). That gives each repository one writer slot shared by every issue, PR and mutation kind, so a single in-flight review or merge on any PR blocks every label write on every other item. These writes all still ran because the boundary is in shadow mode. Once it moves to enforce, they would be denied.

Decision taken by hive

The writer slot is now scoped per target (owner/repo#N) instead of per repository. Mutations with no issue or PR number (branch_push, issue_create, pull_request_create, digest/branch targets) fall back to one shared pool per repo. The default stays max 1 per scope, and the limit can be set with the new key convergence.mutation.max_writers_per_target. We picked per-target over per-(repo, kind) because it is the simplest change that fully separates unrelated items. Writes to the same item still go one at a time, which keeps label, review and merge ordering on one PR serialized.

The change adds code and renames nothing:

  • mutation.OpenLedger and DefaultMaxWritersPerRepo behave as before (per-repo scope).
  • New: WriterScope, RepoWriterScope, TargetWriterScope, OpenLedgerScoped, DefaultMaxWritersPerTarget.
  • cmd/hive now opens the boundary ledger with OpenLedgerScoped(..., cfg.MutationMaxWritersPerTarget(), TargetWriterScope).
  • The capacity error keeps its existing shape. Repo-scoped claims still read repo X writers exhausted (max N), and target-scoped claims read target X#N writers exhausted (max N). The boundary.go comment now describes both.
  • The new key is added to the settings-parity allowlist as yaml-only.

Tests (each fails without the change)

All tests use injected clocks and contain no time.Sleep.

Closes #14645

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson

…f per repo

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 27, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14687 πŸ› fix(agent): log a repeated hub project-config refusal once per distinct misconfiguration same files src/cmd/hive/main.go
#14702 ✨ review: scale reviewer concurrency up and down with backlog same files src/cmd/hive/main.go
#14688 πŸ› fix(adoption): re-sign inherited commits when re-homing a dco-missing contributor PR same files src/cmd/hive/main.go
#14524 ✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity same files src/cmd/hive/main.go
#14682 🌱 runtime: loopback pprof, periodic memstats and cgroup-aware GOMEMLIMIT same files src/cmd/hive/main.go
src/pkg/dashboard/settings_parity_allowlist.txt
#14695 ✨ review: reviewer lane scales concurrency with backlog same files src/cmd/hive/main.go

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. Read against the v5 tree, not just the diff:

  • TargetWriterScope parses the subject the boundary actually mints: claimSubject at src/pkg/convergence/mutation/boundary.go:300 builds repo#kind/target, and every numbered caller I surveyed in src/pkg/github/** passes Target: strconv.Itoa(number); branch/push/create callers pass a ref, which is non-decimal and falls back to the repo pool as intended. No effects.Kind* constant contains /, so stripping to the first / cannot mis-split.
  • Repo-wide safety is unchanged: Overlaps (claim.go:128-130) still makes a ClaimTypeRepo claim conflict with every task claim in the repo via ErrClaimHeld, so narrowing only the capacity bucket does not let a repo claim run alongside an in-flight per-target mutation.
  • main.go:2027 is the only production OpenLedger caller, so the scope switch is complete; OpenLedger keeps its per-repo behaviour for existing tests/callers.
  • settings_parity_allowlist.txt entry included, which is why test (dashboard …) is green here (contrast #14677, same day, which missed it).

One suggestion, not a defect: convergence.mutation.max_writers_per_target is named in the changelog but documented nowhere an operator reads β€” src/hive.yaml.example has no convergence: block and src/docs/env-vars.md is the only place convergence.mode appears. A two-line example under a convergence: stanza would make the knob discoverable. Fine to do in a follow-up.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 27, 2026
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/check-report (head 932034f): the only red check, build (linux/arm64, true, true, 75), is infrastructure, not this diff β€” buildx could not fetch a cached image layer from the registry (failed to compute cache key … could not fetch content descriptor sha256:9787421b… not found). All test shards, coverage and the gates are green; merge relay refused at this head because it moved past the reviewed SHA b99a49d. Deferring to the next sweep.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: bcad0c9b5

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI triage for 932034f: the red build (linux/arm64, true, true, 75) job is an infrastructure failure, not a code failure.

  • arm64 job 122449392089 failed inside buildx while resolving the remote build cache: failed to compute cache key: ... could not fetch content descriptor sha256:9787421b… from remote: not found (a cache layer vanished from the registry).
  • amd64 job 122449392083 was only cancelled by fail-fast as a result; it was still mid-apt-get with no errors of its own.

This PR touches only Go sources, a test, a changelog fragment, and the settings-parity allowlist β€” nothing in the Dockerfile or build-cache configuration β€” so there is no code fix to push. The build should pass on a re-run once the cache is warm/repopulated.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read ledger.go, boundary.go and claim.go at this head. Looks correct to me, and it is the per-target option #14645 asked for first.

What I checked, since TargetWriterScope depends on the subject shape rather than a structured field:

  • boundary.go:302-304 builds the subject as repo#<kind>/<target>; every kind in src/crux/effects/effects.go:19-30 is a plain snake_case token with no /, so ledger.go:79-87 strips exactly the kind and keys on the target. A numeric target (strconv.Itoa(number), the normal case) buckets as repo#N; a branch-named target (automerge_contributor_adopt.go:677, :703 β€” hive/adopt-7) is left with adopt-7 after the first /, fails isDecimal, and falls back to the repo bucket as the doc comment promises.
  • claim.go:116-129 β€” Overlaps is untouched, so the repo-claim-vs-task-claim exclusion still holds across buckets; only the capacity count changed.
  • OpenLedger keeps RepoWriterScope (ledger.go:155), so existing callers and the cross-process tests keep today's semantics; only bootAdvisoryWith opts into the target scope.
  • MutationMaxWritersPerTarget() returns 0 for nil/unset/non-positive and OpenLedgerScoped maps 0 to the default of 1, so an absent convergence.mutation block behaves as the changelog states.

One remark, not a finding: the error-string branch at ledger.go:313-317 picks the "repo … exhausted" wording by scope == c.Repo, which is also what a target-scoped ledger emits for branch pushes/creates β€” accurate, since those are counted per repo.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 10b6b56 into v5 Oct 27, 2026
81 of 86 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-14645 branch October 27, 2026 20:57
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#14671932034f` sha256:e997e395208236b9415e2a2cdae80eeb4660bb12f3ddb8f289f8cee50cadb246 β€” download from the dashboard or `hivectl review evidence #14671`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

0 participants