Skip to content

🐛 fix(registry): enrich descriptions once and back off on GitHub rate limits - #480

Merged
hivecommons-hive[bot] merged 1 commit into
mainfrom
scanner/fix-479
Oct 25, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
mainfrom
scanner/fix-479

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Summary

  • Registry.Sync now tells the hub client which repos already have a stored description, so they are not re-fetched from GitHub every sync.
  • HTTPHubClient remembers completed lookups (empty description / non-200) per process.
  • A 403/429 from GitHub stops the enrichment loop with one slog.Warn and honours X-RateLimit-Reset.
  • Why: anonymous enrichment saturated the 60 req/hr bucket shared with settlement verification, causing 502s.

Closes #479

Validation

Added tests: TestSyncEnrichesEachRepoOnce, TestEnrichmentStopsOnRateLimit, TestEnrichmentNotRetriedForEmptyDescription. Not run locally; CI is the verdict.

— hive: agent=scanner backend=copilot model=claude-sonnet-5.5

… limits

Sync now skips GitHub lookups for repos that already have a stored description,
HTTPHubClient remembers completed lookups per process, and a 403/429 stops the
enrichment loop (honouring X-RateLimit-Reset). Stops starving settlement
verification of the shared anonymous GitHub bucket.

Closes #479

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 25, 2026
@kubestellar-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 25, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 25, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the tree at 2a2f3ea, not just the diff. This looks correct to me.

  • Registry.Merge (pkg/registry/registry.go:302-304) already keeps the stored description when the incoming profile's is empty, so skipping enrichment for known repos via hasDescription cannot blank anything.
  • hasDescription takes r.mu.RLock inside ListReposSkippingDescriptions, which runs before Merge takes the write lock (Sync, lines 338-348) — no lock re-entry.
  • looked is only set after a non-403/429 response (lines 211-216), so a rate-limited repo is retried after notBefore; an empty/404 result is not, matching the PR body.

One low-severity edge case worth a maintainer's call, not a blocker:

pkg/registry/registry.go:201 — every 403 is treated as a rate limit. GitHub also returns 403 for a single repo that is DMCA-blocked or otherwise access-restricted, and that response still carries X-RateLimit-Reset. Because the loop returns before marking that repo looked, such a repo stalls enrichment for every repo after it in hub order, every sync, for as long as it stays blocked. Checking X-RateLimit-Remaining == "0" (or Retry-After) before treating a 403 as a limit, and otherwise marking the repo looked and continuing, would confine the damage to that one repo. Fine as a follow-up.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (correctness) — no blocking findings; one advisory.

  • pkg/registry/registry.go:200-209: the 403 branch treats any 403 as a rate limit and stores X-RateLimit-Reset into notBefore. GitHub sends that header on every response, including non-quota 403s (DMCA/abuse-blocked repo). Such a repo is never added to looked (line 211-215 is after the return), so each sync re-hits it, re-arms notBefore, and enrichment for every repo after it in hub order never runs. Issue #479 asked for "stop on first 403/429", so this is within contract — but gating the back-off on X-RateLimit-Remaining == "0" (or 429) and otherwise marking the repo looked and continuing would close the gap. Tests (registry_test.go:88-109) don't exercise X-RateLimit-Reset/notBefore either.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the tree at the PR head, not just the diff. This looks correct to me and matches what #479 asked for.

Verified

  • Registry.Merge (pkg/registry/registry.go:302-304) already keeps existing.Description when the incoming one is blank, so skipping enrichment via r.hasDescription can never wipe a stored description.
  • hasDescription goes through lookupLocked (registry.go:393-402), so the case-insensitive match the rest of the registry uses applies here too.
  • HTTPHubClient is constructed once in cmd/dibs/main.go:207 and reused by the 5-minute sync loop at main.go:214, so the per-process looked set and notBefore actually persist across syncs.
  • Network errors (client.Do → continue, registry.go:199-201) do not mark the repo looked, so a transient failure is retried; only a completed response is remembered. That's the behaviour the issue asked for.
  • The three new tests exercise exactly the three claims in the body (once-per-repo, stop on first 403, no retry on empty description). CI is green.

One low-severity edge, not a blocker (registry.go:202-211): the 403 branch treats every 403 as a rate limit. GitHub also returns 403 for a repo whose access is blocked (DMCA etc.), and that response still carries X-RateLimit-Reset. Because the branch returns before marking the repo looked, a single such repo early in the hub list would stop enrichment for every repo after it, and park notBefore up to an hour ahead, on every sync. Settlement verification is still protected (that was the actual bug), so this only degrades descriptions. If you want to harden it, distinguishing via X-RateLimit-Remaining: 0 before setting notBefore would do it — a follow-up, not a reason to hold this.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 8870ed6 into main Oct 25, 2026
6 of 7 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-479 branch October 25, 2026 12:58
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#4802a2f3ea` sha256:73e2f65f069069093a04784563c53e9c17ecfa4bd378db49ed389a29e2c2db73 — download from the dashboard or `hivectl review evidence #480`.

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 25, 2026
hivecommons-hive Bot added a commit that referenced this pull request Oct 25, 2026
…umes (#482)

Cover the notBefore early exit and the X-RateLimit-Reset parse added in #480:
a future reset makes later syncs skip GitHub without marking the repo as
looked up, a past reset does not block the next sync, and an unparseable
header stops the loop without arming the back-off.

Closes #481

Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] registry: description enrichment re-queries GitHub anonymously every sync and starves settlement verification

0 participants