Skip to content

[scanner] registry: description enrichment re-queries GitHub anonymously every sync and starves settlement verification #479

Closed

Description

@hivecommons-hive

Problem

HTTPHubClient.enrichDescriptions (pkg/registry/registry.go:138-176) runs on every Registry.Sync — every 5 minutes (cmd/dibs/main.go:46, registrySyncInterval) — and issues one GET /repos/{org}/{repo} to api.github.com for every hub repo whose hub-supplied description is blank.

Two things make that wasteful and then harmful:

  1. It never consults the registry. Merge already keeps the stored description when the incoming one is blank (registry.go:252-254), so a repo enriched on sync ✨ Wave 1: ideate foundation — scaffold, hub auth bridge, idea CRUD, repo registry #1 is re-fetched from GitHub on sync ✨ Wave 2: LLM matching, swipe UX, settlement to GitHub issues, notifications #2, ✨ Wave 2: LLM matching, swipe UX, settlement to credited GitHub issues, notifications #3, … forever. Repos whose GitHub description is genuinely empty are also re-fetched every sync and never succeed.
  2. It is unauthenticated in the default deployment. Token comes from DIBS_GITHUB_TOKEN (main.go:207), which the default matchmaker flow leaves unset (pkg/settle/settle.go package doc). Anonymous GitHub API is 60 requests/hour per IP. With 12 syncs/hour, five blank-description repos already saturate the bucket; beyond that every call 403s and enrichDescriptions silently continues.

The user-visible damage is not the missing descriptions — it is that settle.HTTPClient.GetIssue (pkg/settle/verify.go:66), which verifies the issue URL an ideator pastes back to settle, uses the same anonymous bucket from the same IP. Once the enrichment loop has burned the quota, handleConfirmIssue returns 502 "could not verify the issue with GitHub — try again shortly" (pkg/api/settlement.go:221-223) for everyone until the window resets.

Expected

  • Enrich only repos the registry does not already have a description for (pass the known descriptions in, or do the enrichment after Merge against the stored state).
  • Remember repos whose GitHub lookup returned 200 with an empty description (or any non-200) and do not retry them every sync — a per-process "tried" set, or a retry-after on 403/429 reading X-RateLimit-Reset, is enough.
  • Stop the loop on the first 403/429 from GitHub rather than spending the rest of the bucket.

Evidence

  • pkg/registry/registry.go:134 — c.enrichDescriptions(ctx, repos) unconditionally inside ListRepos.
  • pkg/registry/registry.go:150-152 — skip condition looks only at the incoming hub payload.
  • pkg/registry/registry.go:252 — Merge preserves existing.Description, proving the refetch is redundant.
  • cmd/dibs/main.go:46,207 — 5-minute cadence, token optional.
  • pkg/api/settlement.go:216-223 — settlement verification path that inherits the exhausted bucket.

Validation

Table-driven test in pkg/registry with an httptest GitHub stub counting hits: two consecutive Sync calls against the same hub payload must hit GitHub once per repo, not twice; a stub answering 403 must stop the loop after the first refusal.

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    agent/scannerCreated by Hive for agent-filed issue provenance
    stage/020-claimedClaimed by the scanner or a relay contributor
    and removed
    stage/020-claimedClaimed by the scanner or a relay contributor
    on Oct 25, 2026
  2. added a commit that references this issue on Oct 25, 2026
    8870ed6
  3. added
    hive/likely-doneHive verified that a merged PR references or claims this issue; pending confirmation
    and removed on Oct 25, 2026
  4. hivecommons-hive commented on Oct 25, 2026

    @hivecommons-hive
    ContributorAuthor

    Fixed by #480 (#480, merged). Closing because Hive's PR claim metadata says this PR was the full fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/scannerCreated by Hive for agent-filed issue provenancehive/claimed-by-scannerhive/likely-doneHive verified that a merged PR references or claims this issue; pending confirmationkind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions