Skip to content

fix(unibtc): retry flaky Bedrock reserve API and debounce its error alerts - #373

Merged
spalen0 merged 1 commit into
mainfrom
unibtc-fix
Sep 23, 2026
Merged

spalen0 merged 1 commit into
mainfrom
unibtc-fix

Conversation

@spalen0

@spalen0 spalen0 commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

Last 12h of monitoring.service logs, unibtc task:

Issue Runs Effect
Rejecting Bedrock reserve API response: ['BOB (chain 60808) missing from supplies'] 19:05, 23:05, 01:05, 04:05 Telegram error, PoR coverage and feeder gap skipped
Request failed for https://affiliate-api-eosin.vercel.app/...: Read timed out. (read timeout=30) 22:05, 03:05, 06:05 Telegram "API unavailable", same checks skipped
uniBTC mint baseline missing (1h=True 24h=False) every run Warning noise: history is younger than the 24h window since the #362 deploy

7 of 12 runs skipped the API-dependent checks.

Root cause

I polled the API for about 20 minutes. The backend recomputes every 5 minutes (06:25:07, 06:35:08, 06:40:08 UTC, plus an extra one at 06:33:09).

  • A recompute takes 20–45s, and requests hang while it runs. Our hourly run at :05 often starts during a recompute, which causes the timeouts.
  • One recompute can drop a chain and the next one is complete. Observed at 06:33:09: Berachain was missing and total_supply was 4395.41. At 06:35:08 it was complete again (4546.42).

Fix

  • Retry in-run (up to 3 attempts): 30s after a failed request, and 5.5 min after a rejected snapshot, because a bad snapshot is served until the next recompute. Worst-case run time is about 7 min, within the hourly budget. Validation is unchanged: a rejected snapshot is still never used.
  • Debounce errors: the Telegram error is sent only when the API stays unusable for 3 consecutive runs. UNIBTC_API_FAILURE_STREAK resets on success. Skipping PoR coverage for an hour is harmless because the PoR feed updates daily.
  • Request errors and parse errors now raise ApiStatsError instead of sending separate messages. The fetch uses requests directly, so a timeout no longer logs an extra ERROR line.
  • Mint baseline: logged at info while snapshot history is still warming up (no snapshots yet, or the 1h baseline exists but no snapshot is 20h old yet). A real gap in an established history still warns.

Testing

  • uv run pytest: 1344 passed. New tests cover retry order and delays, the streak alerting only on the 3rd run, timeout wrapping, and warm-up detection.
  • ruff check, ruff format: clean.
  • Live load_api_stats against the real API with an isolated cache: usable, 0.3s.

🤖 Generated with Claude Code

…lerts

The Bedrock API recomputes every 5 minutes (at :x0:08/:x5:08). The hourly
run at :05 often hits a recompute: requests hang 20-45s (read timeouts) or
get a snapshot that dropped a chain (BOB, Berachain), and the next
recompute is complete. Over the last 12h 7 of 12 runs skipped PoR coverage
and each sent a Telegram error.

- Retry within the run: 30s after a failed request, 5.5 min after a
  rejected snapshot so the retry sees the next recompute.
- Send the error only after 3 consecutive unusable runs.
- Log the 24h mint-baseline gap at info while snapshot history is still
  warming up; keep the warning for real gaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@spalen0
spalen0 marked this pull request as ready for review September 23, 2026 07:10
@spalen0
spalen0 merged commit 130fadc into main Sep 23, 2026
3 checks passed
@spalen0
spalen0 deleted the unibtc-fix branch September 23, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant