fix(unibtc): retry flaky Bedrock reserve API and debounce its error alerts - #373
Merged
Merged
Conversation
…lerts The Bedrock API recomputes every 5 minutes (at :x0:08/:x5:08). The hourly run at :05 often hits a recompute: requests hang 20-45s (read timeouts) or get a snapshot that dropped a chain (BOB, Berachain), and the next recompute is complete. Over the last 12h 7 of 12 runs skipped PoR coverage and each sent a Telegram error. - Retry within the run: 30s after a failed request, 5.5 min after a rejected snapshot so the retry sees the next recompute. - Send the error only after 3 consecutive unusable runs. - Log the 24h mint-baseline gap at info while snapshot history is still warming up; keep the warning for real gaps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Last 12h of
monitoring.servicelogs,unibtctask:Rejecting Bedrock reserve API response: ['BOB (chain 60808) missing from supplies']Request failed for https://affiliate-api-eosin.vercel.app/...: Read timed out. (read timeout=30)uniBTC mint baseline missing (1h=True 24h=False)7 of 12 runs skipped the API-dependent checks.
Root cause
I polled the API for about 20 minutes. The backend recomputes every 5 minutes (06:25:07, 06:35:08, 06:40:08 UTC, plus an extra one at 06:33:09).
:05often starts during a recompute, which causes the timeouts.Fix
UNIBTC_API_FAILURE_STREAKresets on success. Skipping PoR coverage for an hour is harmless because the PoR feed updates daily.ApiStatsErrorinstead of sending separate messages. The fetch usesrequestsdirectly, so a timeout no longer logs an extra ERROR line.Testing
uv run pytest: 1344 passed. New tests cover retry order and delays, the streak alerting only on the 3rd run, timeout wrapping, and warm-up detection.ruff check,ruff format: clean.load_api_statsagainst the real API with an isolated cache: usable, 0.3s.🤖 Generated with Claude Code