Skip to content

ci: retry rate-limited fork suites instead of failing the run - #137

Merged
1kresh merged 1 commit into
mainfrom
delegator-simplify
Aug 7, 2026
Merged

ci: retry rate-limited fork suites instead of failing the run#137
1kresh merged 1 commit into
mainfrom
delegator-simplify

Conversation

@1kresh

@1kresh 1kresh commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

The first test run on main after every merge fails, and the cause is not the code under test.

Actions scopes caches per ref, and main cannot read cache entries a PR branch wrote. So the post-merge run starts with a cold Foundry RPC cache, the mainnet suites fetch all fork state at once, and the provider answers HTTP 429:

[FAIL: vm.createSelectFork: could not instantiate forked environment with provider
eth-mainnet.g.alchemy.com; Max retries exceeded HTTP error 429

Same commits, two different outcomes — the only difference is the cache:

PR run (green) main push (red)
Foundry RPC cache Cache restored successfully, ~2 MB Cache not found for input keys
Result all suites pass, 2m0s 429 across the suite, 6m8s

This is structural, not a one-off: run 31197476561 (merge of #136) and run 31138998090 (merge of #135) failed the same way.

Note the Unauthorized request to eth-mainnet.g.alchemy.com annotations on those runs are bullfrog audit-mode noise and a red herring — the requests went through, which is how the provider was able to return 429.

Fix

Retry the forge step up to 3 times with 60s/120s backoff. Foundry persists every response it does receive to ~/.foundry/cache/rpc, so each attempt starts warmer and needs fewer live requests. Retrying in-job converges where a fresh workflow run cannot, because a fresh run on main is cold again by construction.

Only 429s are retried. Any other failure exits on the first attempt, so a real regression still reports immediately rather than after three passes and three minutes of backoff.

Verification

Rendered the step out of the workflow with the matrix expression substituted exactly as Actions would, then ran it under bash -e against a stubbed forge:

Scenario forge invocations Exit
429 twice, then passes 3 0
EvmError: Revert, no 429 1 — no retry 1
429 on every attempt 3 1

This also confirmed the matrix command's single-quoted regexes ('.*Mainnet.*') survive substitution into the loop as single arguments.

Not covered here

This makes cold starts survivable, not rare. A scheduled warm-up run on main would fix the other half of the scoping problem — main's caches are visible to all branches, so new PR branches would stop starting cold too. Worth stacking separately.

🤖 Generated with Claude Code

The first `test` run on main after every merge starts with a cold Foundry RPC
cache: Actions scopes caches per ref, and main cannot read the entries a PR
branch wrote. Cold, the mainnet suites fetch all fork state at once and the
provider answers HTTP 429, so vm.createSelectFork and vm.deployCode fail across
the suite. That failed run 31197476561 (merge of #136) and run 31138998090
(merge of #135); the same commits passed on their PRs, where the cache restored
warm (~2 MB).

Retry the forge step up to 3 times with 60s/120s backoff. Foundry persists every
response it does receive to ~/.foundry/cache/rpc, so each attempt starts warmer
and needs fewer live requests - retrying in-job converges where a fresh run
cannot. Only 429s are retried; any other failure exits on the first attempt, so
a real regression still reports immediately instead of after three passes.

Verified by rendering the step out of the workflow with the matrix expression
substituted as Actions would, then running it under `bash -e` against a stubbed
forge: 429-then-pass exits 0 after 3 invocations, a plain revert exits 1 after 1
invocation with no retry, and persistent 429s exit 1 after 3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@1kresh
1kresh requested a review from a team as a code owner August 7, 2026 22:25
@1kresh
1kresh merged commit 25a065a into main Aug 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant