ci: retry rate-limited fork suites instead of failing the run - #137
Merged
Conversation
The first `test` run on main after every merge starts with a cold Foundry RPC cache: Actions scopes caches per ref, and main cannot read the entries a PR branch wrote. Cold, the mainnet suites fetch all fork state at once and the provider answers HTTP 429, so vm.createSelectFork and vm.deployCode fail across the suite. That failed run 31197476561 (merge of #136) and run 31138998090 (merge of #135); the same commits passed on their PRs, where the cache restored warm (~2 MB). Retry the forge step up to 3 times with 60s/120s backoff. Foundry persists every response it does receive to ~/.foundry/cache/rpc, so each attempt starts warmer and needs fewer live requests - retrying in-job converges where a fresh run cannot. Only 429s are retried; any other failure exits on the first attempt, so a real regression still reports immediately instead of after three passes. Verified by rendering the step out of the workflow with the matrix expression substituted as Actions would, then running it under `bash -e` against a stubbed forge: 429-then-pass exits 0 after 3 invocations, a plain revert exits 1 after 1 invocation with no retry, and persistent 429s exit 1 after 3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first
testrun onmainafter every merge fails, and the cause is not the code under test.Actions scopes caches per ref, and main cannot read cache entries a PR branch wrote. So the post-merge run starts with a cold Foundry RPC cache, the mainnet suites fetch all fork state at once, and the provider answers HTTP 429:
Same commits, two different outcomes — the only difference is the cache:
Cache restored successfully, ~2 MBCache not found for input keysThis is structural, not a one-off: run 31197476561 (merge of #136) and run 31138998090 (merge of #135) failed the same way.
Note the
Unauthorized request to eth-mainnet.g.alchemy.comannotations on those runs are bullfrog audit-mode noise and a red herring — the requests went through, which is how the provider was able to return 429.Fix
Retry the forge step up to 3 times with 60s/120s backoff. Foundry persists every response it does receive to
~/.foundry/cache/rpc, so each attempt starts warmer and needs fewer live requests. Retrying in-job converges where a fresh workflow run cannot, because a fresh run on main is cold again by construction.Only 429s are retried. Any other failure exits on the first attempt, so a real regression still reports immediately rather than after three passes and three minutes of backoff.
Verification
Rendered the step out of the workflow with the matrix expression substituted exactly as Actions would, then ran it under
bash -eagainst a stubbedforge:EvmError: Revert, no 429This also confirmed the matrix command's single-quoted regexes (
'.*Mainnet.*') survive substitution into the loop as single arguments.Not covered here
This makes cold starts survivable, not rare. A scheduled warm-up run on
mainwould fix the other half of the scoping problem — main's caches are visible to all branches, so new PR branches would stop starting cold too. Worth stacking separately.🤖 Generated with Claude Code