Repository navigation
fix(health-probe): bound gRPC clientPool to stop memory leak - #54
Merged
Merged
Conversation
The health-probe clientPool created one gRPC client per provider endpoint and never closed or evicted any of them. Each client owns a live HTTP/2 channel (socket, keepalive timers, native buffers, file descriptors), and provider endpoints churn continuously (restarts, re-IPs, stake changes), so the pool grew without bound. On the 512MB info-mainnet Fargate task this manifested as a monotonic heap climb (~40%→97% over ~36h, no GC recovery) ending in repeated OOM-kills and task restarts. Bound the pool with an LRU cap (HEALTH_PROBE_CLIENT_POOL_MAX, default 256) and close the least-recently-used channel on overflow; a cache hit re-inserts the key so active endpoints stay warm. Client creation is now injectable so the eviction/close behaviour is unit-tested without opening a real channel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
clientPoolcreated one gRPC client per provider endpoint and never closed or evicted any of them. Each client owns a live HTTP/2 channel (socket, keepalive timers, native buffers, file descriptors); provider endpoints churn continuously (restarts, re-IPs, stake changes each epoch), so the pool grew without bound.info-mainnetFargate task this showed up as a monotonic heap climb — ~40% → 97% over ~36 h with no GC recovery — ending in repeated OOM-kills (exit 137) and task restarts (visible as the ~05/28 cliff and clustered task replacements in the ECS service events).HEALTH_PROBE_CLIENT_POOL_MAX, default 256) andclient.close()the least-recently-used channel on overflow. A cache hit re-inserts the key so actively-probed endpoints stay warm. Client creation is now injectable so the eviction/close behaviour is unit-tested without opening a real channel.probeProviderand production behaviour are otherwise unchanged.Evidence (CloudWatch, ECS
info-mainnet/api, us-east-1)Leak located at
apps/api/src/services/grpc-probe.ts—clientPoolkeyed by endpoint, driven by the in-processrunProbeLoop(ENABLE_HEALTH_PROBE=trueinmainnet.tfvars).Test plan
pnpm --filter @info/api typecheck— cleangrpc-probe-pool.test.ts— 7 cases: client reuse, distinct endpoints, pool stays ≤ cap, every evicted client.close()d exactly once, LRU touch keeps recent endpoints warm, reset closes allpnpm --filter @info/api test— 134/134 passingDeploy note
This is the code fix; info-mainnet is still leaking until this image ships. A
--force-new-deploymentcan reset the heap immediately if an OOM is imminent before merge.Follow-up (not in this PR)
Secondary unbounded cache
ibcTraceCache(apps/api/src/rpc/pricing.ts:171) — slow, not the OOM driver; can be bounded separately.