Evict upstream connections whose RPC requests go unanswered (Sapphire zombie-connection watchdog)

[dominicletz]

Problem

During the Oasis Sapphire incident the primary provider (wss://sapphire.oasis.io/ws) turned into a zombie connection: it kept pushing newHeads frames but stopped answering JSON-RPC requests, while the fallback (spectrum-02.simplystaking.xyz) had died at the same moment (block flow stopped, reconnects timed out).

Because connection health is judged only by block flow, the primary passed every staleness check and stayed in rotation. Evidence from the incident (23:59\u201300:08 local):

• The shared chain RPC log shows zero completed Sapphire responses for 8m51s (request ids 409\u2013678 sent, never answered), then one straggler \u22483 min after its caller had already died.
• 141 Timeout calling {:global, {RemoteChain.NodeProxy, Chains.OasisSapphire}} {:rpc, "eth_getBlockByNumber", \u2026} errors in that window; each killed the EdgeV2 connection, so devices reconnected and retried \u2014 a self-perpetuating storm.
• 1000+ Peer \u2026 handle_async_msg(["sapphire:getblockheader", \u2026]) stuck for 10000 warnings: peers requesting the same block serialize behind the per-block :global.trans lock in RPCCache.rpc_direct, each taking a turn at a doomed 25s upstream call.
• NodeProxy itself was healthy throughout (idle gen_server:loop stack at snapshot times, sends succeeding) \u2014 nothing anywhere times out an unanswered upstream request.

The incident only ended with a manual node restart; fresh connections to the same provider worked immediately.

Changes

lib/remote_chain/node_proxy.ex \u2014 unresponsive-request watchdog

• New :watchdog tick every 25s (armed in init/1, self re-arming; interval runtime-overridable via set_watchdog_interval_ms/1 for tests, same pattern as RPCCache.set_refresh_debounce_ms).
• prune_unresponsive_connections/1 evicts any WSConn (primary or fallback) that has an in-flight request unanswered for 5\u00d7 the caller timeout (125s) \u2014 by then five generations of callers have already died on their own 25s timeout, so no useful answer can still arrive.
• Eviction closes the connection, clears its (previously leaked) requests entries, replies {:error, :disconnect} to orphaned callers, logs Evicting unresponsive WSConn \u2026 (request unanswered for > 125000ms) and schedules ensure_connections so ChainList re-tests/refills the pool.

Profiler bump dfc953f \u2192 a9fedba (v0.4.7, dominicletz/profiler)

• Profiler.warn_if_stuck custom callbacks previously ran in the spawned monitor process, so format_stacktrace(self()) captured the monitor\u2019s trace \u2014 which is why every \u201cstuck for 10000\u201d warning in the incident showed a useless single frame. Callbacks now receive the monitored pid (arity-1; arity-0 still supported; covered by new tests in the profiler repo, tagged v0.4.7).
• lib/network/edge_v2.ex now uses the pid argument so future stuck warnings are actually diagnosable.

Tests

New describe "prune_unresponsive_connections/1 watchdog" in test/remote_chain/node_proxy_test.exs (named after the failure they prevent):

• watchdog threshold is 5\u00d7 the 25s caller timeout
• zombie connection with an overdue request is evicted, requests cleared, orphaned caller answered {:error, :disconnect}
• connections with fresh requests are untouched
• unresponsive fallback is evicted (fallback/fallback_url reset)
• handle_info(:watchdog) prunes and re-arms itself

Verification

mix lint          # 0 errors
mix test test/remote_chain/                    # 108 passed (incl. 5 new)
mix test test/network/edge_v2_test.exs         # 3 passed
mix test test/network/edge_v2_message_e2e_test.exs  # 9 passed, 2 skipped