Cache endpoint list so ws_endpoints never blocks the GenServer

[dominicletz]

Root cause

ChainList.filter_endpoints/2 spawned a Task.async_stream over the chainlist and ran the full health probe inline. The probe can take 30+ seconds when providers are slow, and it ran in whatever process called RemoteChain.ws_endpoints/1 — which is the NodeProxy GenServer. While blocked, the GenServer could not process :new_block messages, so lastblocks/1 went stale, every live WSConn was evicted as data-stale, and the eviction triggered another slow ws_endpoints call. On us1 Base the GenServer sat inside filter_endpoints for ~40 hours with 137k messages queued. Full diagnosis: docs/specs/plan-endpoint-list-cache.md.

Fix

Stale-while-revalidate cache in front of filter_endpoints/2 (60s TTL, keyed by chain_id). Warm callers return from ETS in microseconds; cold callers get a deduped best-effort list synchronously and schedule the probe in a detached Task. Debouncer.immediate/3 (with timeout: 0) coalesces concurrent refreshes so 20 cold callers produce one probe. Per-chain generation counter prevents zombie workers from resurrecting a cache that was explicitly cleared. Chainlist refresh (put_chains/2) invalidates the cache so newly-listed providers are picked up on the next call.

The cold path and the warm probe both apply the pocket.network / curie.radiumblock.co filter via a single rejected_provider?/1 helper; probe_pass/2 collapses filter + map into one Enum.flat_map. best_effort_urls/1 is reused by both paths.

Why Debouncer.immediate/3

The plan called for Debouncer.immediate2/3. In practice the closure runs synchronously inside the Debouncer's cast handling, which serialized the actual probe behind any other cast the GenServer was processing — enough to break the 2-second timeout in the "cold-cache callers trigger exactly one probe per chain" test. The implementation now wraps Debouncer.immediate/3 around Task.start: the Debouncer's closure only spawns the detached Task, so the cast handling returns immediately and the probe runs without contending with the Debouncer's mailbox. timeout: 0 makes the cooldown effectively instant: the timer fires immediately, clears the events entry, and the next sequential caller (after the timer) hits the empty-events branch and gets a fresh worker.

TOCTOU fix

The first commit had a TOCTOU window: the generation check ran once at the top of the closure, so a slow probe followed by a clear_chain_cache bump could let a stale write land. The second commit moved the generation check to right before the Globals.put in refresh_endpoint_cache/3, so a worker outliving its cache cannot resurrect the cache even if the generation was bumped mid-probe.

Out of scope (per plan)

Bug B — data_stale?/4 preferring lastblocks[url] over WSConn.lastblock_at/1 — is left as a follow-up. With the GenServer no longer blocked, the cache tracks real block arrivals and Bug B stops being reachable in practice.

Tests

• Cold cache returns synchronously (regression for the Base deadlock)
• Warm cache returns without probing
• Stale cache returns the stale value and schedules a refresh
• Concurrent cold-cache callers trigger exactly one probe (via Debouncer.immediate/3)
• refresh_chains/1 invalidates the endpoint cache for affected chains
• pocket.network / curie.radiumblock.co are dropped on the cold path
• A probe in flight at clear_chain_cache time does not resurrect the cache (TOCTOU regression)
• The pre-existing "drops chainlist URLs that fail the health probe" was rewritten to express the new contract (cold → refresh → filtered).

mix test test/remote_chain/ → 105/105 pass. mix lint → clean.