[nimrod-teich]

#Closes MAG-4639

Jira ticket: MAG-4639

Note

Left unmerged for review. It changes how the health check, and so /readyz, judges a chain after one of its tries stalls.

Stack (merge top to bottom):

1. fix(readiness): give each health-check try its own relay processor (MAG-4639) #584 fix(readiness): give each health-check try its own relay processor (MAG-4639)
2. feat(readiness): readiness follows reachability, three-valued probe (MAG-4641) #586 feat(readiness): readiness follows reachability, three-valued probe (MAG-4641)

Why

The health check (sendRelayWithRetries) makes up to six tries, each bounded by a 5 s deadline, and every try shared one relay processor. An attempt cut off at its try's deadline still delivers its response, late, in the defer that ends it. The next try's WaitForResults returns on the first response it reads, so it took that late response as its own answer within microseconds, gave up, and cut off its own attempt. That attempt's late response went to the try after it, and so on down all six. Only the first try ever waited on its own attempt.

The test below reproduces it against main:

[-] failed sending init relay  cancelled relay processor {total responses:0}             try 1, stalled 5 s
[-] failed sending init relay  ... context deadline exceeded: Caller's context ...      try 2 reads try 1's
[-] failed sending init relay  ... context canceled: Request context was canceled ...   tries 3 to 6, each the previous try's

The first try stalls, so one stalled upstream fails the whole check even when the other upstreams answer. Since MAG-4617 the probe alone decides readiness and runs within 15 s of a failed client relay, so on a quiet chain one stall can turn the interface unhealthy until the next probe.

What changed

• One processor per try (rpcsmartrouter_server.go, sendRelayWithRetries), built on that try's context. A late response lands in a processor nothing reads.
• The state machine and its used providers stay shared, so each try still goes to an endpoint the earlier tries did not use, as before.
• Each try reports its own result. With initialRelays, a later try no longer re-reports an earlier try's success as its own.

How to verify

go test -race -count=1 ./protocol/rpcsmartrouter/ -run 'TestSendCraftedRelays|TestHealthRelay'

TestSendCraftedRelays_AStalledTryDoesNotAnswerTheNext (about 5 s) puts an upstream that stalls on its first request, then answers each request after 50 ms, behind three providers. It fails on main at ec999d5 and passes here: 5 of 5 under -race. The 50 ms is slower than reading a response already waiting and far faster than the try deadline, so the shared processor fails the check every time.

The full go test ./protocol/rpcsmartrouter/ (without -race) passes on the branch in 254 s.

TestHealthCommand_RejectsUnbootableCrossValidation can fail under -race when it runs after any crafted-relay test. The health command sets the global log level while a previous test's connector goroutine is still logging its shutdown. It fails 2 of 4 runs on main paired with TestSendCraftedRelays_ResultAnswerIsHealthy, and 2 of 4 here paired with the new test, so this change does not cause it.

Screenshots

No UI change. The evidence is the log excerpt under Why.