[nimrod-teich]

#Closes MAG-4641

Jira ticket: MAG-4641

Note

Left unmerged for review. The data plane: this changes when /readyz turns 503.

Merge order: after #585 (MAG-4355). Without it a refused gRPC dial reads as the node's answer, so a gRPC chain whose upstreams all refuse connections would keep its last verdict instead of turning unhealthy. The two branches merge cleanly together.

Stack (merge top to bottom):

1. fix(readiness): give each health-check try its own relay processor (MAG-4639) #584 fix(readiness): give each health-check try its own relay processor (MAG-4639)
2. feat(readiness): readiness follows reachability, three-valued probe (MAG-4641) #586 feat(readiness): readiness follows reachability, three-valued probe (MAG-4641)

Why

Since #581 a chain+interface turns unhealthy, and the pod NotReady, when its health probe gets anything but a result. That caught the replica that could not reach its upstreams in the null-route drill. It also catches an upstream that answers with an error, an error status or a rate limit. Every replica shares its upstreams and gets the same answer, so all of them left rotation together, and clients got connection errors from an empty Service where they used to get the upstream's reply. A rate limiter with a cooldown made readiness flap: pulled, traffic stops, the limit clears, back in, the limit trips again.

What differs between replicas is whether each one reaches its upstreams, and that is what readiness should say.

What changed

• Three-valued probe (metrics.ProbeVerdict, RelaysMonitor.recordProbeResult):
  • a result: healthy;
  • no answer at all (every try timed out, failed in transport, or had no upstream to go to): unhealthy;
  • an answer that is an error, an error status or a rate limit: inconclusive, and the chain keeps the health it had. The cadence follows the health: 5 min while healthy, --relays-health-unhealthy-interval while unhealthy.
• What counts as an answer needs positive evidence (upstreamAnswerFrom, protocolErrorIsAnAnswer): a node error, an upstream's HTTP status (JSON-RPC's HTTPStatusError; REST's node-error and rate-limit flags, since REST fails the attempt with a plain error), or a rate limit. Anything else is no answer, including a failure nobody has catalogued, so the rule fails closed. It reads relay_end's inputs, not its attempt outcomes: relay_end files a JSON-RPC 5xx under transport_error.
• The probe (sendRelayWithRetries) aggregates its tries: any result is healthy, otherwise any answer is inconclusive, otherwise unreachable. Each try reads only its own processor (fix(readiness): give each health-check try its own relay processor (MAG-4639) #584), so a try that timed out holds nothing and counts as no answer.
• Client relays (relayHealthSignalOf, read from the relay's attempts rather than its returned result):
  • an upstream's result defers the probe (LogRelay), as before;
  • no upstream answering brings the probe within 15 s (ProbeSoon), as before;
  • an upstream's error answer now does nothing. A revert or a bad parameter no longer schedules a probe;
  • a cache hit or a client that hung up does nothing, as before.
• The Health Check Interval Check log line gains verdict; health result is now the health after the probe.
• docs/METRICS.md ("When they change") describes the three verdicts; docs/LOCAL-COMPOSE.md and docs/RESP-CACHE.md say what a lasting 503 means.

Behaviour to know about

• An upstream outage every replica shares keeps the pods Ready. Clients get the upstream's error or rate limit from the router, not a connection error from an empty Service. The smartrouter_overall_health alert does not fire for it; the tip, stuck-provider and non-200 alerts do.
• A fault no replica can see past still pulls every replica: every upstream refusing connections, a DNS outage, an expired upstream certificate. From inside one replica that is no answer at all, the same as its own egress failing, and the two cannot be told apart.
• Open call: an inconclusive probe after an unhealthy verdict keeps the chain unhealthy until a probe gets a result. After a cut that every replica of a router went through, upstreams that then answer only with errors would keep all of them out. Treating an answer as healthy in that direction instead is one line in recordProbeResult (case ProbeHealthy, ProbeInconclusive when unhealthy); I kept the agreed design and left the call to review.

How to verify

go test -race -count=1 ./protocol/metrics/
go test -race -count=1 ./protocol/rpcsmartrouter/ -run 'TestUpstreamAnswerFrom|TestRelayHealthSignalOf|TestSendParsedRelay_FeedsReadiness|TestSendCraftedRelays'

TestSendCraftedRelays_VerdictFollowsReachability drives the real probe against a live upstream, about 8 s:

Upstream	Verdict
a result	healthy
a JSON-RPC error body (HTTP 200)	inconclusive
HTTP 503 with an HTML body	inconclusive
HTTP 403	inconclusive
HTTP 429	inconclusive
the connection dropped without an answer	unreachable
the port closed (connection refused)	unreachable
no answer until the try deadline	unreachable

Each rule was mutated and the change turned these tests red, then restored:

Mutation	Test that fails
no HTTPStatusError evidence	VerdictFollowsReachability/an_HTTP_503, UpstreamAnswerFrom/a_JSON-RPC_error_status, FeedsReadiness/an_error_status…
no REST flag evidence	UpstreamAnswerFrom/a_REST_error_status
no rate-limit evidence	UpstreamAnswerFrom/a_typed_rate_limit, …/a_classified_rate_limit
every failed attempt counts as an answer	VerdictFollowsReachability/a_dropped_connection, …/a_refused_connection, four UpstreamAnswerFrom rows, FeedsReadiness/no_answer…
the probe never inconclusive (#581's rule)	VerdictFollowsReachability: error body, 503, 403, rate limit
a client error answer brings the probe forward (#581's rule)	FeedsReadiness/an_error_answer…, …/an_error_status…
a client relay no upstream answered does nothing	FeedsReadiness/no_answer…
inconclusive marks unhealthy	TestRelaysMonitor_InconclusiveProbeKeepsAHealthyChainHealthy
inconclusive marks healthy	TestRelaysMonitor_InconclusiveProbeKeepsAnUnhealthyChainUnhealthy

REST's flags and a rate limit carried only by the classifier are covered by the unit table, not end to end: the probe fleet is JSON-RPC, where a 429 also arrives as an HTTPStatusError.

With #585 merged in locally, its real-gRPC harness gives no answer for a refused dial and an answer for a node-sent UNAVAILABLE under this rule. That check was not committed: it needs #585's helpers.

Full runs: go vet ./... passes; go test ./protocol/rpcsmartrouter/ passes in full without -race (253 s); protocol/metrics passes with -race.

protocol/metrics under -race -count=3 fails TestGetUniqueGuidResponseDeterministic on this branch and on main alike: TestHappyFlow leaves a monitor logging on context.Background() while that test sets the global log level.

Screenshots

No UI change.