From 078e27767d5d5c0180400605bdf78afe5813ac70 Mon Sep 17 00:00:00 2001 From: Tomelia1999 Date: Sun, 25 Oct 2026 17:17:52 +0200 Subject: [PATCH 1/3] docs(stickiness): a pinned request fails instead of failing over (MAG-2280, MAG-4123) Three sentences promised that a pinned request falls over to the rest of the pool after an error. The router no longer does that: a request pinned by lava-select-provider or lava-stickiness is answered by its node or fails with that node's error, Lava-Retries 0, and the router neither retries nor hedges it elsewhere. The sticky-sessions page's "route to the session's node, or fail" now holds for errors and for the router's own freshness check too. Co-Authored-By: Claude Fable 5.1 --- docs/api/directives.md | 2 +- docs/configuration/failover/retry.md | 2 +- docs/configuration/sticky-sessions.md | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/api/directives.md b/docs/api/directives.md index 462bf92..7893dd0 100644 --- a/docs/api/directives.md +++ b/docs/api/directives.md @@ -28,7 +28,7 @@ Each is detailed below. The router's [response headers](#response-headers) carry lava-select-provider: ``` -Expresses a **preference** for one named upstream at node selection, bypassing the QoS optimizer. When the request reaches selection, the named upstream serves it. If it fails, [failover](../configuration/failover/index.md) policies still apply against the rest of the pool. +Pins the request to one named upstream at node selection, bypassing the QoS optimizer. When the request reaches selection, the named upstream serves it or the request fails: if the node answers with an error, that error is returned with `Lava-Retries: 0`, and the router neither retries nor hedges on another node. [Failover](../configuration/failover/index.md) policies do not apply to a pinned request. !!! warning "The cache answers first — a pinned request may never reach the named node" The response cache is consulted **before** node selection runs, and pinning does *not* diff --git a/docs/configuration/failover/retry.md b/docs/configuration/failover/retry.md index 1a464c3..3d23716 100644 --- a/docs/configuration/failover/retry.md +++ b/docs/configuration/failover/retry.md @@ -102,7 +102,7 @@ subscriptions, so a node that said stop is not re-probed on a fixed cadence eith ## Pinning to one node -The `lava-select-provider` header pins the request to a specific upstream. If that upstream fails, retry kicks in **on the rest of the pool** — pinning isn't a way to disable retry. See [Directives](../../api/directives.md). +The `lava-select-provider` header pins the request to a specific upstream, and a `lava-stickiness` session does the same for the node it chose. A pinned request is **never retried or hedged on another node**: if the pinned upstream answers with an error, that error is returned to the caller with `Lava-Retries: 0`, and if the router's own freshness check finds the pinned node too far behind the chain tip on a `latest` call, the request fails. See [Directives](../../api/directives.md) and [Sticky sessions](../sticky-sessions.md). ## Observability diff --git a/docs/configuration/sticky-sessions.md b/docs/configuration/sticky-sessions.md index b55c315..84078b2 100644 --- a/docs/configuration/sticky-sessions.md +++ b/docs/configuration/sticky-sessions.md @@ -96,7 +96,7 @@ With either missing, each replica keeps its own private table. The header still The guarantee is **route to the session's node, or fail** — never quietly serve the request from a different one. Substituting a node silently is the problem this feature exists to remove, so the router does not do it, even when another node could answer. -In practice that means a request fails if the session's node cannot serve it, or if the cache backend holding the session's choice is unreachable. +In practice a request fails if the session's node cannot serve it, if that node answers with an error, if the router's own freshness check finds the node too far behind the chain tip on a `latest` call, or if the cache backend holding the session's choice is unreachable. The session keeps its node through such a failure; the choice is remade only when the node leaves the pool or the session expires. !!! info "Requests without the header are never affected" This applies only to traffic carrying `lava-stickiness`. Everything else is served as usual. @@ -108,7 +108,7 @@ In practice that means a request fails if the session's node cannot serve it, or | [`lava-select-provider`](../api/directives.md#pin-to-a-specific-node) | Wins. Naming a node explicitly is the more specific instruction. | | [Cross-validation](failover/cross-validation.md) | Wins. Cross-validation needs several nodes, which is incompatible with pinning to one. | | [`lava-extension`](../api/directives.md#override-the-extension) | A session id used for both plain and `archive` calls gets an independent node for each. | -| [Retry](failover/retry.md) | A retry after a genuine error deliberately goes to a different node. | +| [Retry](failover/retry.md) | Never moves a sticky request. If the session's node answers with an error, that error is returned to the caller with `Lava-Retries: 0`; the router does not retry or hedge on another node. | ## Trade-offs From ad29b13dd4cfac4cf3f8ff670532830f69e90318 Mon Sep 17 00:00:00 2001 From: Tomelia1999 Date: Mon, 26 Oct 2026 21:41:41 +0200 Subject: [PATCH 2/3] docs(api): document lava-pin-allow-retry (MAG-2280) The opt-in that keeps a pin to the first attempt: retries and hedges may go to other nodes, the session keeps its node. Directive table row, its own section with when (not) to use it, and pointers from the sticky-sessions Retry row and the retry page. Co-Authored-By: Claude Opus 5.5 --- docs/api/directives.md | 21 +++++++++++++++++++++ docs/configuration/failover/retry.md | 2 +- docs/configuration/sticky-sessions.md | 2 +- 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/docs/api/directives.md b/docs/api/directives.md index 7893dd0..fec9b9e 100644 --- a/docs/api/directives.md +++ b/docs/api/directives.md @@ -13,6 +13,7 @@ Override Smart Router's default behaviour for a single request by setting reques |---|---| | `lava-select-provider` | [Prefer](#pin-to-a-specific-node) one named upstream — the cache is still consulted first. | | `lava-stickiness` | [Keep a session](#keep-a-session-on-one-node) on one upstream across replicas. | +| `lava-pin-allow-retry` | With a pin, [allow retries on other nodes](#allow-retries-on-a-pinned-request) after the first attempt. | | `lava-providers-block` | [Exclude](#steer-node-selection) named nodes (comma-separated). | | `lava-extension` | [Force an extension](#override-the-extension) such as `archive`. | | `lava-force-cache-refresh` | [Bypass the cache](#force-a-cache-refresh) and refresh the entry. | @@ -96,6 +97,26 @@ curl -X POST http://127.0.0.1:3360 \ Working across replicas requires a cache backend and `--shared-state`; without them the header only applies within a single replica. If the chosen node cannot serve a request, the request fails rather than being served elsewhere. See [Sticky sessions](../configuration/sticky-sessions.md) for the full behaviour and limits. +## Allow retries on a pinned request + +``` +lava-pin-allow-retry: true +``` + +Sent together with `lava-select-provider` or `lava-stickiness`. The pinned node still gets the first attempt, but the request is no longer "this node or fail": if the pinned node answers with an error, is too slow, or is too far behind the chain tip, the router may retry or hedge on another node, and `Lava-Retries` counts it. A sticky session keeps its node; the next request starts on it again. + +Only the value `true` (any case) turns it on. Without a pin header it has no effect. + +**When to use:** you prefer one node but would rather get an answer from another node than an error. Do not use it for a head-then-fetch sequence or for node-local state (filters, subscriptions): an answer from another node is exactly what the pin protects you from there. + +```bash +curl -X POST http://127.0.0.1:3360 \ + -H 'Content-Type: application/json' \ + -H 'lava-select-provider: my-eth-upstream-1' \ + -H 'lava-pin-allow-retry: true' \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' +``` + ## Force a cache refresh ``` diff --git a/docs/configuration/failover/retry.md b/docs/configuration/failover/retry.md index 3d23716..1b9600e 100644 --- a/docs/configuration/failover/retry.md +++ b/docs/configuration/failover/retry.md @@ -102,7 +102,7 @@ subscriptions, so a node that said stop is not re-probed on a fixed cadence eith ## Pinning to one node -The `lava-select-provider` header pins the request to a specific upstream, and a `lava-stickiness` session does the same for the node it chose. A pinned request is **never retried or hedged on another node**: if the pinned upstream answers with an error, that error is returned to the caller with `Lava-Retries: 0`, and if the router's own freshness check finds the pinned node too far behind the chain tip on a `latest` call, the request fails. See [Directives](../../api/directives.md) and [Sticky sessions](../sticky-sessions.md). +The `lava-select-provider` header pins the request to a specific upstream, and a `lava-stickiness` session does the same for the node it chose. A pinned request is **never retried or hedged on another node**: if the pinned upstream answers with an error, that error is returned to the caller with `Lava-Retries: 0`, and if the router's own freshness check finds the pinned node too far behind the chain tip on a `latest` call, the request fails. To keep the pinned node as the first choice but allow retries and hedges elsewhere, add [`lava-pin-allow-retry: true`](../../api/directives.md#allow-retries-on-a-pinned-request). See [Directives](../../api/directives.md) and [Sticky sessions](../sticky-sessions.md). ## Observability diff --git a/docs/configuration/sticky-sessions.md b/docs/configuration/sticky-sessions.md index 84078b2..124874c 100644 --- a/docs/configuration/sticky-sessions.md +++ b/docs/configuration/sticky-sessions.md @@ -108,7 +108,7 @@ In practice a request fails if the session's node cannot serve it, if that node | [`lava-select-provider`](../api/directives.md#pin-to-a-specific-node) | Wins. Naming a node explicitly is the more specific instruction. | | [Cross-validation](failover/cross-validation.md) | Wins. Cross-validation needs several nodes, which is incompatible with pinning to one. | | [`lava-extension`](../api/directives.md#override-the-extension) | A session id used for both plain and `archive` calls gets an independent node for each. | -| [Retry](failover/retry.md) | Never moves a sticky request. If the session's node answers with an error, that error is returned to the caller with `Lava-Retries: 0`; the router does not retry or hedge on another node. | +| [Retry](failover/retry.md) | Never moves a sticky request. If the session's node answers with an error, that error is returned to the caller with `Lava-Retries: 0`; the router does not retry or hedge on another node. Add [`lava-pin-allow-retry: true`](../api/directives.md#allow-retries-on-a-pinned-request) to let a failed request be retried elsewhere; the session keeps its node. | ## Trade-offs From 1afd9a37824a8fa5a12cf967c8be8a2e1cb2ad08 Mon Sep 17 00:00:00 2001 From: Tomelia1999 Date: Tue, 3 Nov 2026 17:35:47 +0200 Subject: [PATCH 3/3] docs(stickiness): sticky requests are strict, lava-select-provider keeps failover, lava-select-provider-strict opts in (MAG-4826) Rewrites this PR to the contract decided on MAG-4826, on top of the current main text: - directives: lava-select-provider keeps its failover text; new lava-select-provider-strict row and "Require the named node" section; lava-pin-allow-retry is gone. - failover/retry: sticky and strict selection are the two exceptions to "pinning does not turn off retry". - sticky-sessions: the full list of ways a sticky request fails (node error, busy node, lagging latest call, own node in lava-providers-block, cache backend down); the session keeps its node until it leaves the pool; the wrong "a retry goes to a different node on purpose" line is replaced. Co-Authored-By: Claude Opus 5.5 --- docs/api/directives.md | 32 ++++++++++++++++++++++++++- docs/configuration/failover/retry.md | 7 ++++++ docs/configuration/sticky-sessions.md | 14 +++++++++--- 3 files changed, 49 insertions(+), 4 deletions(-) diff --git a/docs/api/directives.md b/docs/api/directives.md index 13ce495..0a6bac7 100644 --- a/docs/api/directives.md +++ b/docs/api/directives.md @@ -14,6 +14,7 @@ You send them as HTTP headers, or as call metadata on a gRPC listener ([Over gRP | Header | Effect | |---|---| | `lava-select-provider` | [Prefer](#pin-to-a-specific-node) one named upstream. The cache is still checked first. | +| `lava-select-provider-strict` | With `lava-select-provider`, [require the named node](#require-the-named-node): it answers, or the request fails. | | `lava-stickiness` | [Keep a session](#keep-a-session-on-one-node) on one upstream across replicas. | | `lava-providers-block` | [Exclude](#steer-node-selection) named nodes (comma-separated). | | `lava-extension` | [Force an extension](#override-the-extension) such as `archive`. | @@ -32,6 +33,8 @@ lava-select-provider: This header sets a **preference** for one named upstream at node selection. It skips the QoS optimizer. When the request reaches selection, the named upstream serves it. If that upstream fails, [failover](../configuration/failover/index.md) policies still apply to the rest of the pool. +If no other node may answer the request, also send [`lava-select-provider-strict: true`](#require-the-named-node). + !!! warning "The cache answers first - a pinned request may never reach the named node" The router checks the response cache **before** node selection runs, and pinning does *not* bypass it. For a cacheable method with a warm cache entry, the response carries @@ -73,6 +76,33 @@ curl -X POST http://127.0.0.1:3360 \ If the node is unreachable, the request then fails with an error instead of returning a cached answer. +## Require the named node + +``` +lava-select-provider-strict: true +``` + +Send this header together with `lava-select-provider`. The named node then answers the request, or the request fails. No other node serves it: + +- If the node answers with an error, the router returns that error with `Lava-Retries: 0`. It does not retry the request on another node. +- If the node is slow, the router waits for it, up to the request's time budget. It does not send a hedge to another node. +- If the node cannot take the request, for example because it is blocked or busy, the request fails. +- On a call for the `latest` block, the router checks that the node is not too far behind the chain tip. If it is, the request fails. A call for a specific block number is not checked. + +This is how [`lava-stickiness`](#keep-a-session-on-one-node) always behaves. + +Only the value `true` (any letter case) turns it on. Without `lava-select-provider`, the header has no effect. [Cross-validation](#request-cross-validation) still overrides it, because cross-validation needs several nodes. + +**When to use:** a sequence of calls on a node you choose, where an answer from another node would be wrong. For example, you read the chain head from that node and then fetch that block, or you use state that exists only on that node. + +```bash +curl -X POST http://127.0.0.1:3360 \ + -H 'Content-Type: application/json' \ + -H 'lava-select-provider: my-eth-upstream-1' \ + -H 'lava-select-provider-strict: true' \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' +``` + ## Keep a session on one node ``` @@ -96,7 +126,7 @@ curl -X POST http://127.0.0.1:3360 \ -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' ``` -To work across replicas, the header needs a cache backend and `--shared-state`. Without them, it only applies within a single replica. If the chosen node cannot serve a request, the request fails. No other node serves it. See [Sticky sessions](../configuration/sticky-sessions.md) for the full behaviour and limits. +To work across replicas, the header needs a cache backend and `--shared-state`. Without them, it only applies within a single replica. If the chosen node cannot serve a request, or answers with an error, the request fails. No other node serves it, and the router does not retry it elsewhere. See [Sticky sessions](../configuration/sticky-sessions.md) for the full behaviour and limits. ## Force a cache refresh diff --git a/docs/configuration/failover/retry.md b/docs/configuration/failover/retry.md index f9f325e..81eb579 100644 --- a/docs/configuration/failover/retry.md +++ b/docs/configuration/failover/retry.md @@ -106,6 +106,13 @@ subscriptions, so a rate-limited node is not re-probed on a fixed schedule eithe The `lava-select-provider` header pins the request to a specific upstream. If that upstream fails, retry runs **on the rest of the pool**. Pinning does not turn off retry. See [Directives](../../api/directives.md). +Two kinds of request are the exception. They are **never retried or hedged on another node**: + +- a request with [`lava-stickiness`](../sticky-sessions.md); +- a request with `lava-select-provider` and [`lava-select-provider-strict: true`](../../api/directives.md#require-the-named-node). + +If the node answers with an error, the router returns that error to the caller with `Lava-Retries: 0`. + ## Observability | Metric | Meaning | diff --git a/docs/configuration/sticky-sessions.md b/docs/configuration/sticky-sessions.md index eebda65..63b7a21 100644 --- a/docs/configuration/sticky-sessions.md +++ b/docs/configuration/sticky-sessions.md @@ -96,7 +96,15 @@ If either is missing, each replica keeps its own private table. The header still The guarantee is **route to the session's node, or fail**. The router never quietly serves the request from a different node. A silent switch to another node is the problem this feature exists to remove, so the router does not do it, even when another node could answer. -A request fails if the session's node cannot serve it, or if the cache backend that holds the session's choice is unreachable. +A request fails in these cases: + +- The session's node answers with an error. The router returns that error with `Lava-Retries: 0`. It does not retry or hedge the request on another node. +- The session's node is busy: it has used its request budget for the period, it has no free session, or it has no connection up. +- On a call for the `latest` block, the router finds the session's node too far behind the chain tip. A call for a specific block number is not checked. +- The request also lists the session's node in [`lava-providers-block`](../api/directives.md#steer-node-selection). +- The cache backend that holds the session's choice is unreachable. + +The session keeps its node through all of these. The router picks a new node for the session only when its node leaves the pool, for example because the router blocked it as unhealthy, or when the session expires. !!! info "Requests without the header are never affected" This applies only to traffic that carries `lava-stickiness`. All other traffic is served as usual. @@ -105,10 +113,10 @@ A request fails if the session's node cannot serve it, or if the cache backend t | Directive | Behaviour | |---|---| -| [`lava-select-provider`](../api/directives.md#pin-to-a-specific-node) | Wins. Naming a node explicitly is the more specific instruction. | +| [`lava-select-provider`](../api/directives.md#pin-to-a-specific-node) | Wins the choice of node. Naming a node explicitly is the more specific instruction. The request still behaves as a sticky one: that node answers, or the request fails. | | [Cross-validation](failover/cross-validation.md) | Wins. Cross-validation needs several nodes, so it cannot be pinned to one. | | [`lava-extension`](../api/directives.md#override-the-extension) | A session id used for both plain and `archive` calls gets a separate node for each. | -| [Retry](failover/retry.md) | A retry after a real error goes to a different node on purpose. | +| [Retry](failover/retry.md) | Never moves a sticky request. If the session's node answers with an error, the router returns that error with `Lava-Retries: 0`. | ## Trade-offs