Provider Simulator

Fake blockchain providers for testing the smart-router. The router talks to this simulator instead of real nodes, so a test can make a provider go down, stall, return errors, or fall behind — on demand, deterministically — then check how the router reacts.

It runs as one Python process: many listeners (JSON-RPC, REST, gRPC, Tendermint RPC, GraphQL, WebSocket) plus a control API, all over one in-memory registry of providers.

Architecture

Two planes over one registry. Tests write provider config on the control plane; the router reads it on the data plane. A change is instant — no restart, no IPC.

flowchart TD
    T["Test / sim_control"] -->|"control :19000"| CTRL["control_api"]
    R["smart-router (SUT)"] -->|"provider ports 18545-18585"| LIS["Listeners (daemon threads)"]
    CTRL -->|"writes scenario"| REG["Registry: pool:pid -> Provider"]
    LIS -->|"reads snapshot"| REG
    LIS --> CH["Chains: eth / btc / ln / solana / lava / sui"]
    LIS --> FP["fault_policy (one decision function)"]

Loading

• Identity is pool:pid — one provider is one named node in one router's pool (e.g. eth-sim:1). Two chains never share a provider, so a fault on one chain can never leak onto another.
• Each listener owns one port; the port resolves through the registry to a provider + endpoint. The provider's chain builds the success response, and one fault_policy decides every fault.

Run it locally

Python 3.12. gRPC needs grpcio / grpcio-reflection / protobuf; the JSON-RPC side is standard-library only.

pip install -r requirements.txt
python -u run.py

Smoke it:

curl -s http://localhost:19000/health
curl -s -X POST http://localhost:18545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
grpcurl -plaintext localhost:18548 list

Run the tests (they boot the server in-process on isolated ports):

pytest tests/ -v

Lint, format, and type-check (CI runs all three):

black --check . && ruff check . && mypy .

Ports & dispatch

Each row is a group of 3 providers. The port resolves to a pool:pid provider.

Surface	Ports (primary / backup / solo)	Pool
ETH JSON-RPC	18545–47 / 18560–62 / 18581	eth-sim (pids 1–6) / eth-solo-sim
ETH WebSocket	18557–59 / 18572–74	eth-sim (same providers, ws endpoint)
BTC JSON-RPC	18575–77	btc-sim
LN JSON-RPC	18578–80	ln-sim
Solana JSON-RPC	18582–84 / 18585	solana-sim / solana-solo-sim
gRPC	18548–50 / 18563–65	lava-sim-grpc
REST	18551–53 / 18566–68	lava-sim-rest
Tendermint RPC	18554–56 / 18569–71	lava-sim-tm
Control API	19000	control_api

Providers & state

• A provider is one fake node, addressed pool:pid (e.g. eth-sim:1). Its behaviour lives in a ScenarioConfig (fault mode, latency, error shape, per-method overrides) plus typed per-chain Quirks; its telemetry lives in a CallLog. Nothing is shared with any other provider.
• One ETH provider serves two endpoints under one identity — http and ws — so a down on eth-sim:1 covers both. REST / gRPC / Tendermint are separate pools (lava-sim-rest / -grpc / -tm), so faulting one never touches another.
• snapshot() gives each request a stable read; update() applies a /scenario change under a lock.

Scope a fault with transports, not chain_family. A scenario block's optional transports list (e.g. ["ws"]) scopes its effect to specific endpoints of the provider; omit it and the block covers every endpoint. There is no chain_family field any more — the pool already fixes the chain.

down is provider-wide by default. A provider set to down returns 503 on all of its endpoints. This is the natural default now, not a special case.

Fault primitives

Pick one mode; combine it with the orthogonal fields.

Field	Values	Effect
mode	success | error | rate_limit | down | hang | drop_connection | port_closed	Primary behaviour. port_closed closes the port of the endpoint, see below
latency_ms	int	Sleep N ms before responding
error_probability	0.0–1.0	Random error on top of success
error_code / error_message / http_status	int or str / str / int	Customise the error returned. A GraphQL provider reads a text error_code: the extensions.code of its error
rate_limit_style	status | body	GraphQL only. status is the HTTP 429 with plain text. body is a 200 with a RESOURCE_EXHAUSTED error
error_body	any JSON value or text	GraphQL only. With mode=error, the whole reply, sent with http_status
unreadable_reply	{"status": int, "body": any}	GraphQL only. The reply to every request body that cannot be read. The default imitates the reply of Sui; docs/using_the_simulator.md says where its text differs
corruption_mode	truncated | invalid_json | empty_response | missing_field | wrong_type	Break the response body
missing_field	str	Which field to drop or retype
blocks_behind	int	Report a stale head (eth_blockNumber / eth_getBlockByNumber / gRPC GetLatestBlock)
drop_at	before_headers | after_headers | mid_body	Where drop_connection cuts the socket
pause_at	after_headers | mid_body	Where a reply is HELD and then finished. Not a fault — mode stays success, the body is complete and the Content-Length honest, so the caller gets a correct reply late. before_headers is refused, because latency_ms already delays the first byte. HTTP transports only
pause_ms	int	How long the hold lasts. Measured complete at 240s; the handler's 30s socket timeout governs the request READ, not the response write
fail_first_n / then_mode	int / mode	Fail the first N calls, then switch to then_mode
transports	list of http / http2 / ws	Scope the block to specific endpoints (omit = all)

port_closed: a provider whose port is really closed

In every other mode the provider receives the request. down answers it on an open connection: HTTP 503 on JSON-RPC, REST and Tendermint-RPC, and the status UNAVAILABLE on gRPC. port_closed is the mode where the provider receives nothing. The simulator stops the server of the endpoint and closes the listening socket, so it refuses a new TCP connection. A test chooses between down and port_closed.

The mode works on each provider endpoint: http, ws and http2 (gRPC).

curl -s -X POST localhost:19000/scenario -d '{"providers":{"eth-sim:1":{"mode":"port_closed"}}}'
nc -z localhost 18545   # the http port: refused, on the very next line
nc -z localhost 18557   # the ws port of the same provider: refused too

• With no filter each port of the provider closes. transports and ports name fewer ports, as for every other mode. The other providers are untouched.
• An open connection ends. A request that the provider holds gets no reply. A reply that the provider is sending is cut: the caller can get the status line and the headers, and no body or a part of it. A WebSocket connection ends with no close frame.
• A closed port stores no row. The provider receives nothing, so it records nothing in /history. The row of a request in flight stays as the simulator wrote it when the request arrived. A request that reached the socket before the close, and that the provider did not read yet, gets no row and no reply.
• The subscriptions of a closed ws port leave GET /ws/subscriptions when their connection ends. That is a moment after the control call returns.
• The control call returns after the port has changed. Closed after a call that sets the mode; accepting connections after a call that removes it (another mode, POST /reset, POST /reset/all). No sleep is needed. When the port does not get there within 5 seconds the call answers HTTP 500 and names the pool, the provider and the port. It never answers 200 for a state it did not reach. A call that asks for the state that the port already has makes no new connection check.
• The scenario time-to-live reopens it too. The sweep that reverts the scenario waits for the port, as a control call does, so the port accepts when the sweep pass ends.
• GET /ready stays 200. A port closed by a scenario is left out of the readiness check and listed under closed_by_scenario.
• A gRPC client reconnects with its own back-off. The port accepts at once when it reopens; a client that was refused a moment earlier waits out its reconnect delay before its next call succeeds. Measured with the Python client over six runs: 0.8 to 1.2 seconds.
• No scenario closes a port that is not of a provider. The control port, the cache simulator ports and the RESP ports have no gate.

It is refused, never accepted and ignored:

Request	Answer
It would target no endpoint at all (a transports filter the provider has no endpoint for)	400
The simulator runs no listener for the port (for a gRPC port: grpcio is not installed)	409
Inside a per-method responses override	400. An override answers a request that already arrived
As a then_mode	400. then_mode follows fail_first_n requests, and a closed port receives none
Together with fail_first_n > 0, error_probability > 0, latency_ms > 0, corruption_mode or pause_at	400. Each acts on a request or on the reply to it

Control API (port 19000)

Route	Purpose
POST /scenario	Set per-provider behaviour, keyed pool:pid: {"providers": {"eth-sim:1": {"mode": "down"}}}. An old bare-pid key or a chain_family field gets a 400 naming the new format.
POST /reset	Reset scenario config, keep history. Also returns a cache-sim to answering misses
POST /history/clear	Clear history, keep config. Also clears each cache-sim's call log
POST /reset/all	Reset both, cache-sims included
POST /advance	Move a chain's simulated head (default eth; sync-freshness tests)
POST /ws/emit	Push a WebSocket event to a live subscription
GET /version	Which build this simulator is: see below
GET /health · /ready · /scenario · /stats · /topology · /history · /ws/subscriptions	Health / readiness (all listener ports bound, except a port a port_closed scenario closed) / config / counters / read-only provider/port topology per pool / call log / live subscriptions

Cache-sims (a simulated read-only secondary cache)

A router reaches one of these on its own port through --secondary-cache-be; a test reaches the same one by name, here. A cache is not a provider — it has no pool and no pid — so it is addressed by name, and a scenario set on a chain's providers can never reach it. An unknown name is a 404 listing the names that exist.

Route	Purpose
POST /cache/<name>/entry	What the next lookups get: {"mode": "hit", "entry": {"data": "{\"result\":\"0x1\"}"}}. Modes are hit, miss, error, hang, malformed. entry.data is plain text — it is base64-encoded once on the way out, so encoding it yourself sends it twice-encoded
POST /cache/<name>/reset	Back to answering misses; entry and call log dropped
POST /cache/<name>/calls/clear	Drop the call log, keep the staged entry
POST /cache/<name>/selftest-write	Send this cache-sim one real non-read call, and report whether its call log kept it. Run this BEFORE trusting an absence of writes — see below
GET /cache/<name>/calls	Every call the cache-sim received, oldest first — reads and refused non-reads alike, each with the method it arrived on
GET /cache/<name>	Its current state
GET /caches	Every cache-sim this simulator runs

Before asserting that no write arrived, use selftest-write. An absence is evidence only when the thing it rules out would have shown up, and twice it would not have: this simulator used to register the read method alone, so gRPC answered a write itself and the log never saw it; and after that was fixed, the rows it wrote carried an empty method name. The route sends one real non-read call over a socket, through gRPC's own dispatch and the generic handler, and answers 200 only if the log then holds it under its own name. It answers 502 if the call never arrived and 500 if the cache-sim served it. Clear the log with calls/clear afterwards, not reset, which also drops what you staged.

Read the call log to prove the secondary was reached. Neither cache tier names itself in the response, so that log and the router's own smartrouter_cache_success_total{cache_tier="secondary"} counter are the only two witnesses a test has.

GET /version answers "which simulator am I talking to", so a release run can be identified afterwards without hashing files inside the pod:

{"version": "v1.4.0", "commit": "cd0c74c2ad8e9ff36d24ce19e29e6db2aba6ec3e", "git_describe": "v1.4.0", "state": "release"}

state is one of three, and it never guesses:

state	Means	version
release	Built from a commit carrying a release tag	the tag, e.g. v1.4.0
untagged	Built from a commit between releases	null. git_describe names the release it follows, e.g. v1.4.0-3-gabc1234
unknown	Nothing was stamped in: python run.py from a checkout, or an image built before this route existed	null, and so is commit

The values are stamped into the image at build time: scripts/deploy.sh reads them from the checkout it is building and passes them as Docker build arguments, because a container has no .git directory to ask at runtime. A build made from a tree with uncommitted changes reports untagged with -dirty in git_describe: the tag no longer describes what was built.

GET /history merges every provider's ring buffer, sorts by time, and adds correlation_group (the calls of one router relay) and call_order. Each entry carries pool / pid / interface / transport / port; filter by pool / transport / method / status / request_id / time / lava_header_*. Each row also says what the provider received: headers (every request header), lava_headers, and for gRPC deadline_remaining_ms and grpc_status.

Deploy

• scripts/deploy.sh — build the image, import it, apply the manifests, and rollout-restart. The image tag is always :latest with imagePullPolicy: IfNotPresent, so the restart is what makes the new image take effect.
• Public URLs derive from BASE_DOMAIN in config/base-domain.env: sim-control.<BASE_DOMAIN>, eth-sim-jsonrpc.<BASE_DOMAIN>, lava-sim-grpc.<BASE_DOMAIN>:443.
• The deployment is intentionally 1 replica — history and counters live in memory, so a second pod would split the state.

Docs

• docs/START_HERE.md — index / learning path.
• docs/new_server_setup.md — fresh-server setup, verification, router wiring, debug domain.
• docs/using_the_simulator.md — scenarios, resets, history, fault primitives, with paste-ready examples.
• docs/using_grpc.md — gRPC quickstart, reflection, fault injection, troubleshooting.
• docs/curl_reference.md — full curl catalogue for the JSON-RPC surface.
• docs/ARCHITECTURE_GUIDE.md · docs/CLASS_REFERENCE.md · docs/DATA_FLOWS.md — deeper code walkthroughs.
• docs/kubectl_reference.md · docs/DEPLOY_AFTER_PUSH.md — cluster ops.

Repo layout

run.py                        — entry point
server.py                     — bootstrap: build the registry from TOPOLOGY, bind listeners to ports
provider_simulator/
  topology.py                 — the pool / provider / endpoint table
  fault_policy.py             — the one fault-decision function
  domain/                     — Endpoint, ScenarioConfig, Quirks, CallLog, Pool/Provider, Registry
  chains/                     — one class per chain (eth/btc/solana/ln/lava)
  listeners/                  — base template + jsonrpc/rest/tendermint/grpc, the WebSocket
                                listener and its subscription registry, and the corruption serializer
  control_api.py              — the port-19000 routes over the registry
stubs*.py                     — default success payloads per surface (eth/btc/ln/solana/rest/tm/ws)
constants.py                  — port maps, chain constants, history caps
cosmos_pb2/                   — vendored Cosmos protobufs
config/base-domain.env        — BASE_DOMAIN for deploy
config/values_sim.yml         — smart-router Helm values that wire the sim in
Dockerfile                    — image used by scripts/deploy.sh
k8s/                          — Deployment, Service, HTTPRoute, GRPCRoute
scripts/deploy.sh             — build / import / apply / restart
tests/                        — pytest suite (all surfaces, in-process)