From 5e41db9c80100d4d895c661562d49d38a23d286c Mon Sep 17 00:00:00 2001 From: KonScanner Date: Fri, 21 Aug 2026 17:31:59 +0300 Subject: [PATCH] feat: collapse per-row RPC into batched reads, and add eleven datasets Most datasets read one thing per row, and cost one request per row. Two new routes collapse that. Both are on by default, and both fall back to the per-row path when an endpoint refuses, so batching changes what results cost and never the results themselves. JSON-RPC batching sends one HTTP body per N rows for blocks, nonces and codes. On `blocks -b 21000000:+500`, 501 requests become 6, and the two parquet files are byte-identical. The batch size negotiates itself down when a provider caps it (OP Mainnet answers 413 above ten calls, Base returns -32014) and the discovered size is kept for the rest of the run. State overrides serve slots, balances and proxy_slots. `eth_call` takes a third parameter that applies changes to a scratch copy of state, so replacing the target's code with a 25-byte extractor loop turns "read one slot" into "read every slot I name". One opcode swap gives that loop four readers: SLOAD, BALANCE, EXTCODEHASH and EXTCODESIZE. The account readers take their address from the stack, so they run at a scratch address and batch across unrelated addresses without touching any real contract. The extractor is PUSH0-free on purpose. `eth_call` at a historical block runs under that block's fork rules, and PUSH0 only became valid at Shanghai (mainnet 17034870), so Dedaub's published 23-byte form is rejected below it. An endpoint that accepts the override parameter and ignores it is the dangerous case: our slot keys would reach the real contract, and its fallback output would be decoded as storage. Two checks prevent that. The return must be exactly 32*n bytes, and the storage reader sets a sentinel slot through stateDiff and reads it back as the final word. stateDiff is used and never state, because state replaces an account's whole storage and would return a well-formed file of zeros. Each reader has one blind spot: the input for which our own override is the answer. It is the scratch address for the account readers and the sentinel slot for the storage reader. Both would pass the length check and the sentinel check, so neither is detectable downstream. A row that names either one skips the batch and is read with no override in sight. An endpoint that fails the same way three times in a row stops being asked. Without that count, a proxy that strips the override below the JSON-RPC layer takes the full halving cascade on every chunk, for the length of the run. eth_getStorageValues sits above both rungs where a node has it. It takes a {contract: [slots]} map, so slots for different contracts at one block travel in a single request, which an injected extractor cannot do at any batch size. Support cannot be cached either way, because a load-balanced pool holds a mix of node versions, so consecutive misses are counted rather than a verdict latched. address_appearances made three RPCs in series per block and now makes them concurrently. Eleven datasets are new: approvals_for_all, contract_interfaces, erc1155_metadata, erc1155_transfers, erc20_allowances, erc2612_nonces, erc4626_metadata, erc4626_vault_events, erc777_transfers, proxy_slots and proxy_upgrades. erc20_allowances reads allowance() at a block and sits beside the existing erc20_approvals, which replays the event log. An allowance also falls when the spender spends, because transferFrom decrements it without an Approval event, so the log alone overstates what is spendable. proxy_upgrades takes the implementation address from topic1 when it is indexed and from the data word when it is not. `indexed` is not part of an event signature preimage, so zos-lib's 2018 `Upgraded(address)` and the ERC-1967 one share a topic0. Selecting on the argument's position rather than its presence fetches those logs and then discards them, which would report USDC's proxy as never upgraded. New flags: --no-batch-rpc-calls, --no-batch-state-reads and --state-override-batch-size. Reading the per-row paths closely enough to batch them turned up five correctness bugs, all of them silent. contracts held each hash column's value in the other column: init_code_hash carried keccak(code) and code_hash carried keccak(init_code). code_hash is the usual join key for contract identity, so every match against an EXTCODEHASH, or against an external codehash index, found nothing rather than erroring. Eleven log datasets accepted --address, --from-address, --to-address and --topic0..3 under --txs, counted them into the partition set, printed them in the run summary, and then ignored them. A receipt query asks for one whole transaction, so the narrowing that the by-block path pins into eth_getLogs has to be re-applied by hand, and was not. The run returned rows it had been asked to exclude and said nothing. call_outcome recognised four of geth's execution-failure messages and treated the other ten as node failures, so a stack underflow or a "return data out of bounds" failed a whole chunk where a revert wrote one null. EIP-1474 reserves error code 3 for execution reverted, which is the one structured signal available, and it is now checked before the message. contract_interfaces was the most exposed, because it aims eth_call at arbitrary bytecode. --multicall-require-success was a no-op on a batch of one. Every dataset groups its rows by block before batching, so one address over a block range gives one row per group and every batch is a singleton: the flag asked for strictness and got the default nulls. Five multicall calls_for_row implementations built their target with Address::from_slice, which panics on a width mismatch, and --address is hex-decoded with no length check. A panic there aborts the worker task for a whole batch of rows, where an error demotes the batch and reports one. --l1-rpc's help named six datasets that read it. None of them exist; nothing reads it yet. It connects and reports its chain id, and the help now says so. Documentation covers the eleven datasets, the two batching routes and the three flags. Every schema block in those pages is copied from `triodion help ` rather than derived by hand. --- README.md | 11 + book/SUMMARY.md | 12 + book/basic_usage/request_batching.md | 145 ++ book/datasets/approvals_for_all.md | 114 + book/datasets/contract_interfaces.md | 109 + book/datasets/dataset_reference.md | 11 + book/datasets/erc1155_metadata.md | 88 + book/datasets/erc1155_transfers.md | 139 ++ book/datasets/erc20_allowances.md | 132 ++ book/datasets/erc2612_nonces.md | 85 + book/datasets/erc4626_metadata.md | 85 + book/datasets/erc4626_vault_events.md | 105 + book/datasets/erc777_transfers.md | 131 ++ book/datasets/proxy_slots.md | 97 + book/datasets/proxy_upgrades.md | 112 + crates/cli/src/args.rs | 60 +- crates/cli/src/parse/blocks.rs | 6 + crates/cli/src/parse/query.rs | 3 + crates/cli/src/parse/source.rs | 2 + crates/cli/src/parse/timestamps.rs | 2 + .../src/datasets/address_appearances.rs | 130 +- .../freeze/src/datasets/approvals_for_all.rs | 367 ++++ crates/freeze/src/datasets/balances.rs | 82 +- crates/freeze/src/datasets/blocks.rs | 52 +- crates/freeze/src/datasets/codes.rs | 65 +- .../src/datasets/contract_interfaces.rs | 382 ++++ crates/freeze/src/datasets/contracts.rs | 74 +- .../freeze/src/datasets/erc1155_metadata.rs | 331 +++ .../freeze/src/datasets/erc1155_transfers.rs | 493 +++++ .../freeze/src/datasets/erc20_allowances.rs | 184 ++ crates/freeze/src/datasets/erc20_approvals.rs | 21 +- crates/freeze/src/datasets/erc20_balances.rs | 6 +- crates/freeze/src/datasets/erc20_metadata.rs | 7 +- crates/freeze/src/datasets/erc20_supplies.rs | 9 +- crates/freeze/src/datasets/erc20_transfers.rs | 21 +- .../src/datasets/erc20_wrapper_events.rs | 20 +- crates/freeze/src/datasets/erc2612_nonces.rs | 217 ++ .../freeze/src/datasets/erc4626_metadata.rs | 230 ++ .../src/datasets/erc4626_vault_events.rs | 382 ++++ crates/freeze/src/datasets/erc721_metadata.rs | 7 +- .../freeze/src/datasets/erc721_transfers.rs | 21 +- .../freeze/src/datasets/erc777_transfers.rs | 490 +++++ crates/freeze/src/datasets/eth_calls.rs | 4 +- crates/freeze/src/datasets/logs.rs | 23 +- crates/freeze/src/datasets/mod.rs | 36 +- crates/freeze/src/datasets/nonces.rs | 52 +- crates/freeze/src/datasets/proxy_slots.rs | 191 ++ crates/freeze/src/datasets/proxy_upgrades.rs | 512 +++++ crates/freeze/src/datasets/slots.rs | 61 +- .../freeze/src/multi_datasets/log_events.rs | 37 +- crates/freeze/src/types/datatypes/scalar.rs | 11 + crates/freeze/src/types/errors.rs | 94 +- crates/freeze/src/types/mod.rs | 11 + crates/freeze/src/types/multicall.rs | 17 +- crates/freeze/src/types/queries.rs | 36 + crates/freeze/src/types/rpc_batch.rs | 185 ++ crates/freeze/src/types/rpc_params.rs | 72 +- crates/freeze/src/types/signatures.rs | 135 ++ crates/freeze/src/types/sources.rs | 422 +++- crates/freeze/src/types/state_override.rs | 1891 +++++++++++++++++ crates/python/rust/collect_adapter.rs | 9 + crates/python/rust/freeze_adapter.rs | 9 + examples/all.sh | 53 + 63 files changed, 8771 insertions(+), 130 deletions(-) create mode 100644 book/basic_usage/request_batching.md create mode 100644 book/datasets/approvals_for_all.md create mode 100644 book/datasets/contract_interfaces.md create mode 100644 book/datasets/erc1155_metadata.md create mode 100644 book/datasets/erc1155_transfers.md create mode 100644 book/datasets/erc20_allowances.md create mode 100644 book/datasets/erc2612_nonces.md create mode 100644 book/datasets/erc4626_metadata.md create mode 100644 book/datasets/erc4626_vault_events.md create mode 100644 book/datasets/erc777_transfers.md create mode 100644 book/datasets/proxy_slots.md create mode 100644 book/datasets/proxy_upgrades.md create mode 100644 crates/freeze/src/datasets/approvals_for_all.rs create mode 100644 crates/freeze/src/datasets/contract_interfaces.rs create mode 100644 crates/freeze/src/datasets/erc1155_metadata.rs create mode 100644 crates/freeze/src/datasets/erc1155_transfers.rs create mode 100644 crates/freeze/src/datasets/erc20_allowances.rs create mode 100644 crates/freeze/src/datasets/erc2612_nonces.rs create mode 100644 crates/freeze/src/datasets/erc4626_metadata.rs create mode 100644 crates/freeze/src/datasets/erc4626_vault_events.rs create mode 100644 crates/freeze/src/datasets/erc777_transfers.rs create mode 100644 crates/freeze/src/datasets/proxy_slots.rs create mode 100644 crates/freeze/src/datasets/proxy_upgrades.rs create mode 100644 crates/freeze/src/types/rpc_batch.rs create mode 100644 crates/freeze/src/types/state_override.rs diff --git a/README.md b/README.md index 6082bda..fcc5022 100644 --- a/README.md +++ b/README.md @@ -463,6 +463,7 @@ triodion datasets ───────────────── - access_lists - address_appearances +- approvals_for_all (aliases = erc721_approvals_for_all, erc1155_approvals) - authorizations - balance_diffs - balance_reads @@ -473,16 +474,24 @@ triodion datasets - code_reads - codes - consolidation_requests +- contract_interfaces (alias = erc165) - contracts - deposit_requests +- erc1155_metadata (alias = erc1155_uris) +- erc1155_transfers +- erc20_allowances (alias = allowances) - erc20_balances - erc20_metadata - erc20_supplies - erc20_transfers - erc20_approvals - erc20_wrapper_events (aliases = wrapper_events, weth_events) +- erc2612_nonces +- erc4626_metadata +- erc4626_vault_events (alias = erc4626_events) - erc721_metadata - erc721_transfers +- erc777_transfers - eth_calls - four_byte_counts (alias = 4byte_counts) - geth_calls @@ -497,6 +506,8 @@ triodion datasets - nonce_diffs - nonce_reads - nonces +- proxy_slots (alias = erc1967_slots) +- proxy_upgrades (alias = erc1967_events) - slots (alias = storages) - storage_diffs (alias = slot_diffs) - storage_reads (alias = slot_reads) diff --git a/book/SUMMARY.md b/book/SUMMARY.md index 3c3ffaa..f1ed2e2 100644 --- a/book/SUMMARY.md +++ b/book/SUMMARY.md @@ -12,6 +12,7 @@ - [Data Acquisition](./basic_usage/data_acquisition.md) - [Data Formatting](./basic_usage/data_formatting.md) - [Data Querying](./basic_usage/data_querying.md) + - [Request Batching](./basic_usage/request_batching.md) - [Python Bindings](./triodion_python/overview.md) - [Installation](./triodion_python/installation.md) - [Example Usage](./triodion_python/example_usage.md) @@ -30,6 +31,7 @@ - [Overview of each dataset](./datasets/dataset_reference.md) - [access_lists](./datasets/access_lists.md) - [address_appearances](./datasets/address_appearances.md) + - [approvals_for_all](./datasets/approvals_for_all.md) - [authorizations](./datasets/authorizations.md) - [balance_diffs](./datasets/balance_diffs.md) - [balance_reads](./datasets/balance_reads.md) @@ -40,16 +42,24 @@ - [code_reads](./datasets/code_reads.md) - [codes](./datasets/codes.md) - [consolidation_requests](./datasets/consolidation_requests.md) + - [contract_interfaces](./datasets/contract_interfaces.md) - [contracts](./datasets/contracts.md) - [deposit_requests](./datasets/deposit_requests.md) + - [erc1155_metadata](./datasets/erc1155_metadata.md) + - [erc1155_transfers](./datasets/erc1155_transfers.md) + - [erc20_allowances](./datasets/erc20_allowances.md) - [erc20_balances](./datasets/erc20_balances.md) - [erc20_metadata](./datasets/erc20_metadata.md) - [erc20_supplies](./datasets/erc20_supplies.md) - [erc20_transfers](./datasets/erc20_transfers.md) - [erc20_approvals](./datasets/erc20_approvals.md) - [erc20_wrapper_events](./datasets/erc20_wrapper_events.md) + - [erc2612_nonces](./datasets/erc2612_nonces.md) + - [erc4626_metadata](./datasets/erc4626_metadata.md) + - [erc4626_vault_events](./datasets/erc4626_vault_events.md) - [erc721_metadata](./datasets/erc721_metadata.md) - [erc721_transfers](./datasets/erc721_transfers.md) + - [erc777_transfers](./datasets/erc777_transfers.md) - [eth_calls](./datasets/eth_calls.md) - [four_byte_counts](./datasets/four_byte_counts.md) - [geth_calls](./datasets/geth_calls.md) @@ -64,6 +74,8 @@ - [nonce_diffs](./datasets/nonce_diffs.md) - [nonce_reads](./datasets/nonce_reads.md) - [nonces](./datasets/nonces.md) + - [proxy_slots](./datasets/proxy_slots.md) + - [proxy_upgrades](./datasets/proxy_upgrades.md) - [slots](./datasets/slots.md) - [storage_diffs](./datasets/storage_diffs.md) - [storage_reads](./datasets/storage_reads.md) diff --git a/book/basic_usage/request_batching.md b/book/basic_usage/request_batching.md new file mode 100644 index 0000000..94f55ba --- /dev/null +++ b/book/basic_usage/request_batching.md @@ -0,0 +1,145 @@ +# Request Batching + +Most datasets read one thing per row: one storage slot, one balance, one header. +The obvious way to collect a partition is one RPC per row, and that is what +triodion did. It is also what makes a large query slow, and what gets a public +endpoint to answer `429`. + +Triodion now takes the best route each dataset and endpoint allow. Every route +is **on by default**, and every route falls back to the per-row path when it +fails — so batching changes how many requests your results cost, never the +results themselves. + +## The switches + +| Flag | Turns off | Applies to | +|---|---|---| +| `--no-batch-rpc-calls` | JSON-RPC request batching | `blocks`, `nonces`, `codes` | +| `--no-batch-state-reads` | `eth_call` state overrides | `slots`, `balances`, `proxy_slots` | +| `--state-override-batch-size N` | (sets size, 0 = default 1000) | `slots`, `balances`, `proxy_slots` | +| `--no-multicall` | Multicall3 aggregation | `eth_calls`, the ERC-20 and vault read datasets | + +Multicall3 is on by default like the other two, so the flag that changes +anything is `--no-multicall`. A bare `--multicall` is still accepted, as a +no-op, so older scripts keep working. + +They are separate switches because they ask different things of the node. A +JSON-RPC batch needs nothing beyond JSON-RPC itself. A state override needs an +endpoint that honours the third `eth_call` parameter, which not every endpoint +does. An operator who distrusts an endpoint's override support has no reason to +give up plain batching as well. + +## JSON-RPC batching + +JSON-RPC has always allowed an array of requests in one HTTP body. `blocks`, +`nonces` and `codes` use it, because none of them has a better route available. + +Measured on `blocks -b 21000000:+500` through a counting proxy in front of a +public mainnet endpoint: + +| | HTTP requests | embedded calls | wall clock | +|---|---|---|---| +| `--no-batch-rpc-calls` | 501 | 501 | 0.96s | +| default | **6** | 501 | **0.46s** | + +The two parquet files are byte-identical. + +The request count is the reliable number: 83× fewer, and it follows from the +batch size rather than from the endpoint. The wall-clock figure does not +generalise — this endpoint was fast and did not throttle, so batching only won +2×. The further an endpoint is, and the harder it rate-limits, the more of the +501 round trips batching removes and the wider the gap gets. That is also the +case where the per-request path stops finishing at all. + +The batch size negotiates itself downward when a provider refuses one — OP +Mainnet answers `413` above ten calls, Base returns +`-32014 "maximum 10 calls in 1 batch"` — and the discovered size is kept for the +rest of the call rather than rediscovered per batch. + +`codes` uses a smaller batch (50) because whole contract bytecode is far larger +than a 32-byte word, and a hundred EIP-170-sized contracts is a 2.4MB response. + +`nonces` is the one account field that can *only* batch this way. See +[nonces](../datasets/nonces.md). + +## State overrides + +`eth_call` takes a third parameter, a state-override set, that applies changes to +a scratch copy of state before execution. Overriding a contract's **code** with a +small extractor loop turns "read one slot" into "read every slot I name". + +There is no Multicall3 equivalent for storage, because a deployed contract +cannot read another contract's slots. Bytecode *injected into* the target can +read all of them. + +The same 25-byte loop with one opcode swapped serves four readers: `SLOAD` for +storage, and `BALANCE` / `EXTCODEHASH` / `EXTCODESIZE` for accounts. The account +readers take their address from the stack, so they run at a scratch address and +batch across unrelated addresses without overriding any real contract. + +### Historical blocks + +`eth_call` at a historical block executes under *that block's* fork rules. +Triodion's extractor is deliberately `PUSH0`-free, because `PUSH0` only became +valid at Shanghai (mainnet block 17034870, April 2023). Below that a `PUSH0` +extractor is rejected outright — geth reports `invalid opcode: PUSH0`, reth +reports `EVM error: NotActivated`. + +The published 23-byte extractor from Dedaub's write-up of this technique is the +`PUSH0` form. Triodion emits a 25-byte variant instead, verified byte-identical +to `eth_getStorageAt` at latest, block 15000000 and block 12000000. There is no +fork detection to get wrong. + +## The ladder + +For `slots`, triodion tries three things in order. Measured on 4 contracts × +6 slots at one block, through a proxy that denies methods on demand: + +| rung | condition | requests | +|---|---|---| +| 1 | `eth_getStorageValues` available | **1** | +| 2 | native denied, overrides honoured | 4 `eth_call` + 1 probe | +| 3 | both denied | 24 `eth_getStorageAt` | + +All three produce identical output. + +**Rung 1** is `eth_getStorageValues`, added in geth v1.17.1. It does natively +what the extractor does by injection — many slots, across many contracts, one +round trip — with no override and none of the override failure modes. Because it +takes a `{contract: [slots]}` map, slots for *different* contracts at the same +block travel in one request. That is the one thing the extractor cannot do at +any batch size, since injected code reads only its own address's storage. + +Support for it cannot be cached. A load-balanced pool holds a mix of node +versions: the same URL served this method on one request and returned +`Method not found` on the next. So triodion counts *consecutive* misses rather +than latching a verdict, and a mixed pool keeps using it whenever it works. + +**Rung 2** is the extractor, one `eth_call` per contract. + +**Rung 3** is the per-row path, exactly as it was before any of this existed. + +## Verification + +An endpoint that *accepts* the override parameter and silently ignores it is the +dangerous case. Our slot-key calldata would reach the real contract, and its +fallback output would be decoded as storage — plausible values, right shape, +entirely fabricated, written to a Parquet file that looks correct forever. + +Two things prevent that: + +1. The return must be **exactly** `32 × n` bytes. The extractor returns one word per calldata word by construction, so the length is positive proof that our code produced the answer. +2. The storage reader also sets a sentinel slot via `stateDiff` and appends that slot as the final calldata word. Getting the sentinel value back in the final returned word proves our bytecode ran, that the override was applied, and that words came back in order. + +Anything else is an error, never a partial result — and an error only routes +those rows to the per-row path. + +Note that `stateDiff` is used, never `state`. `state` replaces an account's +entire storage, which would return a perfectly well-formed file of zeros. + +The sentinel slot is itself the one slot this path cannot read, because it is +the one slot this path writes. A row that asks for it would come back holding +the sentinel value, with the length check and the sentinel check both passing. +So a row naming that slot skips the batch and is read by `eth_getStorageAt`, +with no override in sight. The account readers have the same blind spot at the +scratch address they run at, and it is guarded the same way. diff --git a/book/datasets/approvals_for_all.md b/book/datasets/approvals_for_all.md new file mode 100644 index 0000000..bb55e45 --- /dev/null +++ b/book/datasets/approvals_for_all.md @@ -0,0 +1,114 @@ +# approvals_for_all + +One row per `ApprovalForAll(address,address,bool)` log. + +``` +approvals_for_all +───────────────── +- can collect by block or by transaction +- required parameters: [none] +- optional parameters: address, topic1, topic2, from_address, to_address +- dataset aliases: erc721_approvals_for_all, erc1155_approvals +- parameter aliases: + - contracts -> addresses + + +schema for approvals_for_all +──────────────────────────── +- block_number: uint32 +- transaction_index: uint32 +- log_index: uint32 +- transaction_hash: binary +- contract_address: binary +- owner: binary +- operator: binary +- approved: bool +- chain_id: uint64 + +sorting approvals_for_all by: block_number, log_index + +other available columns: block_hash +``` + +Run `triodion help approvals_for_all` to print this from the binary. + +Aliases: `erc721_approvals_for_all`, `erc1155_approvals`. + +## Why the name has no standard in it + +ERC-721 and ERC-1155 declare this event with the same argument types: + +```solidity +// ERC-721 +ApprovalForAll(address indexed owner, address indexed operator, bool approved) +// ERC-1155 +ApprovalForAll(address indexed account, address indexed operator, bool approved) +``` + +The argument names differ; the types do not. Both therefore hash to one topic0, +and a log that carries it does not say which of the two standards the emitting +contract implements. No decoding recovers that — the bytes are identical. + +Calling this dataset `erc1155_approvals` would put a false claim on roughly half +of its rows. Both names are kept as aliases, because both arrive at the same +rows, but neither names the dataset, so no output file claims a standard the +logs cannot prove. + +To classify a contract, join `contract_address` against +[contract_interfaces](./contract_interfaces.md), which asks the contract itself +through the ERC-165 `supportsInterface` call. + +## What the grant covers + +`owner` is the token holder whose permission changes. `operator` is the address +that gains or loses permission over **every** token the owner holds in that +contract. + +Unlike the ERC-721 `Approval` event, this grant names no token id, so it also +covers tokens the owner acquires after this block. A blanket grant made once is +still in force against a token bought a year later. + +## These are events, not state + +`approved == false` is a revocation. It is a measurement of what happened, not +the absence of one, so it is stored as `false` and never as null. + +That makes `WHERE approved` wrong on its own. An operator approved in one block +and revoked in the next leaves both rows behind, and the filter keeps the stale +one. Take the latest row per grant first, then filter: + +```sql +SELECT * FROM ( + SELECT *, ROW_NUMBER() OVER ( + PARTITION BY contract_address, owner, operator + ORDER BY block_number DESC, log_index DESC) AS rn + FROM approvals_for_all +) WHERE rn = 1 AND approved +``` + +## Filtering + +The event carries exactly one signature, so topic1 is the owner and topic2 is +the operator on every row, and address-shaped filtering is unambiguous here. +Both are reachable two ways: as a raw 32-byte topic, or as an ordinary 20-byte +address. + +`--from-address` maps to topic1 and `--to-address` to topic2, matching what +[erc20_approvals](./erc20_approvals.md) calls its owner and spender. The names +are loose — an approval has no direction — but demanding a hand-padded 32-byte +word for a value read off a block explorer is worse. + +Give one form per slot, not both. `--topic1` together with `--from-address` is +an error rather than a precedence rule: both are partition dimensions, so +resolving the conflict silently would still multiply the run by the discarded +one and write the same rows to several files. + +```bash +# every blanket approval a wallet has granted +triodion approvals_for_all -b 18M:+1000 \ + --from-address 0xd8da6bf26964af9d7eed9e03e53415d37aa96045 +``` + +This dataset is not a member of the `log_events` group. Requesting it +alongside `logs` or `erc20_transfers` therefore costs a separate +`eth_getLogs` per block range rather than sharing one with them. diff --git a/book/datasets/contract_interfaces.md b/book/datasets/contract_interfaces.md new file mode 100644 index 0000000..1760261 --- /dev/null +++ b/book/datasets/contract_interfaces.md @@ -0,0 +1,109 @@ +# contract_interfaces + +One row per (block, address): what a contract answers to ERC-165 +`supportsInterface(bytes4)` for a fixed table of interface ids. + +``` +contract_interfaces +─────────────────── +- can collect by block and not by transaction +- required parameters: address +- optional parameters: [none] +- dataset aliases: erc165 +- parameter aliases: + - contracts -> addresses + + +schema for contract_interfaces +────────────────────────────── +- block_number: uint32 +- address: binary +- answers_true_to_everything: bool +- supports_erc165: bool +- supports_erc721: bool +- supports_erc721_metadata: bool +- supports_erc721_enumerable: bool +- supports_erc1155: bool +- supports_erc1155_metadata_uri: bool +- supports_erc2981: bool +- supports_erc1271: bool +- chain_id: uint64 + +sorting contract_interfaces by: block_number, address + +other available columns: [none] +``` + +Run `triodion help contract_interfaces` to print this from the binary. + +Alias: `erc165`. + +## What it is for + +This is the classification layer the token datasets sit on. An address that +answers `true` for ERC-721 can be collected as an ERC-721, instead of being +guessed at from the shape of its logs. + +[approvals_for_all](./approvals_for_all.md) is the clearest case: ERC-721 and +ERC-1155 emit an identical `ApprovalForAll` topic0, so the log cannot say which +standard a contract implements. This dataset asks the contract. + +`--blocks` defaults to `latest`, so a run with no block range reads the current +state. + +## The interface ids + +An interface id is the XOR of the four-byte selectors of every function in the +interface. Each is fixed by its standard: + +| column | interface | id | +| :- | :- | :- | +| `supports_erc165` | ERC-165 | `0x01ffc9a7` | +| `supports_erc721` | ERC-721 | `0x80ac58cd` | +| `supports_erc721_metadata` | ERC-721 Metadata | `0x5b5e139f` | +| `supports_erc721_enumerable` | ERC-721 Enumerable | `0x780e9d63` | +| `supports_erc1155` | ERC-1155 | `0xd9b67a26` | +| `supports_erc1155_metadata_uri` | ERC-1155 MetadataURI | `0x0e89341c` | +| `supports_erc2981` | ERC-2981 royalties | `0x2a55205a` | +| `supports_erc1271` | ERC-1271 signatures | `0x1626ba7e` | + +## Three states, not two + +Every answer is nullable, and the three states are three different facts: + +- **null** — the call reverted, the address has no code, or the return was not an ABI `bool`. The + contract said nothing, which is what a non-ERC-165 address looks like. +- **false** — the contract answered no. +- **true** — the contract answered yes. + +Filling null with `false` would turn "never answered" into "answered no", and a +count of `false` would then include every EOA in the input. + +## The two columns that judge the rest of the row + +`supports_erc165` is required by the standard to be `true`. A contract that +implements `supportsInterface` but answers `false` here is not ERC-165 +compliant, and its other answers are its own convention rather than the +standard's. + +`answers_true_to_everything` is the probe for the reserved id `0xffffffff`. +ERC-165 **requires** a compliant contract to answer `false` to it. So `true` +here means the contract answers yes to every id ever asked: discard every other +column in the row, none of them carry information. + +`null` there is not an all-clear either. It means no decodable answer, so the +contract made no ERC-165 promise about the rest of the row. + +Read both before trusting any `supports_*` column: + +```sql +SELECT * FROM contract_interfaces +WHERE supports_erc721 + AND supports_erc165 + AND NOT COALESCE(answers_true_to_everything, TRUE) +``` + +```bash +triodion contract_interfaces \ + --address 0xed5af388653567af2f388e6224dc7c4b3241c544 +``` diff --git a/book/datasets/dataset_reference.md b/book/datasets/dataset_reference.md index 2ba1865..dc3c6d6 100644 --- a/book/datasets/dataset_reference.md +++ b/book/datasets/dataset_reference.md @@ -14,6 +14,7 @@ authority when the two disagree. | --- | --- | --- | | [access_lists](./access_lists.md) | | EIP-2930 access-list entries, one row per storage key | | [address_appearances](./address_appearances.md) | | every address that appears in a block, with the relationship that made it appear | +| [approvals_for_all](./approvals_for_all.md) | `erc721_approvals_for_all`, `erc1155_approvals` | `ApprovalForAll` events, which ERC-721 and ERC-1155 share | | [authorizations](./authorizations.md) | | EIP-7702 authorization tuples, with the recovered authority | | [balance_diffs](./balance_diffs.md) | | ETH balance changes, as before and after values | | [balance_reads](./balance_reads.md) | | ETH balances read during execution | @@ -24,16 +25,24 @@ authority when the two disagree. | [code_reads](./code_reads.md) | | contract code read during execution | | [codes](./codes.md) | | contract code at a block | | [consolidation_requests](./consolidation_requests.md) | | EIP-7251 consolidation requests, from the consensus layer | +| [contract_interfaces](./contract_interfaces.md) | `erc165` | what a contract answers to ERC-165 `supportsInterface` | | [contracts](./contracts.md) | | contracts deployed, with deployer, factory and init code | | [deposit_requests](./deposit_requests.md) | | EIP-6110 deposit requests, from the consensus layer | +| [erc1155_metadata](./erc1155_metadata.md) | `erc1155_uris` | ERC-1155 `URI` events, with the `{id}` template flagged | +| [erc1155_transfers](./erc1155_transfers.md) | | ERC-1155 transfers, one row per token id moved | +| [erc20_allowances](./erc20_allowances.md) | `allowances` | ERC-20 allowance of a spender over an owner's tokens at a block | | [erc20_balances](./erc20_balances.md) | | ERC-20 balance of an address at a block | | [erc20_metadata](./erc20_metadata.md) | | ERC-20 name, symbol and decimals | | [erc20_supplies](./erc20_supplies.md) | | ERC-20 total supply at a block | | [erc20_transfers](./erc20_transfers.md) | | ERC-20 `Transfer` events | | [erc20_approvals](./erc20_approvals.md) | | ERC-20 `Approval` events | | [erc20_wrapper_events](./erc20_wrapper_events.md) | `wrapper_events`, `weth_events` | WETH-shape `Deposit` and `Withdrawal` events | +| [erc2612_nonces](./erc2612_nonces.md) | | ERC-2612 permit nonce for a token and owner | +| [erc4626_metadata](./erc4626_metadata.md) | | ERC-4626 vault asset, total assets and share supply | +| [erc4626_vault_events](./erc4626_vault_events.md) | `erc4626_events` | ERC-4626 vault `Deposit` and `Withdraw` events | | [erc721_metadata](./erc721_metadata.md) | | ERC-721 name and symbol | | [erc721_transfers](./erc721_transfers.md) | | ERC-721 `Transfer` events, with token id | +| [erc777_transfers](./erc777_transfers.md) | | ERC-777 `Sent`, `Minted` and `Burned` events, with the operator | | [eth_calls](./eth_calls.md) | | `eth_call` outputs for a contract and call data at each block | | [four_byte_counts](./four_byte_counts.md) | `4byte_counts` | count of each four-byte function selector called | | [geth_calls](./geth_calls.md) | | call traces from geth's `callTracer` | @@ -48,6 +57,8 @@ authority when the two disagree. | [nonce_diffs](./nonce_diffs.md) | | nonce changes, as before and after values | | [nonce_reads](./nonce_reads.md) | | nonces read during execution | | [nonces](./nonces.md) | | nonce of an address at a block | +| [proxy_slots](./proxy_slots.md) | `erc1967_slots` | ERC-1967 proxy implementation, admin and beacon slots | +| [proxy_upgrades](./proxy_upgrades.md) | `erc1967_events` | ERC-1967 proxy `Upgraded`, `BeaconUpgraded` and `AdminChanged` events | | [slots](./slots.md) | `storages` | value of a storage slot at a block | | [storage_diffs](./storage_diffs.md) | `slot_diffs` | storage slot changes, as before and after values | | [storage_reads](./storage_reads.md) | `slot_reads` | storage slots read during execution | diff --git a/book/datasets/erc1155_metadata.md b/book/datasets/erc1155_metadata.md new file mode 100644 index 0000000..4c3deed --- /dev/null +++ b/book/datasets/erc1155_metadata.md @@ -0,0 +1,88 @@ +# erc1155_metadata + +One row per ERC-1155 `URI(string value, uint256 indexed id)` log. + +``` +erc1155_metadata +──────────────── +- can collect by block or by transaction +- required parameters: [none] +- optional parameters: address, topic1 +- dataset aliases: erc1155_uris +- parameter aliases: + - contracts -> addresses + + +schema for erc1155_metadata +─────────────────────────── +- block_number: uint32 +- transaction_index: uint32 +- log_index: uint32 +- transaction_hash: binary +- erc1155: binary +- token_id_binary: binary +- token_id_string: string +- token_id_f64: float64 +- uri: string +- is_uri_template: bool +- is_uri_lossy: bool +- chain_id: uint64 + +sorting erc1155_metadata by: block_number, log_index + +other available columns: block_hash +``` + +Run `triodion help erc1155_metadata` to print this from the binary. + +Alias: `erc1155_uris`. + +## An empty result is the normal case + +The `URI` event is optional in the standard, and most contracts never emit it. +They serve one static `{id}` template from `uri(id)` and never announce a +change. A full-range scan of a busy chain can legitimately return zero rows. + +This is not a failure, and it is not a sign the filter is wrong. + +## Why this reads logs and not uri(id) + +A call needs a token id to call with. triodion partitions by block, address and +topic — there is no token-id dimension to enumerate ids from, so the call form +has no input. The log carries its own id, so the log form does. + +`token_id` is indexed, so it arrives in `topics[1]`. `uri` is not indexed, so it +sits in the log data as an ABI-encoded dynamic string and is decoded, not +sliced. + +## is_uri_template + +True when `uri` contains the literal `{id}`. + +Such a string is a template, not a URL, and fetching it verbatim fails. Expand +it with the row's `token_id` first — as 64 lowercase hex digits, no `0x` +prefix, which is what the standard requires of a conforming client. + +It also means the string is shared by every token of the contract, so it does +not identify this token. + +The match is literal and case-sensitive. Matching `{ID}` too would flag strings +that no conforming client expands. + +## is_uri_lossy + +True when the on-chain bytes were not valid UTF-8, so `uri` holds U+FFFD +replacement characters that were never on chain. + +Without the column, a mangled string is indistinguishable from a faithful one. +With it, a pipeline can drop or re-read those rows rather than storing bytes the +contract never wrote. + +```bash +triodion erc1155_metadata -b 18M:+10000 \ + --address 0x76be3b62873462d2142405439777e971754e8e77 +``` + +This dataset is not a member of the `log_events` group. Requesting it +alongside `logs` or `erc20_transfers` therefore costs a separate +`eth_getLogs` per block range rather than sharing one with them. diff --git a/book/datasets/erc1155_transfers.md b/book/datasets/erc1155_transfers.md new file mode 100644 index 0000000..f9e99a4 --- /dev/null +++ b/book/datasets/erc1155_transfers.md @@ -0,0 +1,139 @@ +# erc1155_transfers + +One row per token id moved, over both ERC-1155 transfer events. + +``` +erc1155_transfers +───────────────── +- can collect by block or by transaction +- required parameters: [none] +- optional parameters: address, topic1, from_address, to_address +- dataset aliases: [none] +- parameter aliases: + - contracts -> addresses + + +schema for erc1155_transfers +──────────────────────────── +- block_number: uint32 +- transaction_index: uint32 +- log_index: uint32 +- token_id_index: uint32 +- transaction_hash: binary +- erc1155: binary +- operator: binary +- from_address: binary +- to_address: binary +- token_id_binary: binary +- token_id_string: string +- token_id_f64: float64 +- value_binary: binary +- value_string: string +- value_f64: float64 +- is_batch: bool +- is_mint: bool +- is_burn: bool +- chain_id: uint64 + +sorting erc1155_transfers by: block_number, log_index, token_id_index + +other available columns: block_hash +``` + +Run `triodion help erc1155_transfers` to print this from the binary. + +## One table, two events + +ERC-1155 defines two transfer events: + +```solidity +TransferSingle(address indexed operator, address indexed from, address indexed to, + uint256 id, uint256 value) +TransferBatch (address indexed operator, address indexed from, address indexed to, + uint256[] ids, uint256[] values) +``` + +Both describe the same thing — an amount of one token id moving from one party +to another — so both land in this table. `is_batch` says which event a row came +from. + +Both carry a signature plus three indexed arguments, so the topic count cannot +tell them apart. topic0 is the only discriminator. + +## A batch becomes rows, not an array + +`TransferBatch` carries two parallel arrays, `ids` and `values`. Storing them +whole, or storing only their length, would put an aggregate in a cell that +cannot be taken apart again: which token moved, and how much of it, would be +gone. + +A batch of five ids therefore becomes five rows, and a `TransferSingle` becomes +one row of exactly that shape. Every column outside `is_batch` reads the same +either way, so a query never has to branch on which event it was. + +`token_id_index` is the position of a token id within its own log's `ids` list, +and `0` on a `TransferSingle`. Batch order is emitted order and carries meaning, +so it is kept rather than reconstructed. `(log_index, token_id_index)` is the +unique key within a block. + +The name is not `transfer_index`: that word already counts within a block in +[native_transfers](./native_transfers.md), and one word keeps one meaning +across datasets. + +## Sort order + +The default sort is `block_number, log_index, token_id_index`, one column +longer than the log-dataset default. + +Stopping at `log_index` leaves every row of a batch tied, and polars does not +promise to keep tied rows in input order. Sorting on the ordinal too makes the +order total, so emitted batch order survives the write and is the same on every +run. + +## operator, and the two zero-address flags + +`operator` is the `msg.sender` that performed the transfer. It need not be +`from`: an approved operator moves someone else's tokens under their own name. +See [approvals_for_all](./approvals_for_all.md) for where that permission comes +from. + +`is_mint` and `is_burn` are derived, not reported: `from_address` is the zero +address, and `to_address` is the zero address. ERC-1155 encodes mints and burns +that way, so the zero address is not an account that sent or received anything. + +Without those columns, `GROUP BY from_address` reports the zero address as the +busiest sender on the chain, and a holder count counts it as a holder. + +They are two columns rather than one enum because they are independent facts +about a row, and a single column would have to discard one of them. + +## Filtering + +ERC-1155 indexes `operator` first, so `from` and `to` sit one topic slot later +than in ERC-20 and ERC-721: + +| topic | ERC-20 / ERC-721 | ERC-1155 | +| :- | :- | :- | +| topic1 | `from` | `operator` | +| topic2 | `to` | `from` | +| topic3 | token id (ERC-721) | `to` | + +`--from-address` and `--to-address` reach topic2 and topic3 accordingly, so +they mean what they say. `--topic1` is the operator. + +```bash +# every 1155 movement in a range +triodion erc1155_transfers -b 18M:+1000 + +# one collection +triodion erc1155_transfers -b 18M:+1000 \ + --address 0x76be3b62873462d2142405439777e971754e8e77 + +# mints only, on any collection +triodion erc1155_transfers -b 18M:+1000 \ + --from-address 0x0000000000000000000000000000000000000000 +``` + +This dataset is not a member of the `log_events` group. Requesting it +alongside `logs` or `erc20_transfers` therefore costs a separate +`eth_getLogs` per block range rather than sharing one with them. diff --git a/book/datasets/erc20_allowances.md b/book/datasets/erc20_allowances.md new file mode 100644 index 0000000..9659d3e --- /dev/null +++ b/book/datasets/erc20_allowances.md @@ -0,0 +1,132 @@ +# erc20_allowances + +One row per (token, owner, spender) per block: how much of the owner's balance +the spender is currently permitted to move. + +``` +erc20_allowances +──────────────── +- can collect by block and not by transaction +- required parameters: contract, from_address, to_address +- optional parameters: [none] +- dataset aliases: allowances +- parameter aliases: [none] + + +schema for erc20_allowances +─────────────────────────── +- block_number: uint32 +- erc20: binary +- from_address: binary +- to_address: binary +- allowance_binary: binary +- allowance_string: string +- allowance_f64: float64 +- chain_id: uint64 + +sorting erc20_allowances by: block_number, erc20, from_address, to_address + +other available columns: [none] +``` + +Run `triodion help erc20_allowances` to print this from the binary. + +`--blocks` defaults to `latest`. All three parameters are required, and none of +them can be inferred — see [Why all three are required](#why-all-three-are-required). + +```bash +triodion erc20_allowances \ + --contract 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 \ + --from-address 0xc4a622e178ec7dd5a839c64de1b474f5d299f64e \ + --to-address 0x111111125421ca6dc452d289314280a0f8842a65 +``` + +`--from-address` is the owner, whose tokens may be spent. `--to-address` is the +spender, permitted to move them. They are named that way so a row here joins +directly against [erc20_approvals](./erc20_approvals.md), which stores the +`Approval` event's indexed owner and spender in columns of the same names. + +## Which dataset do you want? + +Two datasets answer questions about ERC-20 approvals, and they answer different +ones. Pick by the question, not by the name. + +| | [erc20_approvals](./erc20_approvals.md) | erc20_allowances | +|---|---|---| +| Source | `Approval` event log | `allowance()` call | +| Gives | every approval *change* in a block range | the value in force at a block | +| Owner and spender | discovered | must be named | +| Also gives | transaction hash, log index | nothing but the value | +| Answers | "who approved what, and when" | "what can be spent right now" | + +The event log alone cannot answer the second question, and the difference is +not a small one. + +An allowance also falls when the spender spends. `transferFrom` decrements it +without emitting `Approval` on most implementations, so replaying approval +events overstates what is actually spendable. Only the call reads the truth. + +Here is that gap in real data. Both rows are USDC approvals granted in block +25803911, read back at the same block: + +| owner | spender | `Approval` event value | allowance at end of block | +|---|---|---|---| +| `0xa8c1…1185` | `0x1111…2a65` | 21838278808 | **0** | +| `0xc4a6…f64e` | `0x1111…2a65` | 12000000000 | 12000000000 | + +The first owner approved a router and the router spent it in the same +transaction. The event says 21838278808. Nothing is spendable. + +The two datasets compose well: run `erc20_approvals` over a block range to +*find* the pairs, then feed those pairs to `erc20_allowances` to read what +survives. + +## Why all three are required + +An `allowance()` call takes an owner and a spender. There is no state read that +enumerates the spenders a token has approved, because the keys of a Solidity +mapping are not recoverable from its slots — only `keccak256(key . slot)` is +stored, and that is one-way. + +So this dataset cannot discover pairs. A user who does not know them yet wants +`erc20_approvals`, which finds them in the event log. + +## Null and zero are different + +A successful read of `0` means the spender may move nothing. That is a real +measurement, and it is the state after a revocation. + +A null means the call did not return an allowance at all: the address has no +code at this block, or is not an ERC-20. + +Filling the null with `0` would report "approved for nothing" where the right +answer is "no such token". + +## On "unlimited" approvals + +The common way to grant an unbounded allowance is to set it to +`type(uint256).max`. It is a convention, not a rule. Some tokens and front-ends +use `2^255 - 1`, and some decrement from whatever was set. + +The `allowance` column is therefore the raw value, with no derived "unlimited" +flag. Testing `== 2^256 - 1` would miss the other conventions, and thresholding +at some round number would be this tool inventing a fact. Threshold it yourself, +in the direction your analysis needs. + +## Batching + +Every row is one `eth_call`, so `--multicall` aggregates the rows that share a +block into one `aggregate3`: + +```bash +triodion erc20_allowances --multicall \ + --contract 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 \ + --from-address 0xc4a622e178ec7dd5a839c64de1b474f5d299f64e \ + --to-address 0x111111125421ca6dc452d289314280a0f8842a65 \ + -b 25803911 +``` + +The `eth_call` state-override path used by [slots](./slots.md) does not apply +here. That trick reads *slots*, and finding the slot of +`allowances[owner][spender]` needs the mapping's base slot, which varies per +token and is not discoverable from the ABI. diff --git a/book/datasets/erc2612_nonces.md b/book/datasets/erc2612_nonces.md new file mode 100644 index 0000000..b19af32 --- /dev/null +++ b/book/datasets/erc2612_nonces.md @@ -0,0 +1,85 @@ +# erc2612_nonces + +One row per (token, owner) per block: the ERC-2612 permit nonce. + +``` +erc2612_nonces +────────────── +- can collect by block and not by transaction +- required parameters: contract, address +- optional parameters: [none] +- dataset aliases: [none] +- parameter aliases: [none] + + +schema for erc2612_nonces +───────────────────────── +- block_number: uint32 +- erc20: binary +- address: binary +- nonce_binary: binary +- nonce_string: string +- nonce_f64: float64 +- chain_id: uint64 + +sorting erc2612_nonces by: block_number, erc20, address + +other available columns: domain_separator +``` + +Run `triodion help erc2612_nonces` to print this from the binary. + +`--blocks` defaults to `latest`. Both `--contract` and `--address` are +required: the nonce is per owner per token, so both dimensions are needed to +name a row. + +## Why this dataset exists + +ERC-2612 lets a token owner approve a spender with an off-chain signature. The +`permit` call that redeems that signature emits the ordinary ERC-20 `Approval` +event and nothing else — the standard defines no event of its own. + +In [erc20_approvals](./erc20_approvals.md) a permit-granted approval is +therefore indistinguishable from one granted by an on-chain `approve()`: same +topic0, same owner, same spender, same value. + +The nonce is the only on-chain counter of how many permits an owner has signed +for a token. Read it at two blocks and the difference is the number of permits +redeemed in between. + +## What it does not give you + +It does not give the linkage. It cannot attribute a specific `Approval` row to a +permit. + +Answering that needs the transaction's calldata — the `permit` selector, +possibly nested inside a router's own multicall — which is a different question +from the one asked here. + +## Null and zero are different + +A successful read of `0` means the owner has signed no permit for this token +yet. That is a real measurement. + +A null means the call did not return a nonce at all, so the token has no +ERC-2612 support and the concept does not exist for it. + +Filling the null with `0` would report "never permitted" for tokens that can +never permit. + +## domain_separator + +Off by default. It is the EIP-712 domain separator, and it is constant per token +per chain, so it repeats identically on every row of a run. + +Its null is the cheapest signal that a token is not a permit token: +`DOMAIN_SEPARATOR()` reverts on anything that does not implement ERC-2612. + +Requesting it costs one extra `eth_call` per row, so it stays opt-in: + +```bash +triodion erc2612_nonces \ + --contract 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 \ + --address 0xd8da6bf26964af9d7eed9e03e53415d37aa96045 \ + -i domain_separator +``` diff --git a/book/datasets/erc4626_metadata.md b/book/datasets/erc4626_metadata.md new file mode 100644 index 0000000..fe906e0 --- /dev/null +++ b/book/datasets/erc4626_metadata.md @@ -0,0 +1,85 @@ +# erc4626_metadata + +One row per (vault, block): the three reads that describe an ERC-4626 tokenised +vault at that block. + +``` +erc4626_metadata +──────────────── +- can collect by block and not by transaction +- required parameters: address +- optional parameters: [none] +- dataset aliases: [none] +- parameter aliases: + - contracts -> addresses + + +schema for erc4626_metadata +─────────────────────────── +- block_number: uint32 +- erc4626: binary +- asset: binary +- total_assets_binary: binary +- total_assets_string: string +- total_assets_f64: float64 +- total_supply_binary: binary +- total_supply_string: string +- total_supply_f64: float64 +- chain_id: uint64 + +sorting erc4626_metadata by: erc4626, block_number + +other available columns: [none] +``` + +Run `triodion help erc4626_metadata` to print this from the binary. + +`--blocks` defaults to `latest`. + +## The two tokens + +`erc4626` is the vault, which is itself a token: the share token. `asset` is the +token it holds, read from `asset()`. + +They have their own decimals, and the two need not agree. Nothing in this table +scales one into the other. + +`total_assets` is in the vault's asset units. `total_supply` is shares +outstanding — **not** the supply of the underlying token. + +## Null and zero are different + +A null means the vault refused the read: a revert, or no code at that address at +that block. Zero means the vault answered zero. + +Zero is a real state. A vault that has just been deployed holds zero assets, and +one that has been fully redeemed has zero shares. Neither is a failed read. + +The combination matters: `total_supply` of zero with a non-zero `total_assets` +is the donated-assets state that the ERC-4626 inflation attack exploits. Never +rewrite either as a null. + +## There is no share-price column + +A price per share is `total_assets / total_supply`. Both operands are already in +the row, so the column would add no information while forcing three choices on +every reader: + +- what to do when `total_supply` is 0 — a fresh vault, or one whose shares were all redeemed. The + ratio is undefined, not `1.0`. +- how many decimals to round to. +- which of the two token decimals to scale by. + +Any answer picked here is wrong for someone. + +`convertToAssets(1e18)` is not the same number either, for vaults that charge an +exit fee, so a single column cannot even name one convention. Divide the two +columns yourself, with your own. + +```bash +# sDAI +triodion erc4626_metadata --address 0x83f20f44975d03b1b09e64809b757c47f942beea +``` + +For the flows into and out of a vault, see +[erc4626_vault_events](./erc4626_vault_events.md). diff --git a/book/datasets/erc4626_vault_events.md b/book/datasets/erc4626_vault_events.md new file mode 100644 index 0000000..1baf347 --- /dev/null +++ b/book/datasets/erc4626_vault_events.md @@ -0,0 +1,105 @@ +# erc4626_vault_events + +One row per ERC-4626 `Deposit` or `Withdraw` event on a tokenised vault. + +``` +erc4626_vault_events +──────────────────── +- can collect by block or by transaction +- required parameters: [none] +- optional parameters: address, topic1 +- dataset aliases: erc4626_events +- parameter aliases: + - contracts -> addresses + + +schema for erc4626_vault_events +─────────────────────────────── +- block_number: uint32 +- transaction_index: uint32 +- log_index: uint32 +- transaction_hash: binary +- vault: binary +- event_name: string +- sender: binary +- owner: binary +- receiver: binary +- assets_binary: binary +- assets_string: string +- assets_f64: float64 +- shares_binary: binary +- shares_string: string +- shares_f64: float64 +- chain_id: uint64 + +sorting erc4626_vault_events by: block_number, log_index + +other available columns: block_hash +``` + +Run `triodion help erc4626_vault_events` to print this from the binary. + +Alias: `erc4626_events`. + +## Two halves of one movement + +```solidity +Deposit (address indexed sender, address indexed owner, uint256 assets, uint256 shares) +Withdraw(address indexed sender, address indexed receiver, address indexed owner, + uint256 assets, uint256 shares) +``` + +Both events are kept in one table because they are the two halves of the same +movement: assets in against shares out, and shares in against assets out. +`event_name` — `"Deposit"` or `"Withdraw"`, spelled as the ABI spells them — +says which half a row is. + +`"Withdraw"` is a different event from the `"withdrawal"` of +[erc20_wrapper_events](./erc20_wrapper_events.md). The word is close; the event +is not. + +`receiver` is null on every `Deposit` row. `Deposit` has no receiver argument at +all — the shares go to `owner` — so there is nothing to record. + +## This is not a WETH deposit + +ERC-4626's `Deposit(address,address,uint256,uint256)` is **not** the WETH-style +`Deposit(address,uint256)` that +[erc20_wrapper_events](./erc20_wrapper_events.md) collects. + +The two signatures hash to different topic0 values, so the filters here and +there select disjoint sets of logs. A vault deposit can never turn up as a +wrapper deposit. Do not dedupe across the two datasets on that assumption. + +## There is no share-price column + +The price is `assets / shares`. That division's scaling and rounding belong to +whoever asks the question, and `shares == 0` makes it undefined — a value this +table would then have to invent. Both operands are on the row; divide them with +your own convention. + +## Filtering + +topic1 is `sender` in both events, so filtering on it means one thing. + +topic2 and topic3 are deliberately not offered: topic2 is `owner` on a +`Deposit` but `receiver` on a `Withdraw`, so one filter value would silently +select two different roles. + +```bash +triodion erc4626_vault_events -b 18M:+1000 \ + --address 0x83f20f44975d03b1b09e64809b757c47f942beea +``` + +## What counts as a vault + +Matching happens at the event-signature level with no per-contract +configuration. Any contract emitting these exact signatures is indexed, whether +or not it is a conforming vault. Use +[contract_interfaces](./contract_interfaces.md) or +[erc4626_metadata](./erc4626_metadata.md) to check what a given address +actually is. + +This dataset is not a member of the `log_events` group. Requesting it +alongside `logs` or `erc20_transfers` therefore costs a separate +`eth_getLogs` per block range rather than sharing one with them. diff --git a/book/datasets/erc777_transfers.md b/book/datasets/erc777_transfers.md new file mode 100644 index 0000000..740f1f6 --- /dev/null +++ b/book/datasets/erc777_transfers.md @@ -0,0 +1,131 @@ +# erc777_transfers + +One row per ERC-777 `Sent`, `Minted` or `Burned` event. + +``` +erc777_transfers +──────────────── +- can collect by block or by transaction +- required parameters: [none] +- optional parameters: address, topic1, topic2, topic3 +- dataset aliases: [none] +- parameter aliases: + - contracts -> addresses + + +schema for erc777_transfers +─────────────────────────── +- block_number: uint32 +- transaction_index: uint32 +- log_index: uint32 +- transaction_hash: binary +- erc777: binary +- event_name: string +- operator: binary +- from_address: binary +- to_address: binary +- amount_binary: binary +- amount_string: string +- amount_f64: float64 +- data: binary +- operator_data: binary +- is_operator_send: bool +- chain_id: uint64 + +sorting erc777_transfers by: block_number, log_index + +other available columns: block_hash +``` + +Run `triodion help erc777_transfers` to print this from the binary. + +## Do not union this with erc20_transfers + +ERC-777 is a superset of ERC-20. A compliant ERC-777 token emits an ERC-20 +`Transfer` **for the same movement** alongside every `Sent`, `Minted` and +`Burned`. + +Both logs are real, both are in the receipt, and both are collected — +`Transfer` by [erc20_transfers](./erc20_transfers.md), the richer one here. +They are two views of one movement, not two movements. + +So this: + +```sql +SELECT ... FROM erc20_transfers +UNION ALL +SELECT ... FROM erc777_transfers -- WRONG +``` + +reports every ERC-777 movement twice, and every volume, count and balance +derived from it is wrong by exactly the ERC-777 share of the chain. + +Pick one table per token: this one when the operator or the data payloads +matter, `erc20_transfers` when only the movement does. If you must have one +table, anti-join on `(transaction_hash, block_number, erc777)` — never on +`log_index`, because the mirrored pair are two different logs with two +different indices. + +The mirroring is a convention, not a consensus rule. A token can emit one +without the other, so neither table is a strict subset of the other. + +## event_name decides which columns are null + +`event_name` holds the Solidity identifier as the standard writes it — +`Sent`, `Minted`, `Burned` — so the value joins straight to an ABI without a +translation table. It also decides which address column is null: + +| event_name | from_address | to_address | is_operator_send | +| :- | :- | :- | :- | +| `Sent` | the holder | the recipient | `operator != from_address` | +| `Minted` | null | the recipient | null | +| `Burned` | the holder | null | `operator != from_address` | + +A mint has no payer and a burn names no recipient, and ERC-777 does not use the +ERC-20 zero-address convention for either. Writing `0x0` there would invent a +party the event never named. + +## operator, and the column that is not it + +`operator` is the party that executed the move. This is what ERC-777 adds over +ERC-20: an authorised operator moves a holder's tokens with no allowance and no +call from the holder. + +It is always present. On an ordinary self-initiated send the holder is its own +operator, so `COUNT(DISTINCT operator)` counts every ordinary sender as an +operator. `is_operator_send` is the column that separates the two — derived, +not reported, as `operator != from_address`. + +It is null on `Minted`, where there is no `from` to compare against. `false` +there would claim a self-send the event never described. + +## data and operator_data + +Both are non-indexed dynamic `bytes` from the log body, ABI-decoded. `data` is +the holder's payload and `operator_data` is the operator's own, which is why +they are separate columns. + +Empty is a real and common value here — most sends carry no payload — so it is +stored as zero-length bytes, never as a null. A null would mean the field does +not exist, and it exists on all three events. + +## Filtering by topic, not by name + +The three events do not agree on what each topic holds: + +| topic | Sent | Minted | Burned | +| :- | :- | :- | :- | +| topic1 | operator | operator | operator | +| topic2 | `from` | `to` | `from` | +| topic3 | `to` | — | — | + +So topic1 is offered, and topics 2 and 3 are offered **by position** rather +than under address names. A `--from-address` dim would have to mean topic2, and +would silently match `Minted` recipients too. + +Filtering on topic3 also drops every `Minted` and `Burned` row, because those +logs have no topic3 for the node to match against. + +This dataset is not a member of the `log_events` group. Requesting it +alongside `logs` or `erc20_transfers` therefore costs a separate +`eth_getLogs` per block range rather than sharing one with them. diff --git a/book/datasets/proxy_slots.md b/book/datasets/proxy_slots.md new file mode 100644 index 0000000..1349fd1 --- /dev/null +++ b/book/datasets/proxy_slots.md @@ -0,0 +1,97 @@ +# proxy_slots + +One row per (block, address), holding the three ERC-1967 proxy storage slots. + +``` +proxy_slots +─────────── +- can collect by block and not by transaction +- required parameters: address +- optional parameters: [none] +- dataset aliases: erc1967_slots +- parameter aliases: + - contracts -> addresses + + +schema for proxy_slots +────────────────────── +- block_number: uint32 +- address: binary +- implementation: binary +- admin: binary +- beacon: binary +- chain_id: uint64 + +sorting proxy_slots by: block_number, address + +other available columns: [none] +``` + +Run `triodion help proxy_slots` to print this from the binary. + +Alias: `erc1967_slots`. + +`--blocks` defaults to `latest`. + +## The slots + +ERC-1967 fixes each slot at `keccak256(label) - 1`: + +| column | label | slot | +| :- | :- | :- | +| `implementation` | `eip1967.proxy.implementation` | `0x360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc` | +| `admin` | `eip1967.proxy.admin` | `0xb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d6103` | +| `beacon` | `eip1967.proxy.beacon` | `0xa3f0ad74e5423aebfd80d3ef4346578335a9a72aeaee59ff6cb3582b35133d50` | + +The `- 1` is the point of the construction. It puts the slot outside the image +of `keccak256`, so no mapping or array the contract declares can ever be laid +out on top of it. + +## The fourth slot is not read + +ERC-1967 defines a rollback slot at +`0x4910fdfa16fed3260ed0e7147f7cc6da11a60208b5b9406d12a635614ffd9143`. It is +deliberately absent. + +It is written and cleared inside a single upgrade transaction, so any read at a +block boundary sees it unset. A column of it would be all nulls, and would +invite the reading "no proxy here ever rolled back", which the data cannot +support. + +## Null means not that kind of proxy + +A null means the slot is unset at that block. Every ordinary, non-proxy contract +answers null in all three columns, which is the expected result for most +addresses. + +## implementation and beacon are separate on purpose + +- `implementation` set, `beacon` null — a standard ERC-1967 proxy. The code it delegates to is in + `implementation`. +- `beacon` set — a beacon proxy. Its real implementation is held by the beacon contract, reachable + only through `IBeacon.implementation()` on that address. It is **not** in this row and must not be + inferred from it. + +The two columns are separate precisely so no single column has to mean both +things. + +## The slot view and the event view + +This dataset answers "what is the implementation at block N". +[proxy_upgrades](./proxy_upgrades.md) reads the ERC-1967 events and answers +"when did it change, and in which transaction". Join `proxy_slots.address` to +`proxy_upgrades.proxy_address`. + +A slot read is the only authority on the current implementation. The events are +a convention, not an EVM rule, so a proxy can write the slot and emit nothing — +and a beacon that swaps its own implementation moves every proxy behind it with +no event at any of them. + +```bash +# what does this proxy point at now +triodion proxy_slots --address 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 + +# and how did that change over a range +triodion proxy_slots -b 18M:19M:100000 \ + --address 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 +``` diff --git a/book/datasets/proxy_upgrades.md b/book/datasets/proxy_upgrades.md new file mode 100644 index 0000000..eb56f2f --- /dev/null +++ b/book/datasets/proxy_upgrades.md @@ -0,0 +1,112 @@ +# proxy_upgrades + +One row per ERC-1967 proxy event: `Upgraded`, `BeaconUpgraded` or +`AdminChanged`. + +``` +proxy_upgrades +────────────── +- can collect by block or by transaction +- required parameters: [none] +- optional parameters: address +- dataset aliases: erc1967_events +- parameter aliases: + - contracts -> addresses + + +schema for proxy_upgrades +───────────────────────── +- block_number: uint32 +- transaction_index: uint32 +- log_index: uint32 +- transaction_hash: binary +- proxy_address: binary +- event_name: string +- implementation: binary +- beacon: binary +- previous_admin: binary +- new_admin: binary +- chain_id: uint64 + +sorting proxy_upgrades by: block_number, log_index + +other available columns: block_hash +``` + +Run `triodion help proxy_upgrades` to print this from the binary. + +Alias: `erc1967_events`. + +## One table, three events + +```solidity +Upgraded (address indexed implementation) +BeaconUpgraded(address indexed beacon) +AdminChanged (address previousAdmin, address newAdmin) +``` + +`event_name` holds the identifier as the ABI spells it, so a row joins back to +the event definition without a translation table. + +Three separate tables would each be near-empty, and would have to be merged +again before "what changed on this proxy, and in what order" could be asked. So +this is one table, and each event fills only its own columns: + +| event_name | implementation | beacon | previous_admin | new_admin | +| :- | :- | :- | :- | :- | +| `Upgraded` | set | null | null | null | +| `BeaconUpgraded` | null | set | null | null | +| `AdminChanged` | null | null | set | set | + +`proxy_address` is the contract that emitted the event: the proxy itself, never +the implementation it points at. + +## Null, not zero + +The columns an event does not carry are null. The zero address is a real value +here and must stay distinguishable from an absence: + +- a proxy upgraded to `0x0` is bricked +- an admin changed to `0x0` has been renounced + +Both are things that happened. A null means the event never named that field at +all. + +## What BeaconUpgraded does not tell you + +The implementation behind a beacon is not in the log. `BeaconUpgraded` names +the beacon contract; the code it serves must be read from that contract, via +`IBeacon.implementation()`. + +## The event view and the slot view + +This dataset answers "when did it change, and in which transaction". +[proxy_slots](./proxy_slots.md) reads the same ERC-1967 storage slots directly +and answers "what is the implementation at block N". Join +`proxy_upgrades.proxy_address` to `proxy_slots.address`. + +Neither replaces the other: + +- These events are a convention, not an EVM rule. A proxy can write the implementation slot and + emit nothing, and a beacon that swaps its own implementation moves every proxy behind it with no + event at any of them. **No rows here is not evidence that nothing changed** — only a slot read + is. +- A slot read cannot say how many times the slot changed between two blocks, or which transaction + changed it. + +## Filtering + +Only `--address` is offered. topic1 is deliberately not exposed: it is the +implementation on `Upgraded`, the beacon on `BeaconUpgraded`, and does not exist +on `AdminChanged`. Any `--topic1` value would silently delete every +`AdminChanged` row and mean two different things across the rows it kept. + +```bash +# every upgrade of one proxy +triodion proxy_upgrades -b 1:latest \ + --address 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 +``` + +This dataset is not a member of the `log_events` group. Requesting it +alongside `logs` or `erc20_transfers` therefore costs a separate +`eth_getLogs` per block range rather than sharing one with them. diff --git a/crates/cli/src/args.rs b/crates/cli/src/args.rs index 51af3ac..f098b73 100644 --- a/crates/cli/src/args.rs +++ b/crates/cli/src/args.rs @@ -90,11 +90,11 @@ pub struct Args { #[arg(short, long, help_heading = "Source Options")] pub rpc: Option, - /// L1 (settlement) RPC url for L2 datasets that read L1-side events. + /// L1 (settlement) RPC url, reserved for L2 datasets that read L1-side events. /// - /// Required by datasets like `op_batch_submissions`, `arb_retryable_tickets`, - /// `zks_l1_batches`, `l1_to_l2_inbox`, `l2_output_roots_on_l1` and the pan-L2 - /// `blob_sidecars` table. Ignored when the active dataset is L2-only. + /// No dataset reads it yet. It connects and reports its chain id so the + /// plumbing stays exercised, and is otherwise inert. The help text here + /// used to name six such datasets, none of which exist. #[arg(long, value_name = "URL", help_heading = "Source Options")] pub l1_rpc: Option, @@ -336,6 +336,58 @@ pub struct Args { /// null) #[arg(long, help_heading = "Dataset-specific Options")] pub multicall_require_success: bool, + + /// Disable `eth_call` state-override reads for `slots` / `balances` / `proxy_slots`. + /// + /// On by default. Replaces the target's code with a small extractor loop + /// that answers a whole batch of slots (or balances) in one request, + /// instead of one `eth_getStorageAt` / `eth_getBalance` per row. + /// + /// Turn this off for an endpoint you do not trust to honour the third + /// `eth_call` parameter. Plain JSON-RPC batching is a separate switch + /// (`--no-batch-rpc-calls`) because it asks nothing of the node. + /// + /// A batch that fails falls back to one request per row, so this switch + /// cannot change results — only how many requests they cost. + #[arg( + long = "no-batch-state-reads", + action = clap_cryo::ArgAction::SetFalse, + default_value_t = true, + help_heading = "Dataset-specific Options" + )] + pub batch_state_reads: bool, + + /// Slots / addresses per state-override call (0 = the dataset's own default). + /// + /// Applies to the extractor path only (`slots`, `balances`, `proxy_slots`); the JSON-RPC + /// batch path used by `blocks`, `codes` and `nonces` negotiates its own size + /// with the provider. + /// + /// The default is 1000, chosen from measurement rather than the gas ceiling: + /// 10,000 slots in one call works but a 20,000-slot request is refused for + /// payload size, and ten parallel 1000-slot calls finish sooner than one + /// 10,000-slot call. Raise it when pointing at your own node. + #[arg(long, value_name = "N", default_value_t = 0, help_heading = "Dataset-specific Options")] + pub state_override_batch_size: u32, + + /// Disable JSON-RPC request batching for `blocks` / `nonces` / `codes`. + /// + /// On by default. Packs many identical calls into one HTTP request — 100 + /// `eth_getBlockByNumber` calls in one envelope rather than 100 requests. + /// It needs nothing from the node beyond JSON-RPC itself, which is why it + /// is a separate switch from `--no-batch-state-reads`. + /// + /// The batch size negotiates itself downward when a provider refuses one + /// (OP Mainnet answers `413` above ten calls), and a batch that still fails + /// falls back to one request per row. Turn this off only for an endpoint + /// that mishandles batch envelopes outright. + #[arg( + long = "no-batch-rpc-calls", + action = clap_cryo::ArgAction::SetFalse, + default_value_t = true, + help_heading = "Dataset-specific Options" + )] + pub batch_rpc_calls: bool, } impl Args { diff --git a/crates/cli/src/parse/blocks.rs b/crates/cli/src/parse/blocks.rs index 67c25b3..3e4bbf4 100644 --- a/crates/cli/src/parse/blocks.rs +++ b/crates/cli/src/parse/blocks.rs @@ -439,6 +439,8 @@ mod tests { l1_provider: None, l1_chain_id: None, l1_rpc_url: None, + state_override_support: Arc::new(Default::default()), + storage_values_misses: Arc::new(std::sync::atomic::AtomicU32::new(0)), beacon: None, }; let source = Arc::new(source); @@ -505,6 +507,8 @@ mod tests { l1_provider: None, l1_chain_id: None, l1_rpc_url: None, + state_override_support: Arc::new(Default::default()), + storage_values_misses: Arc::new(std::sync::atomic::AtomicU32::new(0)), beacon: None, }); for (test, res) in tests { @@ -582,6 +586,8 @@ mod tests { l1_provider: None, l1_chain_id: None, l1_rpc_url: None, + state_override_support: Arc::new(Default::default()), + storage_values_misses: Arc::new(std::sync::atomic::AtomicU32::new(0)), beacon: None, }; let source = Arc::new(source); diff --git a/crates/cli/src/parse/query.rs b/crates/cli/src/parse/query.rs index 0bf1050..eae8ef7 100644 --- a/crates/cli/src/parse/query.rs +++ b/crates/cli/src/parse/query.rs @@ -28,6 +28,9 @@ pub async fn parse_query(args: &Args, source: Arc) -> Result Result { l1_provider, l1_chain_id, l1_rpc_url, + state_override_support: Arc::new(Default::default()), + storage_values_misses: Arc::new(std::sync::atomic::AtomicU32::new(0)), beacon, }; diff --git a/crates/cli/src/parse/timestamps.rs b/crates/cli/src/parse/timestamps.rs index d5b60dd..c3f6f96 100644 --- a/crates/cli/src/parse/timestamps.rs +++ b/crates/cli/src/parse/timestamps.rs @@ -376,6 +376,8 @@ mod tests { l1_provider: None, l1_chain_id: None, l1_rpc_url: None, + state_override_support: Arc::new(Default::default()), + storage_values_misses: Arc::new(std::sync::atomic::AtomicU32::new(0)), beacon: None, }) } diff --git a/crates/freeze/src/datasets/address_appearances.rs b/crates/freeze/src/datasets/address_appearances.rs index 1689279..1636133 100644 --- a/crates/freeze/src/datasets/address_appearances.rs +++ b/crates/freeze/src/datasets/address_appearances.rs @@ -48,9 +48,6 @@ impl CollectByBlock for AddressAppearances { async fn extract(request: Params, source: Arc, _: Arc) -> R { let block_number = request.ethers_block_number()?; - let block = - source.get_block(request.block_number()?, BlockTransactionsKind::Hashes).await?; - let block = block.ok_or(CollectError::CollectError("block not found".to_string()))?; let filter = Filter { block_option: FilterBlockOption::Range { from_block: Some(block_number.into()), @@ -58,8 +55,21 @@ impl CollectByBlock for AddressAppearances { }, ..Default::default() }; - let logs = source.get_logs(&filter).await?; - let traces = source.trace_block(request.block_number()?).await?; + + // Three independent reads of the same block: none of them needs another's + // answer, so awaiting them in sequence spent three round trips on one + // block's worth of latency. `try_join!` issues all three and waits once. + // + // The request count is unchanged — the node still sees three calls, and + // still charges three permits from the concurrency semaphore. What + // changes is that a partition of N blocks costs N round trips of + // latency instead of 3N. + let (block, logs, traces) = futures::try_join!( + source.get_block(request.block_number()?, BlockTransactionsKind::Hashes), + source.get_logs(&filter), + source.trace_block(request.block_number()?), + )?; + let block = block.ok_or(CollectError::CollectError("block not found".to_string()))?; Ok((block, logs, traces)) } @@ -75,9 +85,24 @@ impl CollectByTransaction for AddressAppearances { async fn extract(request: Params, source: Arc, _: Arc) -> R { let tx_hash = request.ethers_transaction_hash()?; - let tx_data = source.get_transaction_by_hash(tx_hash).await?.ok_or_else(|| { + // The receipt and the traces are keyed by transaction hash, which we + // already have, so neither waits on the transaction body. Only the block + // does — it is fetched by number, and the number comes out of the body. + // So the four reads collapse into two round trips rather than four. + let (tx_data, receipt, traces) = futures::try_join!( + source.get_transaction_by_hash(tx_hash), + source.get_transaction_receipt(tx_hash), + source.trace_transaction(tx_hash), + )?; + + let tx_data = tx_data.ok_or_else(|| { CollectError::CollectError("could not find transaction data".to_string()) })?; + let logs = receipt + .ok_or(CollectError::CollectError("could not get tx receipt".to_string()))? + .inner + .logs() + .to_vec(); let block_number = tx_data .block_number @@ -87,18 +112,6 @@ impl CollectByTransaction for AddressAppearances { .await? .ok_or(CollectError::CollectError("could not get block".to_string()))?; - // logs - let logs = source - .get_transaction_receipt(tx_hash) - .await? - .ok_or(CollectError::CollectError("could not get tx receipt".to_string()))? - .inner - .logs() - .to_vec(); - - // traces - let traces = source.trace_transaction(request.ethers_transaction_hash()?).await?; - Ok((block, logs, traces)) } @@ -123,6 +136,21 @@ fn name(log: &Log) -> Option<&'static str> { } } +/// The two parties to an ERC-20 or ERC-721 `Transfer`, in `(from, to)` order. +/// +/// Returned as a pair deliberately. Both addresses are the low 20 bytes of an +/// indexed topic and differ only in *which* topic, so reading one of them twice +/// reads correctly at the call site. That is exactly what happened here: both +/// reads took `topics[1]`, so every transfer recipient was dropped and every +/// sender was written twice. +/// +/// # Panics +/// +/// If `log` carries fewer than three topics. Callers filter on that first. +fn transfer_parties(log: &Log) -> (Address, Address) { + (Address::from_slice(&log.topics()[1][12..32]), Address::from_slice(&log.topics()[2][12..32])) +} + impl AddressAppearances { fn process_first_transaction( &mut self, @@ -139,31 +167,21 @@ impl AddressAppearances { if let Some(logs) = logs_by_tx.get(&tx_hash) { for log in logs.iter() { if log.topics().len() >= 3 { - if let Some(name) = name(log) { - let mut from: [u8; 20] = [0; 20]; - from.copy_from_slice(&log.topics()[1][12..32]); - - let name = &(name.to_string() + "_from"); - self.process_address( - Address::from_slice(&from), - name, - block_number, - &block_hash, - tx_hash, - schema, - ); - - let mut to: [u8; 20] = [0; 20]; - to.copy_from_slice(&log.topics()[1][12..32]); - let name = &(name.to_string() + "_to"); - self.process_address( - Address::from_slice(&to), - name, - block_number, - &block_hash, - tx_hash, - schema, - ); + if let Some(event) = name(log) { + // Derive both labels from `event`, never from each + // other: the old code shadowed `name`, so the second + // label came out as "..._from_to". + let (from, to) = transfer_parties(log); + for (address, suffix) in [(from, "_from"), (to, "_to")] { + self.process_address( + address, + &(event.to_string() + suffix), + block_number, + &block_hash, + tx_hash, + schema, + ); + } } } } @@ -339,3 +357,29 @@ fn process_appearances( Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + use alloy::primitives::{Bytes, LogData}; + + fn transfer_log(from: Address, to: Address) -> Log { + let inner = alloy::primitives::Log { + address: Address::repeat_byte(0xda), + data: LogData::new_unchecked( + vec![ERC20::Transfer::SIGNATURE_HASH, from.into_word(), to.into_word()], + Bytes::from(vec![0u8; 32]), + ), + }; + Log { inner, ..Default::default() } + } + + #[test] + fn the_transfer_parties_come_from_different_topics() { + let from = Address::repeat_byte(0x11); + let to = Address::repeat_byte(0x22); + // Before the fix this returned `(from, from)`: the recipient of every + // ERC-20 and ERC-721 transfer was absent from the dataset. + assert_eq!(transfer_parties(&transfer_log(from, to)), (from, to)); + } +} diff --git a/crates/freeze/src/datasets/approvals_for_all.rs b/crates/freeze/src/datasets/approvals_for_all.rs new file mode 100644 index 0000000..5e753a4 --- /dev/null +++ b/crates/freeze/src/datasets/approvals_for_all.rs @@ -0,0 +1,367 @@ +use crate::{ + types::rpc_params::{ + address_topic_matches, fixed_from_slice, log_address_matches, topic_matches, + }, + *, +}; +use alloy::{ + primitives::B256, + rpc::types::{Filter, Log, Topic}, + sol_types::SolEvent, +}; +use polars::prelude::*; + +/// columns for `ApprovalForAll(address,address,bool)` logs. +/// +/// ERC-721 and ERC-1155 declare this event with the same argument types, so +/// both hash to one topic0. A log that carries it does not say which of the two +/// standards the emitting contract implements, and no decoding can recover +/// that: the bytes are identical. This is why the dataset is not called +/// `erc1155_approvals` — that name would be false for about half of its rows. +/// To classify a contract, join `contract_address` against the +/// `contract_interfaces` dataset, which asks the contract itself through the +/// ERC-165 `supportsInterface` call. +/// +/// `approved == false` is a revocation. It is a measurement of what happened, +/// not an absence of one, so it is stored as `false` and never as null. +/// +/// These rows are an event log, not a state table. `WHERE approved` on its own +/// does not give the live approvals: an operator approved in one block and +/// revoked in the next leaves both rows behind, and the filter keeps the stale +/// one. First take the row with the largest `(block_number, log_index)` for +/// each `(contract_address, owner, operator)`, then filter on `approved`. +#[triodion_macros::to_df(Datatype::ApprovalsForAll)] +#[derive(Default)] +pub struct ApprovalsForAll { + n_rows: u64, + block_number: Vec, + block_hash: Vec>>, + transaction_index: Vec, + log_index: Vec, + transaction_hash: Vec>, + // The token contract that emitted the log. Which standard it implements is + // not recorded anywhere in the log; see the note above the struct. + contract_address: Vec>, + // The token holder whose permission changes. ERC-1155 names this argument + // `account` and ERC-721 names it `owner`; it is the same topic either way. + owner: Vec>, + // The address that gains or loses permission over every token the owner + // holds in this contract. Unlike the ERC-721 `Approval` event, this grant + // names no token id, so it also covers tokens acquired after this block. + operator: Vec>, + // `true` grants, `false` revokes. Both are events that happened. + approved: Vec, + chain_id: Vec, +} + +impl Dataset for ApprovalsForAll { + fn aliases() -> Vec<&'static str> { + // Both names arrive at the same rows, because both standards emit the + // same topic0. Neither name is used for the dataset itself, so that no + // output file claims a standard the logs cannot prove. + vec!["erc721_approvals_for_all", "erc1155_approvals"] + } + + fn default_columns() -> Option> { + Some(vec![ + "block_number", + // "block_hash", + "transaction_index", + "log_index", + "transaction_hash", + "contract_address", + "owner", + "operator", + "approved", + "chain_id", + ]) + } + + fn optional_parameters() -> Vec { + // Topic0 is fixed by the definition of this dataset, so it is not + // offered. Topic1 is the owner and Topic2 is the operator, and both are + // reachable two ways: as a raw 32-byte topic, or as an ordinary 20-byte + // address via `--from-address` / `--to-address`, which is what + // `erc20_approvals` calls its owner and spender. The names are loose — + // an approval has no direction — but demanding a hand-padded 32-byte + // word for a value the user reads off a block explorer is worse. + // Unlike the multi-event log datasets, this dataset carries exactly one + // event signature, so topic1 and topic2 mean the same thing on every + // row and address-shaped filtering is unambiguous here. + vec![Dim::Address, Dim::Topic1, Dim::Topic2, Dim::FromAddress, Dim::ToAddress] + } + + fn use_block_ranges() -> bool { + true + } + + fn default_inner_request_size() -> u64 { + // Log fetch is one HTTP request per block range. 50 blocks per request + // matches the other log datasets; ApprovalForAll is far rarer per block + // than Transfer, so the response stays small. Override with + // --inner-request-size. + 50 + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for ApprovalsForAll { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let mut topics: [Topic; 4] = Default::default(); + topics[0] = APPROVAL_FOR_ALL_TOPIC0.into(); + + // Each slot is reachable two ways, and supplying both is refused + // rather than resolved by precedence. Preferring one silently discards + // the other, and the discarded dimension does not go away: it is still + // a partition dimension, so the run is still multiplied by it and + // emits N identically-named-apart files holding identical rows from + // one identical `eth_getLogs`. An error costs the user one flag; the + // alternative costs them N copies of the same data. + if request.topic1.is_some() && request.from_address.is_some() { + return Err(err("--topic1 and --from-address both select topic1; give one, not both")) + } + if request.topic2.is_some() && request.to_address.is_some() { + return Err(err("--topic2 and --to-address both select topic2; give one, not both")) + } + + if let Some(owner) = &request.topic1 { + topics[1] = fixed_from_slice::(owner, "topic1")?.into(); + } else if let Some(owner) = &request.from_address { + let v = address_dim_as_topic(owner) + .ok_or_else(|| err("from_address must be at most 32 bytes"))?; + topics[1] = v.into(); + } + if let Some(operator) = &request.topic2 { + topics[2] = fixed_from_slice::(operator, "topic2")?.into(); + } else if let Some(operator) = &request.to_address { + let v = address_dim_as_topic(operator) + .ok_or_else(|| err("to_address must be at most 32 bytes"))?; + topics[2] = v.into(); + } + let filter = Filter { topics, ..request.ethers_log_filter()? }; + let logs = source.get_logs(&filter).await?; + Ok(logs.into_iter().filter(is_approval_for_all_shape).collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::ApprovalsForAll)?; + process_approvals_for_all(response, columns, schema) + } +} + +impl CollectByTransaction for ApprovalsForAll { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let logs = source.get_transaction_logs(request.transaction_hash()?).await?; + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + // + // Each slot is reachable two ways, and the by-block path lets the raw + // topic win, so this one does too. It cannot be reached with both set: + // `CollectByBlock::extract` refuses that combination. + let owner_matches = |log: &Log| match &request.topic1 { + Some(_) => topic_matches(log, 1, &request.topic1), + None => address_topic_matches(log, 1, &request.from_address), + }; + let operator_matches = |log: &Log| match &request.topic2 { + Some(_) => topic_matches(log, 2, &request.topic2), + None => address_topic_matches(log, 2, &request.to_address), + }; + Ok(logs + .into_iter() + .filter(|log| { + is_approval_for_all(log) && + log_address_matches(log, &request.address) && + owner_matches(log) && + operator_matches(log) + }) + .collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::ApprovalsForAll)?; + process_approvals_for_all(response, columns, schema) + } +} + +/// topic0 of `ApprovalForAll(address,address,bool)`. +/// +/// Taken from the ERC-721 declaration, but the ERC-1155 declaration produces +/// the same 32 bytes, and one filter therefore returns the logs of both. The +/// test below holds the two constants equal so that a future edit to either +/// `sol!` block cannot make this dataset silently drop half of its rows. +const APPROVAL_FOR_ALL_TOPIC0: B256 = ERC721::ApprovalForAll::SIGNATURE_HASH; + +/// True iff `log` has the `ApprovalForAll` shape: 3 topics (signature, indexed +/// owner, indexed operator) and a single 32-byte word of data. +fn is_approval_for_all_shape(log: &Log) -> bool { + log.topics().len() == 3 && log.data().data.len() == 32 +} + +/// True iff `log` is an `ApprovalForAll`: the shape check plus a topic0 match. +/// The per-transaction path needs the topic0 check because, unlike the by-block +/// path, `eth_getLogs` did not pre-filter the signature — we hold every log of +/// the transaction and must pick these out. Written to stay reusable by a +/// coalesced [`crate::LogEvents`] fan-out, which this datatype is NOT a member +/// of today (see `MultiDatatype::LogEvents`). +pub(crate) fn is_approval_for_all(log: &Log) -> bool { + is_approval_for_all_shape(log) && + log.topics().first().is_some_and(|t| *t == APPROVAL_FOR_ALL_TOPIC0) +} + +/// Read an ABI-encoded bool out of its 32-byte word. +/// +/// Solidity writes 0 or 1, but a log built by hand in assembly is under no such +/// rule, and a dirty word must not read as `false`. Judge it the way the EVM +/// judges truth, with `ISZERO`: any non-zero word is `true`. +fn decode_bool_word(word: &[u8]) -> bool { + word.iter().any(|byte| *byte != 0) +} + +/// process logs into columns +fn process_approvals_for_all( + logs: Vec, + columns: &mut ApprovalsForAll, + schema: &Table, +) -> R<()> { + for log in logs.iter() { + // Shape guard first. The columns below index topics()[1] and [2], so a + // log with this topic0 but fewer topics would panic the worker task. + // Every current caller pre-filters; this keeps a future caller that + // arrives with mixed logs safe, and skips such a log instead. + if !is_approval_for_all(log) { + continue; + } + if let (Some(bn), Some(tx), Some(ti), Some(li)) = + (log.block_number, log.transaction_hash, log.transaction_index, log.log_index) + { + columns.n_rows += 1; + store!(schema, columns, block_number, bn as u32); + store!(schema, columns, block_hash, log.block_hash.map(|bh| bh.to_vec())); + store!(schema, columns, transaction_index, ti as u32); + store!(schema, columns, log_index, li as u32); + store!(schema, columns, transaction_hash, tx.to_vec()); + store!(schema, columns, contract_address, log.address().to_vec()); + // Indexed addresses sit left-padded in a 32-byte topic; keep the + // low 20 bytes. The guard above proves both topics are present. + store!(schema, columns, owner, log.topics()[1][12..].to_vec()); + store!(schema, columns, operator, log.topics()[2][12..].to_vec()); + store!(schema, columns, approved, decode_bool_word(&log.data().data)); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy::primitives::{Address, Bytes}; + + /// real `ApprovalsForAll` schema, so `store!` actually evaluates each column + fn schema() -> Table { + Datatype::ApprovalsForAll + .table_schema( + &[U256Type::String], + &ColumnEncoding::Hex, + &None, + &None, + &None, + None, + None, + ) + .expect("ApprovalsForAll has a valid default schema") + } + + /// rpc `Log` carrying the block/tx/index fields `process_approvals_for_all` needs + fn log_with(topics: Vec, data: Vec) -> Log { + let inner = alloy::primitives::Log::new_unchecked(Address::ZERO, topics, Bytes::from(data)); + Log { + inner, + block_number: Some(1), + transaction_hash: Some(B256::ZERO), + transaction_index: Some(0), + log_index: Some(0), + ..Default::default() + } + } + + /// 32-byte ABI word holding `value` in its last byte + fn word(value: u8) -> Vec { + let mut data = vec![0u8; 32]; + data[31] = value; + data + } + + #[test] + fn erc721_and_erc1155_share_one_topic0() { + // The whole name of this dataset rests on this equality. If it ever + // fails, one topic0 no longer covers both standards and the extractor + // must filter on a list of two. + assert_eq!(ERC721::ApprovalForAll::SIGNATURE_HASH, ERC1155::ApprovalForAll::SIGNATURE_HASH); + } + + #[test] + fn revocation_is_stored_as_false_not_dropped() { + let logs = vec![ + log_with(vec![APPROVAL_FOR_ALL_TOPIC0, B256::ZERO, B256::ZERO], word(1)), + log_with(vec![APPROVAL_FOR_ALL_TOPIC0, B256::ZERO, B256::ZERO], word(0)), + ]; + let mut columns = ApprovalsForAll::default(); + process_approvals_for_all(logs, &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 2, "a revocation is a row, not a missing row"); + assert_eq!(columns.approved, vec![true, false]); + } + + #[test] + fn a_dirty_bool_word_reads_as_true() { + // Not written by solc, but the EVM would treat it as true, and reading + // it as false would report a grant as a revocation. + let log = log_with(vec![APPROVAL_FOR_ALL_TOPIC0, B256::ZERO, B256::ZERO], word(2)); + let mut columns = ApprovalsForAll::default(); + process_approvals_for_all(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.approved, vec![true]); + } + + #[test] + fn owner_and_operator_come_from_topics_1_and_2() { + let owner = B256::left_padding_from(&[0xaa; 20]); + let operator = B256::left_padding_from(&[0xbb; 20]); + let log = log_with(vec![APPROVAL_FOR_ALL_TOPIC0, owner, operator], word(1)); + let mut columns = ApprovalsForAll::default(); + process_approvals_for_all(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.owner, vec![vec![0xaau8; 20]]); + assert_eq!(columns.operator, vec![vec![0xbbu8; 20]]); + } + + #[test] + fn malformed_and_foreign_logs_are_skipped_not_panicking() { + let missing_operator_topic = log_with(vec![APPROVAL_FOR_ALL_TOPIC0, B256::ZERO], word(1)); + let wrong_data_width = + log_with(vec![APPROVAL_FOR_ALL_TOPIC0, B256::ZERO, B256::ZERO], vec![0u8; 8]); + let foreign_topic0 = + log_with(vec![B256::repeat_byte(0xfe), B256::ZERO, B256::ZERO], word(1)); + + assert!(!is_approval_for_all(&missing_operator_topic)); + assert!(!is_approval_for_all(&wrong_data_width)); + assert!(!is_approval_for_all(&foreign_topic0)); + + let mut columns = ApprovalsForAll::default(); + process_approvals_for_all( + vec![missing_operator_topic, wrong_data_width, foreign_topic0], + &mut columns, + &schema(), + ) + .unwrap(); + assert_eq!(columns.n_rows, 0); + } +} diff --git a/crates/freeze/src/datasets/balances.rs b/crates/freeze/src/datasets/balances.rs index e2e3bb6..e160efd 100644 --- a/crates/freeze/src/datasets/balances.rs +++ b/crates/freeze/src/datasets/balances.rs @@ -1,6 +1,7 @@ use crate::*; -use alloy::primitives::{Address, U256}; +use alloy::primitives::{Address, B256, U256}; use polars::prelude::*; +use std::collections::HashMap; /// columns for balances #[triodion_macros::to_df(Datatype::Balances)] @@ -44,12 +45,91 @@ impl CollectByBlock for Balances { let schema = query.schemas.get(&Datatype::Balances).ok_or(err("schema not provided"))?; process_balance(columns, response, schema) } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.batch_state_reads { + state_override_collect_by_block::(partition, source, query, inner_request_size) + .await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } } impl CollectByTransaction for Balances { type Response = (); } +/// Read a whole block's worth of balances per request. +/// +/// `BALANCE` takes its account from the stack rather than from the execution +/// context, so unlike the storage extractor this one needs no cooperation from +/// the addresses being read: the loop runs at +/// [`SCRATCH_ADDRESS`](crate::SCRATCH_ADDRESS) and the calldata names the +/// accounts. Every row at a block therefore batches together, whether or not +/// the addresses are related, and no real account's code is overridden at all. +/// +/// # Why balances may be read this way and `codes` may not +/// +/// The batch path is a routing decision, so it has to be invisible in the +/// output: the opcode and the JSON-RPC method it stands in for must agree on +/// *every* account, not only on the ones that exist. `BALANCE` and +/// `eth_getBalance` both report zero for an account that has never been +/// touched, so a row answered by the extractor and the same row answered by +/// [`extract`](CollectByBlock::extract) after a demotion write the same cell. +/// +/// `EXTCODEHASH` has no such agreement — it returns zero for an account that +/// does not exist and `keccak256("")` for one that exists with no code, where +/// `eth_getCode` answers `0x` to both — so [`Codes`](crate::Codes) batches at +/// the transport instead. That asymmetry, not the shape of the loop, is what +/// decides which datasets belong here. +impl StateOverrideBatchable for Balances { + fn reader() -> StateReader { + StateReader::Balance + } + + fn target(_params: &Params) -> R
{ + // The account readers ignore the target — the calldata names the + // accounts and the code is injected at `SCRATCH_ADDRESS` — so a + // constant is what puts every row at a block into one group. + // + // The trap: `Address::ZERO` is inside the reserved precompile range + // that `StateReader::refuses_target` rejects, and it survives only + // because that check is applied to `StateReader::Storage` alone. + // Widening it to every reader would corrupt no row, but it would make + // every balances row ineligible and silently return the dataset to one + // request per row. + Ok(Address::ZERO) + } + + fn input_word(params: &Params) -> R { + // `BALANCE` masks its operand to the low 160 bits, so the calldata + // word is the address right-aligned — which is exactly what widening + // 20 big-endian bytes to `U256` gives. + // + // `ethers_address` is the width check, not decoration: `--address` is + // hex-decoded without a length check, and `U256::from_be_slice` PANICS + // above 32 bytes. This runs in the grouping loop, not in a worker task, + // so a panic here would take the whole freeze down instead of one row. + // As an error it makes the row ineligible, and the per-call path + // reports the bad argument exactly as it did before batching existed. + Ok(U256::from_be_slice(params.ethers_address()?.as_slice())) + } + + fn decode_row(params: &Params, value: B256) -> R { + // `BALANCE` pushes the balance as one full word, so re-widening those + // 32 big-endian bytes is the same `U256` `eth_getBalance` hands back on + // the per-row path. Both paths write identical cells, which is the + // property that lets a batch demote mid-partition. + Ok((params.block_number()? as u32, None, params.address()?, U256::from_be_bytes(value.0))) + } +} + fn process_balance(columns: &mut Balances, data: BlockTxAddressOutput, schema: &Table) -> R<()> { let (block, _tx, address, balance) = data; columns.n_rows += 1; diff --git a/crates/freeze/src/datasets/blocks.rs b/crates/freeze/src/datasets/blocks.rs index 916d50b..8d269c8 100644 --- a/crates/freeze/src/datasets/blocks.rs +++ b/crates/freeze/src/datasets/blocks.rs @@ -1,6 +1,7 @@ use crate::*; -use alloy::{primitives::U256, rpc::types::BlockTransactionsKind}; +use alloy::{eips::BlockNumberOrTag, primitives::U256, rpc::types::BlockTransactionsKind}; use polars::prelude::*; +use std::collections::HashMap; /// columns for transactions #[triodion_macros::to_df(Datatype::Blocks)] @@ -85,6 +86,19 @@ impl CollectByBlock for Blocks { let schema = query.schemas.get_schema(&Datatype::Blocks)?; process_block(response, columns, schema) } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.batch_rpc_calls { + rpc_batch_collect_by_block::(partition, source, query, inner_request_size).await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } } impl CollectByTransaction for Blocks { @@ -111,6 +125,42 @@ impl CollectByTransaction for Blocks { } } +/// A hundred headers per request instead of a hundred requests. +/// +/// Every row of this dataset is one block, and the per-row path spends one +/// `eth_getBlockByNumber` on each — so a year of mainnet is 2.6 million HTTP +/// requests to fetch 2.6 million headers that would fit in 26,000 of them. No +/// state override is involved and no aggregator contract exists for headers; +/// the whole win is in the envelope. +/// +/// `fullTransactions` stays `false`, matching [`CollectByBlock::extract`]: +/// this dataset stores the transaction *root*, never the bodies, so asking for +/// them would multiply the response size for columns nothing reads. +/// +/// A block the node does not have comes back as JSON `null`, which is why +/// [`Self::Item`] is an `Option`. That is a real answer to a real question, not +/// a transport failure, so it becomes the same "block not found" error the +/// per-row path raises rather than demoting the whole chunk to retry it. +impl RpcBatchable for Blocks { + type Param = (BlockNumberOrTag, bool); + type Item = Option; + + fn method() -> &'static str { + "eth_getBlockByNumber" + } + + fn param_for_row(params: &Params) -> R { + Ok((BlockNumberOrTag::Number(params.block_number()?), false)) + } + + fn decode_row(params: &Params, item: Self::Item) -> R { + item.ok_or_else(|| { + let n = params.block_number.map_or("?".to_string(), |n| n.to_string()); + CollectError::CollectError(format!("block not found: {n}")) + }) + } +} + /// process block into columns /// /// Takes an [`RpcBlock`] (alloy's `AnyRpcBlock`) rather than an diff --git a/crates/freeze/src/datasets/codes.rs b/crates/freeze/src/datasets/codes.rs index 04fdffd..a9e9aea 100644 --- a/crates/freeze/src/datasets/codes.rs +++ b/crates/freeze/src/datasets/codes.rs @@ -1,6 +1,10 @@ use crate::*; -use alloy::primitives::Address; +use alloy::{ + eips::BlockNumberOrTag, + primitives::{Address, Bytes}, +}; use polars::prelude::*; +use std::collections::HashMap; /// columns for codes /// @@ -69,12 +73,71 @@ impl CollectByBlock for Codes { let schema = query.schemas.get_schema(&Datatype::Codes)?; process_code(columns, response, schema) } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.batch_rpc_calls { + rpc_batch_collect_by_block::(partition, source, query, inner_request_size).await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } } impl CollectByTransaction for Codes { type Response = (); } +/// Whole bytecode batches at the transport, not through an extractor. +/// +/// An `EXTCODESIZE` / `EXTCODEHASH` extractor batches beautifully — one word +/// out per address in, exactly like the balance reader — but this dataset's +/// `code` column wants the bytes themselves, and returning those needs +/// `EXTCODECOPY` into memory with a length-prefixed layout. That costs quadratic +/// memory expansion and a hand-written variable-stride loop, to move payload +/// that is bandwidth-bound rather than round-trip-bound: the bytes have to cross +/// the wire either way, and a hundred contracts' code is a large response +/// however it is requested. +/// +/// JSON-RPC batching gets the request-count win — a hundred `eth_getCode` calls +/// in one HTTP body — for four lines and no new failure modes. That is the right +/// trade here. `rows_per_request` is lowered to 50 because these responses are +/// far larger than a 32-byte word. +impl RpcBatchable for Codes { + type Param = (Address, BlockNumberOrTag); + type Item = Bytes; + + fn method() -> &'static str { + "eth_getCode" + } + + fn param_for_row(params: &Params) -> R { + // `ethers_address`, never `Address::from_slice`, which PANICS on a width + // mismatch — and `--address` is hex-decoded with no length check, so + // `--address 0xdead` arrives here as two bytes. This runs once per row + // of a whole chunk, so a panic would abort fifty rows where the per-row + // path aborted one. As an error the chunk simply demotes, and the + // per-row path reports that address exactly as it did before batching + // existed. + Ok((params.ethers_address()?, BlockNumberOrTag::Number(params.block_number()?))) + } + + fn decode_row(params: &Params, item: Self::Item) -> R { + Ok((params.block_number()? as u32, None, params.address()?, item.to_vec())) + } + + fn default_rpc_batch_rows() -> usize { + // A contract can be 24 KB (EIP-170), so a hundred of them is a 2.4 MB + // response body. Fifty keeps the worst case inside the request-size + // limits providers actually enforce. + 50 + } +} + fn process_code(columns: &mut Codes, data: BlockTxAddressOutput, schema: &Table) -> R<()> { let (block, _tx, address, output) = data; let delegate = delegate_address(&output); diff --git a/crates/freeze/src/datasets/contract_interfaces.rs b/crates/freeze/src/datasets/contract_interfaces.rs new file mode 100644 index 0000000..c0def06 --- /dev/null +++ b/crates/freeze/src/datasets/contract_interfaces.rs @@ -0,0 +1,382 @@ +use crate::*; +use alloy::{ + primitives::{Bytes, FixedBytes, U256}, + sol_types::SolCall, +}; +use polars::prelude::*; +use std::collections::HashMap; + +/// columns for contract_interfaces +/// +/// One row per (block, address): what a contract answers to ERC-165 +/// `supportsInterface(bytes4)` for a fixed table of interface ids. It is the +/// classification layer the token datasets sit on — an address that answers +/// `true` for ERC-721 can be collected as an ERC-721, instead of being guessed +/// at from the shape of its logs. +/// +/// Every answer is `Option` and the three states are three different +/// facts: +/// +/// - `null` — the call reverted, the address has no code, or the return was not an ABI `bool`. The +/// contract said nothing, which is what a non-ERC-165 address looks like. +/// - `false` — the contract answered no. +/// - `true` — the contract answered yes. +/// +/// Filling `null` with `false` would turn "never answered" into "answered no", +/// and a count of `false` would then include every EOA in the input. +/// +/// `answers_true_to_everything` overrides the rest of the row — see the column. +#[triodion_macros::to_df(Datatype::ContractInterfaces)] +#[derive(Default)] +pub struct ContractInterfaces { + n_rows: u64, + block_number: Vec, + address: Vec>, + // The probe for the reserved id 0xffffffff. ERC-165 REQUIRES a compliant + // contract to answer `false` here, so `true` means this contract answers + // yes to every id ever asked: discard every other column in this row, none + // of them carry information. `null` is not an all-clear either — it means + // no decodable answer, so the contract made no ERC-165 promise about the + // rest of the row. + answers_true_to_everything: Vec>, + // ERC-165 requires this one to be `true`. A contract that implements + // `supportsInterface` but answers `false` here is not ERC-165 compliant, + // and its other answers are its own convention rather than the standard's. + supports_erc165: Vec>, + supports_erc721: Vec>, + supports_erc721_metadata: Vec>, + supports_erc721_enumerable: Vec>, + supports_erc1155: Vec>, + supports_erc1155_metadata_uri: Vec>, + supports_erc2981: Vec>, + supports_erc1271: Vec>, + chain_id: Vec, +} + +// An ERC-165 interface id is the XOR of the four-byte selectors of every +// function the interface declares, so each id below is derived rather than +// looked up. Each derivation was recomputed from keccak-256 of the signatures +// and each result matches the value published in its EIP. + +/// `supportsInterface(bytes4)` 0x01ffc9a7 — a one-function interface, so the id +/// is that selector on its own. +const ID_ERC165: FixedBytes<4> = FixedBytes::new([0x01, 0xff, 0xc9, 0xa7]); + +/// `balanceOf(address)` 0x70a08231 ^ `ownerOf(uint256)` 0x6352211e ^ +/// `safeTransferFrom(address,address,uint256)` 0x42842e0e ^ +/// `safeTransferFrom(address,address,uint256,bytes)` 0xb88d4fde ^ +/// `transferFrom(address,address,uint256)` 0x23b872dd ^ +/// `approve(address,uint256)` 0x095ea7b3 ^ `getApproved(uint256)` 0x081812fc ^ +/// `setApprovalForAll(address,bool)` 0xa22cb465 ^ +/// `isApprovedForAll(address,address)` 0xe985e9c5 = 0x80ac58cd. +const ID_ERC721: FixedBytes<4> = FixedBytes::new([0x80, 0xac, 0x58, 0xcd]); + +/// `name()` 0x06fdde03 ^ `symbol()` 0x95d89b41 ^ `tokenURI(uint256)` +/// 0xc87b56dd = 0x5b5e139f. +const ID_ERC721_METADATA: FixedBytes<4> = FixedBytes::new([0x5b, 0x5e, 0x13, 0x9f]); + +/// `totalSupply()` 0x18160ddd ^ `tokenOfOwnerByIndex(address,uint256)` +/// 0x2f745c59 ^ `tokenByIndex(uint256)` 0x4f6ccce7 = 0x780e9d63. +const ID_ERC721_ENUMERABLE: FixedBytes<4> = FixedBytes::new([0x78, 0x0e, 0x9d, 0x63]); + +/// `balanceOf(address,uint256)` 0x00fdd58e ^ +/// `balanceOfBatch(address[],uint256[])` 0x4e1273f4 ^ +/// `setApprovalForAll(address,bool)` 0xa22cb465 ^ +/// `isApprovedForAll(address,address)` 0xe985e9c5 ^ +/// `safeTransferFrom(address,address,uint256,uint256,bytes)` 0xf242432a ^ +/// `safeBatchTransferFrom(address,address,uint256[],uint256[],bytes)` +/// 0x2eb2c2d6 = 0xd9b67a26. +const ID_ERC1155: FixedBytes<4> = FixedBytes::new([0xd9, 0xb6, 0x7a, 0x26]); + +/// `uri(uint256)` 0x0e89341c — a one-function interface. +const ID_ERC1155_METADATA_URI: FixedBytes<4> = FixedBytes::new([0x0e, 0x89, 0x34, 0x1c]); + +/// `royaltyInfo(uint256,uint256)` 0x2a55205a — a one-function interface. +const ID_ERC2981: FixedBytes<4> = FixedBytes::new([0x2a, 0x55, 0x20, 0x5a]); + +/// `isValidSignature(bytes32,bytes)` 0x1626ba7e — a one-function interface. +/// +/// The superseded ERC-1271 draft took `(bytes,bytes)` and has a different id, +/// 0x20c13b0b. It is not probed, so a `null` here does not prove the contract +/// cannot verify a signature — only that it does not implement this version. +const ID_ERC1271: FixedBytes<4> = FixedBytes::new([0x16, 0x26, 0xba, 0x7e]); + +/// The id ERC-165 reserves as invalid and requires every compliant contract to +/// answer `false` for. See the `answers_true_to_everything` column. +const INVALID_INTERFACE_ID: FixedBytes<4> = FixedBytes::new([0xff, 0xff, 0xff, 0xff]); + +/// One `eth_call` — or one Multicall3 inner call — per probe, per row. +const N_PROBES: usize = 9; + +/// Probe order. It is load-bearing: `extract`, `decode_row` and `transform` +/// all read the answers positionally, so reordering this array relabels every +/// column without changing a column name. +/// +/// The invalid id goes first so that the column that invalidates the row is +/// the first one a reader meets. +const PROBES: [FixedBytes<4>; N_PROBES] = [ + INVALID_INTERFACE_ID, + ID_ERC165, + ID_ERC721, + ID_ERC721_METADATA, + ID_ERC721_ENUMERABLE, + ID_ERC1155, + ID_ERC1155_METADATA_URI, + ID_ERC2981, + ID_ERC1271, +]; + +/// (block_number, address, one answer per entry of [`PROBES`], in that order) +type InterfaceAnswers = (u32, Vec, [Option; N_PROBES]); + +impl Dataset for ContractInterfaces { + fn aliases() -> Vec<&'static str> { + vec!["erc165"] + } + + fn default_columns() -> Option> { + Some(vec![ + "block_number", + "address", + "answers_true_to_everything", + "supports_erc165", + "supports_erc721", + "supports_erc721_metadata", + "supports_erc721_enumerable", + "supports_erc1155", + "supports_erc1155_metadata_uri", + "supports_erc2981", + "supports_erc1271", + "chain_id", + ]) + } + + fn default_sort() -> Option> { + Some(vec!["block_number", "address"]) + } + + fn default_blocks() -> Option { + Some("latest".to_string()) + } + + fn required_parameters() -> Vec { + vec![Dim::Address] + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +/// Gas ceiling for one `supportsInterface` probe. +/// +/// ERC-165 requires the function to use at most 30,000 gas, so a conforming +/// contract never reaches this and a cap costs it nothing. +/// +/// The cap is not for conforming contracts. This dataset points `eth_call` at +/// arbitrary addresses, most of which do not implement ERC-165 at all, so each +/// probe runs whatever fallback function the address happens to have — under +/// the node's own default limit, which geth sets at 50,000,000 gas. A fallback +/// that loops until the gas runs out then takes the node seconds per probe, +/// nine probes per row, and the chunk fails on a timeout instead of writing +/// one row of nulls. With the cap, the same contract produces an out-of-gas +/// revert that `contract_read` folds into `None` for that address alone. +const PROBE_GAS_LIMIT: u64 = 30_000; + +impl CollectByBlock for ContractInterfaces { + type Response = InterfaceAnswers; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let block_number = request.ethers_block_number()?; + let address = request.ethers_address()?; + + // Each probe folds a *contract-level* refusal (a revert, or an address + // with no code) into `None`, while a *node-level* failure — pruned + // state on a non-archive endpoint, a rate limit, a timeout — + // propagates via `?`. Without that split, pointing this dataset at a + // non-archive RPC classifies every contract as "not ERC-165" under a + // "chunks errored: 0" banner. + let mut answers = [None; N_PROBES]; + for (answer, interface_id) in answers.iter_mut().zip(PROBES) { + let call_data = probe_calldata(interface_id); + let output = contract_read( + source + .call_with_gas_limit(address, call_data, block_number, Some(PROBE_GAS_LIMIT)) + .await, + )?; + *answer = output.and_then(|bytes| decode_bool_word(&bytes)); + } + + Ok((request.block_number()? as u32, request.address()?, answers)) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::ContractInterfaces)?; + let (block, address, answers) = response; + // Positional unpack, in `PROBES` order. + let [answers_true_to_everything, supports_erc165, supports_erc721, supports_erc721_metadata, supports_erc721_enumerable, supports_erc1155, supports_erc1155_metadata_uri, supports_erc2981, supports_erc1271] = + answers; + columns.n_rows += 1; + store!(schema, columns, block_number, block); + store!(schema, columns, address, address); + store!(schema, columns, answers_true_to_everything, answers_true_to_everything); + store!(schema, columns, supports_erc165, supports_erc165); + store!(schema, columns, supports_erc721, supports_erc721); + store!(schema, columns, supports_erc721_metadata, supports_erc721_metadata); + store!(schema, columns, supports_erc721_enumerable, supports_erc721_enumerable); + store!(schema, columns, supports_erc1155, supports_erc1155); + store!(schema, columns, supports_erc1155_metadata_uri, supports_erc1155_metadata_uri); + store!(schema, columns, supports_erc2981, supports_erc2981); + store!(schema, columns, supports_erc1271, supports_erc1271); + Ok(()) + } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.multicall { + multicall_collect_by_block::(partition, source, query, inner_request_size).await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } +} + +// The interfaces a contract declares are a property of its code at a block, not +// of any transaction. `-t` has no relationship to resolve here: it could only +// mean "the interfaces of some address this transaction happened to touch", +// which is a different question from the one this dataset answers. +impl CollectByTransaction for ContractInterfaces { + type Response = (); +} + +impl MulticallBatchable for ContractInterfaces { + fn calls_for_row(params: &Params, require_success: bool) -> R> { + let target = params.ethers_address()?; + let allow_failure = !require_success; + Ok(PROBES + .into_iter() + .map(|interface_id| Multicall3::Call3 { + target, + allowFailure: allow_failure, + callData: Bytes::from(probe_calldata(interface_id)), + }) + .collect()) + } + + fn default_multicall_batch_size() -> u32 { + // 90 inner calls, i.e. 10 rows of 9 probes, against the crate default + // of 250 (27 rows). + // + // `aggregate3` hands each inner call all the gas remaining at that + // point, and this dataset aims its probes at arbitrary addresses whose + // fallback functions are arbitrary code. One address whose fallback + // burns the whole `eth_call` budget starves every later call in the + // same batch, and a starved inner call returns `success: false` with + // empty `returnData` — byte-identical to a revert. `aggregate3` still + // returns normally, so those rows are published as nulls with nothing + // reported. + // + // A smaller batch does not prevent that; nothing available in + // `aggregate3` can, because `Call3` has no per-call gas field. It + // bounds the damage to nine other rows instead of twenty-six. The + // per-call path caps each probe at `PROBE_GAS_LIMIT` and does not have + // the problem, so `--no-multicall` is the way to rule it out entirely. + 90 + } + + fn decode_row(params: &Params, results: &[Multicall3::Result]) -> R { + // `calls_for_row` emits exactly `N_PROBES` calls. A shorter slice means + // the node returned a malformed aggregate3; zipping it would leave the + // trailing columns null and silently publish a partial row. + if results.len() != N_PROBES { + return Err(err("multicall returned the wrong number of results for row")) + } + let mut answers = [None; N_PROBES]; + for (answer, result) in answers.iter_mut().zip(results.iter()) { + // A failed inner call is the contract declining to answer, exactly + // as a revert is on the per-call path. + *answer = if result.success { decode_bool_word(&result.returnData) } else { None }; + } + Ok((params.block_number()? as u32, params.address()?, answers)) + } +} + +/// Calldata for one `supportsInterface(interface_id)` probe. +/// +/// Encoded through the ABI rather than by hand: `bytes4` is left-aligned in its +/// word, and hand-padding it on the wrong side asks about a different id. +fn probe_calldata(interface_id: FixedBytes<4>) -> Vec { + ERC165::supportsInterfaceCall { interfaceId: interface_id }.abi_encode() +} + +/// Decode an `eth_call` return that should be a single ABI `bool`. +/// +/// Solidity's own decoder accepts only the two canonical words and reverts on +/// anything else, so an on-chain caller of `supportsInterface` never sees a +/// third value. Mirroring that is the only reading that invents nothing: an +/// empty return (what an address with no code answers), a short or long +/// return, and a dirty word all mean the contract did not answer — `None`, not +/// `false`. It also means a contract that returns garbage for every id lands +/// with `answers_true_to_everything` null rather than false, so nothing in the +/// row claims a standard. +fn decode_bool_word(data: &[u8]) -> Option { + // Exact-width check lives in `decode_u256_word`: a 1-byte or 33-byte return + // is not a `bool` answer. + let word = decode_u256_word(data)?; + if word == U256::ZERO { + Some(false) + } else if word == U256::from(1u8) { + Some(true) + } else { + None + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn only_the_two_canonical_words_are_an_answer() { + let mut yes = [0u8; 32]; + yes[31] = 1; + assert_eq!(decode_bool_word(&yes), Some(true)); + assert_eq!(decode_bool_word(&[0u8; 32]), Some(false)); + + // An address with no code answers with an empty return. Reading that + // as `false` would claim the contract said no. + assert_eq!(decode_bool_word(&[]), None); + + // Dirty words: Solidity's decoder reverts on both, so neither is an + // answer a caller could have acted on. + let mut two = [0u8; 32]; + two[31] = 2; + assert_eq!(decode_bool_word(&two), None); + let mut high_bit_set = [0u8; 32]; + high_bit_set[0] = 1; + high_bit_set[31] = 1; + assert_eq!(decode_bool_word(&high_bit_set), None); + } + + #[test] + fn every_probe_is_distinct() { + let mut ids = PROBES.to_vec(); + ids.sort(); + ids.dedup(); + // A duplicated id would label two columns from one question, and the + // duplicate column would look like independent evidence. + assert_eq!(ids.len(), N_PROBES); + } + + #[test] + fn the_invalid_id_is_the_reserved_one() { + // The whole point of the sentinel column: ask for the id ERC-165 + // reserves, not for a real interface. + assert_eq!(INVALID_INTERFACE_ID, FixedBytes::new([0xff; 4])); + assert_eq!(PROBES.first(), Some(&INVALID_INTERFACE_ID)); + } +} diff --git a/crates/freeze/src/datasets/contracts.rs b/crates/freeze/src/datasets/contracts.rs index fbdef5f..5b1b6a9 100644 --- a/crates/freeze/src/datasets/contracts.rs +++ b/crates/freeze/src/datasets/contracts.rs @@ -95,11 +95,81 @@ pub(crate) fn process_contracts( store!(schema, columns, factory, create.from.to_vec()); store!(schema, columns, init_code, create.init.to_vec()); store!(schema, columns, code, result.code.to_vec()); - store!(schema, columns, init_code_hash, keccak256(result.code.clone()).to_vec()); - store!(schema, columns, code_hash, keccak256(create.init.clone()).to_vec()); + // Each column held the *other* column's hash. `code_hash` is the + // usual join key for contract identity, so every match against an + // `EXTCODEHASH` or an external codehash index silently found + // nothing rather than erroring. + store!(schema, columns, init_code_hash, keccak256(create.init.clone()).to_vec()); + store!(schema, columns, code_hash, keccak256(result.code.clone()).to_vec()); store!(schema, columns, n_init_code_bytes, create.init.len() as u32); store!(schema, columns, n_code_bytes, result.code.len() as u32); } } Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashMap; + + /// A `create` whose init code and deployed code differ, so a transposition + /// of the two hash columns cannot pass unnoticed. + fn create_trace() -> LocalizedTransactionTrace { + serde_json::from_value(serde_json::json!({ + "action": { + "from": "0x1111111111111111111111111111111111111111", + "gas": "0x100000", + "init": "0xdeadbeef", + "value": "0x0" + }, + "blockHash": + "0x2222222222222222222222222222222222222222222222222222222222222222", + "blockNumber": 1, + "result": { + "address": "0x3333333333333333333333333333333333333333", + "code": "0xc0ffee", + "gasUsed": "0x1000" + }, + "subtraces": 0, + "traceAddress": [], + "transactionHash": + "0x4444444444444444444444444444444444444444444444444444444444444444", + "transactionPosition": 0, + "type": "create" + })) + .expect("create trace fixture deserializes") + } + + fn schemas() -> Schemas { + let columns: Vec = + Datatype::Contracts.column_types().keys().map(|name| name.to_string()).collect(); + let schema = Datatype::Contracts + .table_schema( + &[U256Type::String], + &ColumnEncoding::Hex, + &None, + &None, + &Some(columns), + None, + None, + ) + .expect("every column is nameable"); + HashMap::from([(Datatype::Contracts, schema)]) + } + + #[test] + fn each_hash_column_hashes_its_own_column() { + let mut columns = Contracts::default(); + process_contracts(&[create_trace()], &mut columns, &schemas()) + .expect("one create trace yields one row"); + + assert_eq!(columns.init_code, vec![vec![0xde, 0xad, 0xbe, 0xef]]); + assert_eq!(columns.code, vec![vec![0xc0, 0xff, 0xee]]); + // These two were transposed: `init_code_hash` held keccak(code) and + // `code_hash` held keccak(init_code), so `code_hash` matched no + // `EXTCODEHASH` anywhere. + assert_eq!(columns.init_code_hash, vec![keccak256([0xde, 0xad, 0xbe, 0xef]).to_vec()]); + assert_eq!(columns.code_hash, vec![keccak256([0xc0, 0xff, 0xee]).to_vec()]); + } +} diff --git a/crates/freeze/src/datasets/erc1155_metadata.rs b/crates/freeze/src/datasets/erc1155_metadata.rs new file mode 100644 index 0000000..70e97c2 --- /dev/null +++ b/crates/freeze/src/datasets/erc1155_metadata.rs @@ -0,0 +1,331 @@ +use crate::{ + types::rpc_params::{fixed_from_slice, log_address_matches, topic_matches}, + *, +}; +use alloy::{ + primitives::{B256, U256}, + rpc::types::{Filter, Log, Topic}, + sol_types::SolEvent, +}; +use polars::prelude::*; + +/// The ERC-1155 metadata substitution token. A conforming client replaces this +/// exact lowercase literal with the token id as 64 lowercase hex digits and no +/// `0x` prefix. Matching case-insensitively would flag `{ID}` strings that no +/// conforming client expands, so the match stays literal. +const ID_PLACEHOLDER: &str = "{id}"; + +/// columns for erc1155_metadata +/// +/// One row per ERC-1155 `URI(string value, uint256 indexed id)` log. +/// +/// An empty result is the normal case here, not a failure. The event is +/// optional in the standard, and most contracts never emit it: they serve one +/// static `{id}` template from `uri(id)` and never announce a change. A +/// full-range scan of a busy chain can legitimately return zero rows. +/// +/// This reads logs instead of calling `uri(id)`. A call needs a token id to +/// call with, and triodion partitions by block, address and topic — there is no +/// token-id dimension to enumerate ids from, so the call form has no input. +#[triodion_macros::to_df(Datatype::Erc1155Metadata)] +#[derive(Default)] +pub struct Erc1155Metadata { + n_rows: u64, + block_number: Vec, + block_hash: Vec>>, + transaction_index: Vec, + log_index: Vec, + transaction_hash: Vec>, + erc1155: Vec>, + // Indexed, so it arrives in topics[1] rather than in the log data. + token_id: Vec, + // The event's `value` argument. It is not indexed, so it sits in the log + // data as an ABI-encoded dynamic string and has to be decoded, not sliced. + uri: Vec, + // Derived, not reported: `uri` contains the literal `{id}`. Such a string + // is a template, not a URL, and fetching it verbatim fails. Expand it with + // this row's `token_id` first. It also means the string is shared by every + // token of the contract, so it does not identify this token. + is_uri_template: Vec, + // True when the on-chain bytes were not valid UTF-8 and `uri` therefore + // holds U+FFFD replacement characters that were never on chain. Without + // this the mangled string is indistinguishable from a faithful one. + is_uri_lossy: Vec, + chain_id: Vec, +} + +impl Dataset for Erc1155Metadata { + fn aliases() -> Vec<&'static str> { + vec!["erc1155_uris"] + } + + fn default_columns() -> Option> { + Some(vec![ + "block_number", + // "block_hash", + "transaction_index", + "log_index", + "transaction_hash", + "erc1155", + "token_id", + "uri", + "is_uri_template", + "is_uri_lossy", + "chain_id", + ]) + } + + fn optional_parameters() -> Vec { + // Topic0 is pinned to the URI signature. Topic1 is the indexed token + // id, so `--topic1` narrows to one token. The event has one indexed + // argument, so topic2 and topic3 do not exist on it. + vec![Dim::Address, Dim::Topic1] + } + + fn use_block_ranges() -> bool { + true + } + + fn default_inner_request_size() -> u64 { + // Same as the other log datasets. A wider window would still return + // few rows, since URI logs are rare, but the dispatcher takes + // max(user, default): raising this would stop a user shrinking the + // window for a provider that caps the block span of eth_getLogs. + 50 + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for Erc1155Metadata { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let mut topics: [Topic; 4] = Default::default(); + topics[0] = ERC1155::URI::SIGNATURE_HASH.into(); + if let Some(token_id) = &request.topic1 { + // `--topic1` is hex-decoded without a width check, so a short value + // must become an error here rather than panic in `B256::from_slice`. + topics[1] = fixed_from_slice::(token_id, "topic1")?.into(); + } + let filter = Filter { topics, ..request.ethers_log_filter()? }; + let logs = source.get_logs(&filter).await?; + Ok(logs.into_iter().filter(is_erc1155_uri).collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc1155Metadata)?; + process_erc1155_uris(response, columns, schema) + } +} + +impl CollectByTransaction for Erc1155Metadata { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let logs = source.get_transaction_logs(request.transaction_hash()?).await?; + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_erc1155_uri(log) && + log_address_matches(log, &request.address) && + topic_matches(log, 1, &request.topic1) + }) + .collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc1155Metadata)?; + process_erc1155_uris(response, columns, schema) + } +} + +/// True iff `log` has the ERC-1155 `URI` shape: the URI signature and exactly 2 +/// topics (sig + indexed id). The data width is not checked here — the string +/// body is dynamic, so only the ABI decode can judge it. +/// +/// The per-transaction path needs the signature check because it receives every +/// log of the transaction, unfiltered. +pub(crate) fn is_erc1155_uri(log: &Log) -> bool { + log.topics().len() == 2 && + log.topics().first().is_some_and(|t| *t == ERC1155::URI::SIGNATURE_HASH) +} + +/// process logs into columns +fn process_erc1155_uris(logs: Vec, columns: &mut Erc1155Metadata, schema: &Table) -> R<()> { + for log in logs.iter() { + // Shape guard first: a log carrying the URI topic0 with fewer than 2 + // topics would otherwise reach the decode below with a missing id. + // Every current caller pre-filters; this keeps a future caller that + // arrives with mixed logs safe. + if !is_erc1155_uri(log) { + continue; + } + if let (Some(bn), Some(tx), Some(ti), Some(li)) = + (log.block_number, log.transaction_hash, log.transaction_index, log.log_index) + { + // Reads topics[1] for the indexed `id` and ABI-decodes the data for + // the non-indexed `value`. `decode_log_data` detokenises a `string` + // with `String::from_utf8_lossy`, so non-UTF-8 bytes would silently + // become U+FFFD and be written as if measured. Validate first, and + // fall back to the lossy read only to keep the row: dropping it + // would lose the fact that the URI changed, which is the signal + // this dataset exists for. A truncated body or a bad offset still + // fails both decoders and skips the row. + let (event, is_lossy) = match ERC1155::URI::decode_log_data_validate(log.data()) { + Ok(event) => (event, false), + Err(_) => match ERC1155::URI::decode_log_data(log.data()) { + Ok(event) => (event, true), + Err(_) => continue, + }, + }; + let is_template = event.value.contains(ID_PLACEHOLDER); + + columns.n_rows += 1; + store!(schema, columns, block_number, bn as u32); + store!(schema, columns, block_hash, log.block_hash.map(|bh| bh.to_vec())); + store!(schema, columns, transaction_index, ti as u32); + store!(schema, columns, log_index, li as u32); + store!(schema, columns, transaction_hash, tx.to_vec()); + store!(schema, columns, erc1155, log.address().to_vec()); + store!(schema, columns, token_id, event.id); + store!(schema, columns, uri, event.value); + store!(schema, columns, is_uri_template, is_template); + store!(schema, columns, is_uri_lossy, is_lossy); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy::primitives::Address; + + /// real `Erc1155Metadata` schema, so `store!` actually evaluates each column + fn schema() -> Table { + Datatype::Erc1155Metadata + .table_schema( + &[U256Type::String], + &ColumnEncoding::Hex, + &None, + &None, + &None, + None, + None, + ) + .expect("Erc1155Metadata has a valid default schema") + } + + /// rpc `Log` carrying the block/tx/index fields `process_erc1155_uris` needs + fn log_from(data: alloy::primitives::LogData) -> Log { + Log { + inner: alloy::primitives::Log { address: Address::ZERO, data }, + block_number: Some(1), + transaction_hash: Some(B256::ZERO), + transaction_index: Some(0), + log_index: Some(0), + ..Default::default() + } + } + + /// a well-formed URI log, encoded the way a contract would emit it + fn uri_log(value: &str, id: u64) -> Log { + log_from(ERC1155::URI { value: value.to_string(), id: U256::from(id) }.encode_log_data()) + } + + #[test] + fn decodes_the_indexed_id_and_the_non_indexed_string() { + // The point of the dataset: `id` is a topic, `value` is in the data. + let log = uri_log("ipfs://QmExample/7.json", 7); + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.token_id, vec![U256::from(7u64)]); + assert_eq!(columns.uri, vec!["ipfs://QmExample/7.json".to_string()]); + assert_eq!(columns.is_uri_template, vec![false]); + } + + #[test] + fn flags_a_template_uri() { + let log = uri_log("https://example.com/api/{id}.json", 1); + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.is_uri_template, vec![true]); + } + + #[test] + fn does_not_flag_a_non_conforming_placeholder() { + // `{ID}` is not the spec's token, so no client expands it. Flagging it + // would claim a substitution that never happens. + let log = uri_log("https://example.com/api/{ID}.json", 1); + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.is_uri_template, vec![false]); + } + + #[test] + fn keeps_an_empty_uri_as_a_value() { + // A contract can emit `URI("", id)` to clear a token's metadata. That + // is a measurement, not an absence, so the row exists and holds "". + let log = uri_log("", 3); + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.uri, vec![String::new()]); + } + + #[test] + fn uri_topic0_with_missing_topic1_is_skipped_not_panicking() { + let mut data = ERC1155::URI { value: "x".to_string(), id: U256::ZERO }.encode_log_data(); + data.set_topics_unchecked(vec![ERC1155::URI::SIGNATURE_HASH]); + let malformed = log_from(data); + assert!(!is_erc1155_uri(&malformed)); + + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(vec![malformed], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0, "malformed log must not produce a row"); + } + + #[test] + fn undecodable_data_is_skipped() { + // Right topics, but the body is not an ABI-encoded string. + let data = alloy::primitives::LogData::new_unchecked( + vec![ERC1155::URI::SIGNATURE_HASH, B256::ZERO], + alloy::primitives::Bytes::from(vec![0xffu8; 16]), + ); + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(vec![log_from(data)], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn skips_a_log_with_a_foreign_topic0() { + let data = alloy::primitives::LogData::new_unchecked( + vec![B256::repeat_byte(0xfe), B256::ZERO], + alloy::primitives::Bytes::new(), + ); + let logs = vec![log_from(data), uri_log("ipfs://a", 1)]; + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(logs, &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 1); + } + + #[test] + fn skips_logs_missing_identity_fields() { + let mut log = uri_log("ipfs://a", 1); + log.log_index = None; + let mut columns = Erc1155Metadata::default(); + process_erc1155_uris(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } +} diff --git a/crates/freeze/src/datasets/erc1155_transfers.rs b/crates/freeze/src/datasets/erc1155_transfers.rs new file mode 100644 index 0000000..48a65c1 --- /dev/null +++ b/crates/freeze/src/datasets/erc1155_transfers.rs @@ -0,0 +1,493 @@ +use crate::{ + types::rpc_params::{ + address_topic_matches, fixed_from_slice, log_address_matches, topic_matches, + }, + *, +}; +use alloy::{ + primitives::{Address, B256, U256}, + rpc::types::{Filter, Log, Topic}, + sol_types::SolEvent, +}; +use polars::prelude::*; + +/// columns for erc1155_transfers +/// +/// One row per token id moved. Covers `TransferSingle` and `TransferBatch` in +/// one table, because both describe the same event: an amount of one token id +/// moving from one party to another. +/// +/// `TransferBatch` carries two parallel arrays, `ids` and `values`. Storing +/// them whole, or storing their length, would put an aggregate in a cell that +/// cannot be taken apart again — which token moved, and how much of it, is +/// gone. A batch of five ids therefore becomes five rows, and a +/// `TransferSingle` becomes one row of exactly that shape. +/// `(log_index, token_id_index)` is the unique key within a block. +/// +/// Both events carry a signature plus three indexed arguments, so the topic +/// count cannot tell them apart. topic0 is the only discriminator. +/// +/// ERC-1155 indexes `operator` first, so `from` and `to` sit one topic slot +/// later than in ERC-20 and ERC-721 — see the filter in `extract`. +#[triodion_macros::to_df(Datatype::Erc1155Transfers)] +#[derive(Default)] +pub struct Erc1155Transfers { + n_rows: u64, + block_number: Vec, + block_hash: Vec>>, + transaction_index: Vec, + log_index: Vec, + // Position of this token id within its own log's `ids` list, and 0 for a + // `TransferSingle`. Batch order is emitted order and carries meaning, so + // it is kept rather than reconstructed. Not `transfer_index`: that name + // already counts within a block in `native_transfers`, and one word must + // keep one meaning across datasets. + token_id_index: Vec, + transaction_hash: Vec>, + erc1155: Vec>, + // The `msg.sender` that performed the transfer, which need not be `from`: + // an approved operator moves someone else's tokens under their own name. + operator: Vec>, + from_address: Vec>, + to_address: Vec>, + token_id: Vec, + value: Vec, + // True when the row came from a `TransferBatch`. ERC-1155 is a final + // standard with exactly these two transfer events, so a flag states the + // whole fact; every other column on the row reads the same either way. + is_batch: Vec, + // Derived: `from_address` is the zero address. ERC-1155 encodes a mint + // that way, so the zero address is not an account that sent anything. + // Without this column `GROUP BY from_address` reports it as the busiest + // sender on the chain, and a supply figure counts it as a holder. + is_mint: Vec, + // Derived: `to_address` is the zero address, which encodes a burn. Kept + // apart from `is_mint` because the two are independent facts about a row + // and a single enum column would have to discard one of them. + is_burn: Vec, + chain_id: Vec, +} + +impl Dataset for Erc1155Transfers { + fn default_columns() -> Option> { + Some(vec![ + "block_number", + // "block_hash", + "transaction_index", + "log_index", + "token_id_index", + "transaction_hash", + "erc1155", + "operator", + "from_address", + "to_address", + "token_id", + "value", + "is_batch", + "is_mint", + "is_burn", + "chain_id", + ]) + } + + fn default_sort() -> Option> { + // The inherited default stops at `log_index`, which leaves every row of + // a batch tied. Polars does not promise to keep tied rows in input + // order, so emitted batch order would be lost on write. Sorting by the + // ordinal too makes the order total and reproducible. + Some(vec!["block_number", "log_index", "token_id_index"]) + } + + fn optional_parameters() -> Vec { + // Topic0 is fixed to the two transfer signatures. Topic1 is the + // indexed `operator`; `--from-address` and `--to-address` reach + // topic2 and topic3, so they are named rather than exposed as raw + // topic dimensions. + vec![Dim::Address, Dim::Topic1, Dim::FromAddress, Dim::ToAddress] + } + + fn use_block_ranges() -> bool { + true + } + + fn default_inner_request_size() -> u64 { + // Same as the other log-shaped datasets: one `eth_getLogs` per 50 + // blocks rather than one per block. Override with --inner-request-size. + 50 + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for Erc1155Transfers { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let mut topics: [Topic; 4] = Default::default(); + // A `Topic` is a set of values and `eth_getLogs` reads a set at one + // position as OR, so both signatures come back from one request. + topics[0] = + vec![ERC1155::TransferSingle::SIGNATURE_HASH, ERC1155::TransferBatch::SIGNATURE_HASH] + .into(); + // ERC-1155 indexes operator, from, to — in that order. `from` is + // topic2 and `to` is topic3, not topic1 and topic2 as in ERC-20 and + // ERC-721. Copying that layout filters on the operator instead and + // returns an empty file with no error at all. + if let Some(operator) = &request.topic1 { + topics[1] = fixed_from_slice::(operator, "topic1")?.into(); + } + if let Some(from_address) = &request.from_address { + // `--from-address` is documented as a 20-byte address; left-pad it + // into the 32-byte topic slot rather than panicking on a short one. + let v = address_dim_as_topic(from_address).ok_or_else(|| { + CollectError::CollectError("from_address must be at most 32 bytes".to_string()) + })?; + topics[2] = v.into(); + } + if let Some(to_address) = &request.to_address { + let v = address_dim_as_topic(to_address).ok_or_else(|| { + CollectError::CollectError("to_address must be at most 32 bytes".to_string()) + })?; + topics[3] = v.into(); + } + let filter = Filter { topics, ..request.ethers_log_filter()? }; + let logs = source.get_logs(&filter).await?; + Ok(logs.into_iter().filter(is_erc1155_transfer).collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc1155Transfers)?; + process_erc1155_transfers(response, columns, schema) + } +} + +impl CollectByTransaction for Erc1155Transfers { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let logs = source.get_transaction_logs(request.transaction_hash()?).await?; + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_erc1155_transfer(log) && + log_address_matches(log, &request.address) && + topic_matches(log, 1, &request.topic1) && + address_topic_matches(log, 2, &request.from_address) && + address_topic_matches(log, 3, &request.to_address) + }) + .collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc1155Transfers)?; + process_erc1155_transfers(response, columns, schema) + } +} + +/// True iff `log` carries one of the two ERC-1155 transfer signatures and the +/// topic count both events share: signature plus indexed operator, from, to. +/// +/// Data width is deliberately not checked. A `TransferSingle` body is two +/// static words, but a `TransferBatch` body is two dynamic arrays behind +/// offsets and has no fixed length, so the ABI decoder in +/// [`process_erc1155_transfers`] is what finally validates a body. Written to +/// stay reusable by a coalesced [`crate::LogEvents`] fan-out, which this +/// datatype is NOT a member of today (see `MultiDatatype::LogEvents`). +pub(crate) fn is_erc1155_transfer(log: &Log) -> bool { + log.topics().len() == 4 && + log.topics().first().is_some_and(|t| { + *t == ERC1155::TransferSingle::SIGNATURE_HASH || + *t == ERC1155::TransferBatch::SIGNATURE_HASH + }) +} + +/// One decoded transfer log, flattened to the parties plus the (id, value) +/// pairs it moved. A `TransferSingle` yields one pair, a `TransferBatch` as +/// many as it listed. +struct DecodedTransfer { + operator: Address, + from: Address, + to: Address, + is_batch: bool, + /// `(token_id, value)`, in the order the contract emitted them. + items: Vec<(U256, U256)>, +} + +/// Decode either transfer event, or `None` for anything this dataset cannot +/// represent. Every rejection here becomes a skipped log, never a panic. +fn decode_transfer(log: &Log) -> Option { + // Both events have 4 topics, so only topic0 separates them. + let topic0 = *log.topics().first()?; + + if topic0 == ERC1155::TransferSingle::SIGNATURE_HASH { + // `decode_log_data` checks the topic count and the signature before it + // touches the body. The single body is two static words and could be + // sliced by hand, but it goes through the same path as the batch so + // the two cannot drift apart. + let event = ERC1155::TransferSingle::decode_log_data(log.data()).ok()?; + Some(DecodedTransfer { + operator: event.operator, + from: event.from, + to: event.to, + is_batch: false, + items: vec![(event.id, event.value)], + }) + } else if topic0 == ERC1155::TransferBatch::SIGNATURE_HASH { + // The body is two dynamically sized arrays reached through offsets. + // Let alloy walk it: a hand-written slice of an offset-encoded body is + // where the bugs live. + let event = ERC1155::TransferBatch::decode_log_data(log.data()).ok()?; + // The two arrays are parallel by definition. Unequal lengths mean the + // emitter is not compliant and no pairing can be recovered: padding + // either side would invent a transfer that never happened, and + // truncating would drop one that did. Skip the whole log. + if event.ids.len() != event.values.len() { + return None + } + Some(DecodedTransfer { + operator: event.operator, + from: event.from, + to: event.to, + is_batch: true, + // An empty batch is well formed and moved nothing, so it yields no + // rows. That is an absence, not a zero-value transfer. + items: event.ids.into_iter().zip(event.values).collect(), + }) + } else { + None + } +} + +/// Explode each transfer log into one row per token id. +fn process_erc1155_transfers( + logs: Vec, + columns: &mut Erc1155Transfers, + schema: &Table, +) -> R<()> { + for log in logs.iter() { + let (Some(bn), Some(tx), Some(ti), Some(li)) = + (log.block_number, log.transaction_hash, log.transaction_index, log.log_index) + else { + continue + }; + // A foreign topic0, a truncated body, or a batch whose two arrays + // disagree all land here and produce no rows. Callers that reach this + // function with mixed logs — a future coalesced [`crate::LogEvents`] + // extractor, or a future one — therefore stay safe without a + // pre-filter of their own. + let Some(transfer) = decode_transfer(log) else { continue }; + let DecodedTransfer { operator, from, to, is_batch, items } = transfer; + + let block_hash = log.block_hash.map(|bh| bh.to_vec()); + let erc1155 = log.address().to_vec(); + for (index, (token_id, value)) in items.into_iter().enumerate() { + columns.n_rows += 1; + store!(schema, columns, block_number, bn as u32); + store!(schema, columns, block_hash, block_hash.clone()); + store!(schema, columns, transaction_index, ti as u32); + store!(schema, columns, log_index, li as u32); + store!(schema, columns, token_id_index, index as u32); + store!(schema, columns, transaction_hash, tx.to_vec()); + store!(schema, columns, erc1155, erc1155.clone()); + store!(schema, columns, operator, operator.to_vec()); + store!(schema, columns, from_address, from.to_vec()); + store!(schema, columns, to_address, to.to_vec()); + store!(schema, columns, token_id, token_id); + store!(schema, columns, value, value); + store!(schema, columns, is_batch, is_batch); + store!(schema, columns, is_mint, from == Address::ZERO); + store!(schema, columns, is_burn, to == Address::ZERO); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy::primitives::LogData; + + /// real `Erc1155Transfers` schema, so `store!` actually evaluates each column + fn schema() -> Table { + Datatype::Erc1155Transfers + .table_schema( + &[U256Type::String], + &ColumnEncoding::Hex, + &None, + &None, + &None, + None, + None, + ) + .expect("Erc1155Transfers has a valid default schema") + } + + /// rpc `Log` carrying the block/tx/index fields `process_erc1155_transfers` requires + fn rpc_log(data: LogData) -> Log { + Log { + inner: alloy::primitives::Log { address: Address::repeat_byte(0x11), data }, + block_number: Some(1), + transaction_hash: Some(B256::ZERO), + transaction_index: Some(0), + log_index: Some(7), + ..Default::default() + } + } + + fn addr(byte: u8) -> Address { + Address::repeat_byte(byte) + } + + #[test] + fn batch_explodes_into_one_row_per_token_id() { + let event = ERC1155::TransferBatch { + operator: addr(0xaa), + from: addr(0xbb), + to: addr(0xcc), + ids: vec![U256::from(1), U256::from(2), U256::from(3)], + values: vec![U256::from(10), U256::from(20), U256::from(30)], + }; + let mut columns = Erc1155Transfers::default(); + process_erc1155_transfers(vec![rpc_log(event.encode_log_data())], &mut columns, &schema()) + .unwrap(); + + assert_eq!(columns.n_rows, 3); + assert_eq!(columns.token_id_index, vec![0, 1, 2]); + assert_eq!(columns.token_id, vec![U256::from(1), U256::from(2), U256::from(3)]); + assert_eq!(columns.value, vec![U256::from(10), U256::from(20), U256::from(30)]); + assert_eq!(columns.is_batch, vec![true, true, true]); + // every row of a batch repeats the one log_index it came from + assert_eq!(columns.log_index, vec![7, 7, 7]); + } + + #[test] + fn single_is_one_row_of_the_same_shape() { + let event = ERC1155::TransferSingle { + operator: addr(0xaa), + from: addr(0xbb), + to: addr(0xcc), + id: U256::from(42), + value: U256::from(5), + }; + let mut columns = Erc1155Transfers::default(); + process_erc1155_transfers(vec![rpc_log(event.encode_log_data())], &mut columns, &schema()) + .unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.token_id_index, vec![0]); + assert_eq!(columns.is_batch, vec![false]); + assert_eq!(columns.operator, vec![vec![0xaau8; 20]]); + assert_eq!(columns.from_address, vec![vec![0xbbu8; 20]]); + assert_eq!(columns.to_address, vec![vec![0xccu8; 20]]); + } + + #[test] + fn batch_with_mismatched_array_lengths_is_skipped_whole() { + // Two ids and one value: no pairing is recoverable. The log must + // produce zero rows — never a padded or truncated one, never a panic. + let event = ERC1155::TransferBatch { + operator: addr(0xaa), + from: addr(0xbb), + to: addr(0xcc), + ids: vec![U256::from(1), U256::from(2)], + values: vec![U256::from(10)], + }; + let mut columns = Erc1155Transfers::default(); + process_erc1155_transfers(vec![rpc_log(event.encode_log_data())], &mut columns, &schema()) + .unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn empty_batch_yields_no_rows() { + let event = ERC1155::TransferBatch { + operator: addr(0xaa), + from: addr(0xbb), + to: addr(0xcc), + ids: vec![], + values: vec![], + }; + let mut columns = Erc1155Transfers::default(); + process_erc1155_transfers(vec![rpc_log(event.encode_log_data())], &mut columns, &schema()) + .unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn zero_address_sides_are_flagged_as_mint_and_burn() { + let mint = ERC1155::TransferSingle { + operator: addr(0xaa), + from: Address::ZERO, + to: addr(0xcc), + id: U256::from(1), + value: U256::from(1), + }; + let burn = ERC1155::TransferSingle { + operator: addr(0xaa), + from: addr(0xbb), + to: Address::ZERO, + id: U256::from(1), + value: U256::from(1), + }; + let logs = vec![rpc_log(mint.encode_log_data()), rpc_log(burn.encode_log_data())]; + let mut columns = Erc1155Transfers::default(); + process_erc1155_transfers(logs, &mut columns, &schema()).unwrap(); + + assert_eq!(columns.is_mint, vec![true, false]); + assert_eq!(columns.is_burn, vec![false, true]); + } + + #[test] + fn topic_count_alone_does_not_separate_the_two_events() { + // Regression guard for the dispatch: both signatures carry 4 topics, so + // a shape-only check would decode a batch body as a single. + assert_ne!(ERC1155::TransferSingle::SIGNATURE_HASH, ERC1155::TransferBatch::SIGNATURE_HASH); + let batch = ERC1155::TransferBatch { + operator: addr(0xaa), + from: addr(0xbb), + to: addr(0xcc), + ids: vec![U256::from(9)], + values: vec![U256::from(9)], + }; + let log = rpc_log(batch.encode_log_data()); + assert_eq!(log.topics().len(), 4); + assert!(is_erc1155_transfer(&log)); + assert!(ERC1155::TransferSingle::decode_log_data(log.data()).is_err()); + } + + #[test] + fn foreign_topic0_is_rejected_by_the_predicate_and_the_decoder() { + let log = rpc_log(LogData::new_unchecked( + vec![B256::repeat_byte(0xfe), B256::ZERO, B256::ZERO, B256::ZERO], + Default::default(), + )); + assert!(!is_erc1155_transfer(&log)); + let mut columns = Erc1155Transfers::default(); + process_erc1155_transfers(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn log_missing_identity_fields_is_skipped() { + let event = ERC1155::TransferSingle { + operator: addr(0xaa), + from: addr(0xbb), + to: addr(0xcc), + id: U256::from(1), + value: U256::from(1), + }; + let mut log = rpc_log(event.encode_log_data()); + log.log_index = None; + let mut columns = Erc1155Transfers::default(); + process_erc1155_transfers(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } +} diff --git a/crates/freeze/src/datasets/erc20_allowances.rs b/crates/freeze/src/datasets/erc20_allowances.rs new file mode 100644 index 0000000..d2047df --- /dev/null +++ b/crates/freeze/src/datasets/erc20_allowances.rs @@ -0,0 +1,184 @@ +use crate::*; +use alloy::{ + primitives::{Bytes, U256}, + sol_types::SolCall, +}; +use polars::prelude::*; +use std::collections::HashMap; + +/// columns for erc20_allowances +/// +/// One row per `(token, owner, spender)` per block: how much of the owner's +/// balance the spender is currently permitted to move. +/// +/// # This is the state, not the history +/// +/// Two datasets answer questions about ERC-20 approvals, and they answer +/// different ones. Pick by the question, not by the name: +/// +/// * [`Erc20Approvals`] reads the `Approval` **event** log. It gives every approval *change* in a +/// block range, with the transaction that made it, and needs no owner or spender named in advance +/// — it discovers them. Use it to ask "who approved what, and when". +/// * This dataset issues an `allowance(owner, spender)` **call**. It gives the value in force at a +/// block, whatever sequence of approvals produced it, and requires the owner and spender to be +/// named. Use it to ask "what can be spent right now". +/// +/// The event log alone cannot answer the second question. An allowance also +/// falls when the spender spends — `transferFrom` decrements it without +/// emitting `Approval` on most implementations — so replaying approval events +/// overstates what is actually spendable. Only the call reads the truth. +/// +/// # On "unlimited" approvals +/// +/// The common way to grant an unbounded allowance is to set it to +/// `type(uint256).max`. It is a convention, not a rule: some tokens and +/// front-ends use `2^255 - 1`, and some decrement from whatever was set. The +/// `allowance` column is therefore the raw value, with no derived "unlimited" +/// flag — testing `== U256::MAX` would miss the other conventions, and +/// thresholding at some round number would be this tool inventing a fact. +/// Threshold it yourself, in the direction your analysis needs. +#[triodion_macros::to_df(Datatype::Erc20Allowances)] +#[derive(Default)] +pub struct Erc20Allowances { + n_rows: u64, + block_number: Vec, + // The token, `Dim::Contract`. + erc20: Vec>, + // The owner whose tokens may be spent, `Dim::FromAddress`. Named + // `from_address` rather than `owner` so a row here joins directly against + // [`Erc20Approvals`], which stores the `Approval` event's indexed owner in + // a column of the same name. The whole point of having both datasets is + // being able to put them side by side. + from_address: Vec>, + // The spender permitted to move them, `Dim::ToAddress`. Matches + // `Erc20Approvals::to_address` for the same reason. + to_address: Vec>, + // 0 and null are different facts. A successful read of 0 means the spender + // may move nothing — a real measurement, and the state after a revocation. + // A null means the call did not return an allowance at all: the address has + // no code at this block, or is not an ERC-20. Filling the null with 0 would + // report "approved for nothing" where the right answer is "no such token". + allowance: Vec>, + chain_id: Vec, +} + +impl Dataset for Erc20Allowances { + fn default_blocks() -> Option { + Some("latest".to_string()) + } + + fn aliases() -> Vec<&'static str> { + vec!["allowances"] + } + + fn required_parameters() -> Vec { + // All three are required, and none of them can be inferred. An + // `allowance()` call takes an owner and a spender; there is no state + // read that enumerates the spenders a token has approved, because the + // mapping's keys are not recoverable from its slots. A user who does + // not know the pairs yet wants `erc20_approvals`, which finds them in + // the event log, and can then feed them back here. + vec![Dim::Contract, Dim::FromAddress, Dim::ToAddress] + } + + fn default_sort() -> Option> { + Some(vec!["block_number", "erc20", "from_address", "to_address"]) + } +} + +type BlockErc20OwnerSpenderAllowance = (u32, Vec, Vec, Vec, Option); + +impl CollectByBlock for Erc20Allowances { + type Response = BlockErc20OwnerSpenderAllowance; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let owner = request.ethers_from_address()?; + let spender = request.ethers_to_address()?; + let contract = request.ethers_contract()?; + let block_number = request.ethers_block_number()?; + let call_data = ERC20::allowanceCall { owner, spender }.abi_encode(); + // A revert, or an address with no code, means "no allowance to report" + // and becomes a null. A node that could not serve the state propagates, + // so the chunk is counted as errored rather than written out as nulls. + let output = contract_read(source.call2(contract, call_data, block_number).await)?; + let allowance = output.and_then(|bytes| decode_u256_word(&bytes)); + Ok(( + request.block_number()? as u32, + request.contract()?, + request.from_address()?, + request.to_address()?, + allowance, + )) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc20Allowances)?; + process_allowance(response, columns, schema) + } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.multicall { + multicall_collect_by_block::(partition, source, query, inner_request_size).await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } +} + +impl CollectByTransaction for Erc20Allowances { + type Response = (); +} + +/// Allowances aggregate through Multicall3 like any other `eth_call` dataset. +/// +/// One `(token, owner, spender)` triple is one `allowance()` call, and calls +/// sharing a block aggregate into a single `aggregate3`. There is no state +/// override to be had here: the extractor trick reads *slots*, and finding the +/// slot of `allowances[owner][spender]` needs the mapping's base slot, which +/// varies per token and is not discoverable from the ABI. +impl MulticallBatchable for Erc20Allowances { + fn calls_for_row(params: &Params, require_success: bool) -> R> { + let owner = params.ethers_from_address()?; + let spender = params.ethers_to_address()?; + let contract = params.ethers_contract()?; + let call_data = ERC20::allowanceCall { owner, spender }.abi_encode(); + Ok(vec![Multicall3::Call3 { + target: contract, + allowFailure: !require_success, + callData: Bytes::from(call_data), + }]) + } + + fn decode_row(params: &Params, results: &[Multicall3::Result]) -> R { + // Indexing would panic the worker task on a short aggregate3 return. + let r = results.first().ok_or_else(|| err("multicall returned no result for row"))?; + let allowance = if r.success { decode_u256_word(&r.returnData) } else { None }; + Ok(( + params.block_number()? as u32, + params.contract()?, + params.from_address()?, + params.to_address()?, + allowance, + )) + } +} + +fn process_allowance( + response: BlockErc20OwnerSpenderAllowance, + columns: &mut Erc20Allowances, + schema: &Table, +) -> R<()> { + let (block, erc20, from_address, to_address, allowance) = response; + columns.n_rows += 1; + store!(schema, columns, block_number, block); + store!(schema, columns, erc20, erc20); + store!(schema, columns, from_address, from_address); + store!(schema, columns, to_address, to_address); + store!(schema, columns, allowance, allowance); + Ok(()) +} diff --git a/crates/freeze/src/datasets/erc20_approvals.rs b/crates/freeze/src/datasets/erc20_approvals.rs index 44bfb5b..e040874 100644 --- a/crates/freeze/src/datasets/erc20_approvals.rs +++ b/crates/freeze/src/datasets/erc20_approvals.rs @@ -1,4 +1,7 @@ -use crate::*; +use crate::{ + types::rpc_params::{address_topic_matches, log_address_matches}, + *, +}; use alloy::{ primitives::U256, rpc::types::{Filter, Log, Topic}, @@ -100,7 +103,21 @@ impl CollectByTransaction for Erc20Approvals { async fn extract(request: Params, source: Arc, _: Arc) -> R { let logs = source.get_transaction_logs(request.transaction_hash()?).await?; - Ok(logs.into_iter().filter(is_erc20_approval).collect()) + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_erc20_approval(log) && + log_address_matches(log, &request.address) && + address_topic_matches(log, 1, &request.from_address) && + address_topic_matches(log, 2, &request.to_address) + }) + .collect()) } fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { diff --git a/crates/freeze/src/datasets/erc20_balances.rs b/crates/freeze/src/datasets/erc20_balances.rs index 0a470d1..62646d9 100644 --- a/crates/freeze/src/datasets/erc20_balances.rs +++ b/crates/freeze/src/datasets/erc20_balances.rs @@ -1,6 +1,6 @@ use crate::*; use alloy::{ - primitives::{Address, Bytes, U256}, + primitives::{Bytes, U256}, sol_types::SolCall, }; use polars::prelude::*; @@ -77,8 +77,8 @@ impl CollectByTransaction for Erc20Balances { impl MulticallBatchable for Erc20Balances { fn calls_for_row(params: &Params, require_success: bool) -> R> { - let owner = Address::from_slice(¶ms.address()?); - let contract = Address::from_slice(¶ms.contract()?); + let owner = params.ethers_address()?; + let contract = params.ethers_contract()?; let call_data = ERC20::balanceOfCall { owner }.abi_encode(); Ok(vec![Multicall3::Call3 { target: contract, diff --git a/crates/freeze/src/datasets/erc20_metadata.rs b/crates/freeze/src/datasets/erc20_metadata.rs index 2b2c4cb..b7cedf2 100644 --- a/crates/freeze/src/datasets/erc20_metadata.rs +++ b/crates/freeze/src/datasets/erc20_metadata.rs @@ -1,8 +1,5 @@ use crate::*; -use alloy::{ - primitives::{Address, Bytes}, - sol_types::SolCall, -}; +use alloy::{primitives::Bytes, sol_types::SolCall}; use polars::prelude::*; use std::collections::HashMap; @@ -100,7 +97,7 @@ impl CollectByTransaction for Erc20Metadata { impl MulticallBatchable for Erc20Metadata { fn calls_for_row(params: &Params, require_success: bool) -> R> { - let target = Address::from_slice(¶ms.address()?); + let target = params.ethers_address()?; let allow_failure = !require_success; Ok(vec![ Multicall3::Call3 { diff --git a/crates/freeze/src/datasets/erc20_supplies.rs b/crates/freeze/src/datasets/erc20_supplies.rs index 89d4caa..68337fe 100644 --- a/crates/freeze/src/datasets/erc20_supplies.rs +++ b/crates/freeze/src/datasets/erc20_supplies.rs @@ -1,6 +1,6 @@ use crate::*; use alloy::{ - primitives::{Address, Bytes, U256}, + primitives::{Bytes, U256}, sol_types::SolCall, }; use polars::prelude::*; @@ -88,7 +88,7 @@ impl CollectByTransaction for Erc20Supplies { impl MulticallBatchable for Erc20Supplies { fn calls_for_row(params: &Params, require_success: bool) -> R> { - let target = Address::from_slice(¶ms.address()?); + let target = params.ethers_address()?; // totalSupply() takes no args; emit just the selector. (The legacy // per-call path above concatenates the address to the calldata — // harmless because extra calldata is ignored, but the batched path @@ -136,6 +136,8 @@ mod tests { l1_provider: None, l1_chain_id: None, l1_rpc_url: None, + state_override_support: Arc::new(Default::default()), + storage_values_misses: Arc::new(std::sync::atomic::AtomicU32::new(0)), beacon: None, }) } @@ -162,6 +164,9 @@ mod tests { multicall: false, multicall_batch_size: 0, multicall_require_success: false, + batch_state_reads: false, + batch_rpc_calls: false, + state_override_batch_size: 0, }) } diff --git a/crates/freeze/src/datasets/erc20_transfers.rs b/crates/freeze/src/datasets/erc20_transfers.rs index 9c8c845..9420e4c 100644 --- a/crates/freeze/src/datasets/erc20_transfers.rs +++ b/crates/freeze/src/datasets/erc20_transfers.rs @@ -1,4 +1,7 @@ -use crate::*; +use crate::{ + types::rpc_params::{address_topic_matches, log_address_matches}, + *, +}; use alloy::{ primitives::U256, rpc::types::{Filter, Log, Topic}, @@ -99,7 +102,21 @@ impl CollectByTransaction for Erc20Transfers { async fn extract(request: Params, source: Arc, _: Arc) -> R { let logs = source.get_transaction_logs(request.transaction_hash()?).await?; - Ok(logs.into_iter().filter(is_erc20_transfer).collect()) + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_erc20_transfer(log) && + log_address_matches(log, &request.address) && + address_topic_matches(log, 1, &request.from_address) && + address_topic_matches(log, 2, &request.to_address) + }) + .collect()) } fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { diff --git a/crates/freeze/src/datasets/erc20_wrapper_events.rs b/crates/freeze/src/datasets/erc20_wrapper_events.rs index e0755ed..052d316 100644 --- a/crates/freeze/src/datasets/erc20_wrapper_events.rs +++ b/crates/freeze/src/datasets/erc20_wrapper_events.rs @@ -1,4 +1,7 @@ -use crate::{types::rpc_params::fixed_from_slice, *}; +use crate::{ + types::rpc_params::{fixed_from_slice, log_address_matches, topic_matches}, + *, +}; use alloy::{ primitives::{B256, U256}, rpc::types::{Filter, Log, Topic}, @@ -112,7 +115,20 @@ impl CollectByTransaction for Erc20WrapperEvents { async fn extract(request: Params, source: Arc, _: Arc) -> R { let logs = source.get_transaction_logs(request.transaction_hash()?).await?; - Ok(logs.into_iter().filter(is_wrapper_event).collect()) + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_wrapper_event(log) && + log_address_matches(log, &request.address) && + topic_matches(log, 1, &request.topic1) + }) + .collect()) } fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { diff --git a/crates/freeze/src/datasets/erc2612_nonces.rs b/crates/freeze/src/datasets/erc2612_nonces.rs new file mode 100644 index 0000000..74b4fd8 --- /dev/null +++ b/crates/freeze/src/datasets/erc2612_nonces.rs @@ -0,0 +1,217 @@ +use crate::*; +use alloy::{ + primitives::{Bytes, U256}, + sol_types::SolCall, +}; +use polars::prelude::*; +use std::collections::HashMap; + +/// columns for erc2612_nonces +/// +/// One row per (token, owner) per block: the ERC-2612 permit nonce. +/// +/// ERC-2612 lets a token owner approve a spender with an off-chain signature. +/// The `permit` call that redeems that signature emits the ordinary ERC-20 +/// `Approval` event and nothing else — the standard defines no event of its +/// own. In [`Erc20Approvals`] a permit-granted approval is therefore +/// indistinguishable from one granted by an on-chain `approve()`: same topic0, +/// same owner, same spender, same value. +/// +/// The nonce is the only on-chain counter of how many permits an owner has +/// signed for a token. Read it at two blocks and the difference is the number +/// of permits redeemed in between. +/// +/// This dataset gives that count. It does **not** give the linkage: it cannot +/// attribute a specific `Approval` row to a permit. Answering that needs the +/// transaction's calldata — the `permit` selector, possibly nested inside a +/// router's own multicall — which is a different question from the one asked +/// here. +#[triodion_macros::to_df(Datatype::Erc2612Nonces)] +#[derive(Default)] +pub struct Erc2612Nonces { + n_rows: u64, + block_number: Vec, + // The token, `Dim::Contract`. + erc20: Vec>, + // The permit owner, `Dim::Address`. The nonce is per owner per token, so + // both dimensions are required to name a row. + address: Vec>, + // 0 and null are different facts here. A successful read of 0 means the + // owner has signed no permit for this token yet — a real measurement. A + // null means the call did not return a nonce at all, so the token has no + // ERC-2612 support and the concept does not exist for it. Filling the null + // with 0 would report "never permitted" for tokens that can never permit. + nonce: Vec>, + // EIP-712 domain separator, opt-in because it is constant per token per + // chain and repeats identically on every row. Its null is the cheapest + // signal that a token is not a permit token: `DOMAIN_SEPARATOR()` reverts + // on anything that does not implement ERC-2612. + domain_separator: Vec>>, + chain_id: Vec, +} + +impl Dataset for Erc2612Nonces { + fn default_columns() -> Option> { + Some(vec![ + "block_number", + "erc20", + "address", + "nonce", + // "domain_separator", + "chain_id", + ]) + } + + fn default_sort() -> Option> { + Some(vec!["block_number", "erc20", "address"]) + } + + fn required_parameters() -> Vec { + vec![Dim::Contract, Dim::Address] + } + + fn default_blocks() -> Option { + Some("latest".to_string()) + } +} + +type BlockTokenOwnerNonce = (u32, Vec, Vec, Option, Option>); + +impl CollectByBlock for Erc2612Nonces { + type Response = BlockTokenOwnerNonce; + + async fn extract(request: Params, source: Arc, query: Arc) -> R { + let block_number = request.ethers_block_number()?; + let token = request.ethers_contract()?; + let owner_bytes = request.address()?; + let owner = request.ethers_address()?; + + // `contract_read` keeps the two failure modes apart. A revert, or an + // address with no code, is a real answer about the chain and becomes a + // null cell; a node that could not serve the state — pruned history on + // a non-archive endpoint, a rate limit, a timeout — propagates, so the + // chunk is counted as errored instead of written out as nulls. + + let call_data = ERC2612::noncesCall { owner }.abi_encode(); + let output = contract_read(source.call2(token, call_data, block_number).await)?; + let nonce = output.and_then(|bytes| decode_u256_word(&bytes)); + + // Skipped unless the column was actually selected. It is off by default + // and constant per (token, chain), so issuing it regardless doubled the + // request count of every default run for a value that was then thrown + // away by `store!`. + // + // The Multicall3 path below still sends it unconditionally, because + // `calls_for_row` gets no schema and must return a fixed call count. + // That is not free there either: `multicall_batch_size` caps *inner + // calls*, not rows, so two calls per row halves the rows per + // `aggregate3` (125 instead of 250) and doubles the number of + // `aggregate3` requests. Multicall is on by default, so the default + // path is the one paying it. Making the saving real on that path needs + // the schema threaded into `MulticallBatchable::calls_for_row`; until + // then, `--no-multicall` is the only way to avoid the wasted call. + let domain_separator = if query + .schemas + .get_schema(&Datatype::Erc2612Nonces) + .is_ok_and(|schema| schema.has_column("domain_separator")) + { + let call_data = ERC2612::DOMAIN_SEPARATORCall {}.abi_encode(); + let output = contract_read(source.call2(token, call_data, block_number).await)?; + output.and_then(|bytes| decode_bytes32_word(&bytes)) + } else { + None + }; + + let block = request.block_number()? as u32; + Ok((block, request.contract()?, owner_bytes, nonce, domain_separator)) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc2612Nonces)?; + let (block, erc20, address, nonce, domain_separator) = response; + columns.n_rows += 1; + store!(schema, columns, block_number, block); + store!(schema, columns, erc20, erc20); + store!(schema, columns, address, address); + store!(schema, columns, nonce, nonce); + store!(schema, columns, domain_separator, domain_separator); + Ok(()) + } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.multicall { + multicall_collect_by_block::(partition, source, query, inner_request_size).await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } +} + +// A nonce is a state read at a block, not something a transaction produces. The +// permit that moved it is one transaction, but the value is only defined +// relative to a block, so `-t` has no row to return. +impl CollectByTransaction for Erc2612Nonces { + type Response = (); +} + +impl MulticallBatchable for Erc2612Nonces { + fn calls_for_row(params: &Params, require_success: bool) -> R> { + let owner = params.ethers_address()?; + let target = params.ethers_contract()?; + // Failure is the expected answer for every token that is not an + // ERC-2612 permit token, so `--multicall-require-success` is the only + // thing that should ever make these calls fatal. + let allow_failure = !require_success; + Ok(vec![ + Multicall3::Call3 { + target, + allowFailure: allow_failure, + callData: Bytes::from(ERC2612::noncesCall { owner }.abi_encode()), + }, + Multicall3::Call3 { + target, + allowFailure: allow_failure, + callData: Bytes::from(ERC2612::DOMAIN_SEPARATORCall {}.abi_encode()), + }, + ]) + } + + fn decode_row(params: &Params, results: &[Multicall3::Result]) -> R { + // `calls_for_row` emits exactly two calls; a shorter slice means the + // node returned a malformed aggregate3, and indexing would panic the + // worker task rather than surface that as an error. + let [nonce_result, domain_result] = results else { + return Err(err("multicall returned the wrong number of results for row")) + }; + let nonce = + if nonce_result.success { decode_u256_word(&nonce_result.returnData) } else { None }; + let domain_separator = if domain_result.success { + decode_bytes32_word(&domain_result.returnData) + } else { + None + }; + Ok(( + params.block_number()? as u32, + params.contract()?, + params.address()?, + nonce, + domain_separator, + )) + } +} + +/// Decode an `eth_call` return that should be a single `bytes32`. +/// +/// A conformant `DOMAIN_SEPARATOR()` returns exactly one 32-byte word. Anything +/// else — an empty return (what an address with no code answers), a short +/// payload, or a longer one — is not a domain separator. Padding a short return +/// or taking the leading word of a long one would mint a separator that never +/// existed on chain, and a wrong separator reads as a valid one. +fn decode_bytes32_word(data: &[u8]) -> Option> { + (data.len() == 32).then(|| data.to_vec()) +} diff --git a/crates/freeze/src/datasets/erc4626_metadata.rs b/crates/freeze/src/datasets/erc4626_metadata.rs new file mode 100644 index 0000000..181c45d --- /dev/null +++ b/crates/freeze/src/datasets/erc4626_metadata.rs @@ -0,0 +1,230 @@ +use crate::*; +use alloy::{ + primitives::{Bytes, U256}, + sol_types::SolCall, +}; +use polars::prelude::*; +use std::collections::HashMap; + +/// columns for erc4626_metadata +/// +/// One row per (vault, block): the three reads that describe an ERC-4626 vault +/// at that block. +/// +/// There is deliberately no share-price column. A price per share is +/// `total_assets / total_supply`, and both operands are already in the row, so +/// the column would add no information while forcing three choices on every +/// reader: what to do when `total_supply` is 0 (a fresh vault, or one whose +/// shares were all redeemed — the ratio is undefined, not 1.0), how many +/// decimals to round to, and which of the two token decimals to scale by. Any +/// answer picked here is wrong for someone. `convertToAssets(1e18)` is also not +/// the same number for vaults that charge an exit fee, so a single column +/// cannot even name one convention. Consumers divide the two columns +/// themselves, with their own convention. +#[triodion_macros::to_df(Datatype::Erc4626Metadata)] +#[derive(Default)] +pub struct Erc4626Metadata { + n_rows: u64, + block_number: Vec, + // The vault, i.e. the share token. `asset` below is the token it holds. + erc4626: Vec>, + asset: Vec>>, + // Null means the vault refused the read (a revert, or no code at that + // address at that block). Zero means the vault answered zero, which is a + // real state: a vault that has just been deployed, or fully drained. + total_assets: Vec>, + // Shares outstanding, not the supply of the underlying token. Zero here is + // real on a fresh vault; with a non-zero `total_assets` it is the donated + // assets state that the inflation attack exploits. Never rewrite it as a + // null. + total_supply: Vec>, + chain_id: Vec, +} + +impl Dataset for Erc4626Metadata { + fn default_columns() -> Option> { + Some(vec!["block_number", "erc4626", "asset", "total_assets", "total_supply", "chain_id"]) + } + + fn default_sort() -> Option> { + Some(vec!["erc4626", "block_number"]) + } + + fn default_blocks() -> Option { + Some("latest".to_string()) + } + + fn required_parameters() -> Vec { + vec![Dim::Address] + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for Erc4626Metadata { + type Response = (u32, Vec, Option>, Option, Option); + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let block_number = request.ethers_block_number()?; + let address = request.ethers_address()?; + + // Each read folds a *contract-level* refusal (revert, or an address + // with no code) into `None`, while a *node-level* failure — pruned + // state on a non-archive endpoint, a rate limit, a timeout — + // propagates via `?`. Without that split, pointing this dataset at a + // non-archive RPC yields a file of nulls under a "chunks errored: 0" + // banner. + + // asset + let call_data = ERC4626::assetCall {}.abi_encode(); + let asset = contract_read(source.call2(address, call_data, block_number).await)? + .and_then(|output| decode_address_word(&output)); + + // totalAssets + let call_data = ERC4626::totalAssetsCall {}.abi_encode(); + let total_assets = contract_read(source.call2(address, call_data, block_number).await)? + .and_then(|output| decode_u256_word(&output)); + + // totalSupply + let call_data = ERC4626::totalSupplyCall {}.abi_encode(); + let total_supply = contract_read(source.call2(address, call_data, block_number).await)? + .and_then(|output| decode_u256_word(&output)); + + Ok((request.block_number()? as u32, request.address()?, asset, total_assets, total_supply)) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc4626Metadata)?; + let (block, vault, asset, total_assets, total_supply) = response; + columns.n_rows += 1; + store!(schema, columns, block_number, block); + store!(schema, columns, erc4626, vault); + store!(schema, columns, asset, asset); + store!(schema, columns, total_assets, total_assets); + store!(schema, columns, total_supply, total_supply); + Ok(()) + } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.multicall { + multicall_collect_by_block::(partition, source, query, inner_request_size).await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } +} + +impl CollectByTransaction for Erc4626Metadata { + type Response = (); +} + +impl MulticallBatchable for Erc4626Metadata { + fn calls_for_row(params: &Params, require_success: bool) -> R> { + let target = params.ethers_address()?; + let allow_failure = !require_success; + // Same encodings as the per-call path above, so both paths send the + // same bytes for the same row. + Ok(vec![ + Multicall3::Call3 { + target, + allowFailure: allow_failure, + callData: Bytes::from(ERC4626::assetCall {}.abi_encode()), + }, + Multicall3::Call3 { + target, + allowFailure: allow_failure, + callData: Bytes::from(ERC4626::totalAssetsCall {}.abi_encode()), + }, + Multicall3::Call3 { + target, + allowFailure: allow_failure, + callData: Bytes::from(ERC4626::totalSupplyCall {}.abi_encode()), + }, + ]) + } + + fn default_multicall_batch_size() -> u32 { + // 60 inner calls, i.e. 20 rows of 3, against the crate default of 250 + // (83 rows). + // + // `totalAssets()` is the expensive inner call the default's own doc + // says to override for. On a vault that iterates strategies it is a + // six-figure-gas read, so 83 rows of it can approach geth's 50,000,000 + // `eth_call` cap — and a batch that runs out of gas does not error: + // the starved inner calls return `success: false` with empty + // `returnData`, which this dataset maps to nulls. + 60 + } + + fn decode_row(params: &Params, results: &[Multicall3::Result]) -> R { + // `calls_for_row` emits exactly three calls; a shorter slice means the + // node returned a malformed aggregate3, and indexing would panic the + // worker task rather than surface that as an error. + let [asset_result, total_assets_result, total_supply_result] = results else { + return Err(err("multicall returned the wrong number of results for row")) + }; + let asset = + if asset_result.success { decode_address_word(&asset_result.returnData) } else { None }; + let total_assets = if total_assets_result.success { + decode_u256_word(&total_assets_result.returnData) + } else { + None + }; + let total_supply = if total_supply_result.success { + decode_u256_word(&total_supply_result.returnData) + } else { + None + }; + Ok((params.block_number()? as u32, params.address()?, asset, total_assets, total_supply)) + } +} + +/// Decode an `eth_call` return that should be a single `address`. +/// +/// A conformant `asset()` returns exactly one 32-byte word whose upper 12 +/// bytes are zero. Nothing else is an address: +/// +/// - an empty return is what an address with no code answers, and taking its low 20 bytes is not +/// possible at all; +/// - a word with dirty upper bytes is a packed storage slot or some other payload, and masking it +/// down to 20 bytes invents an underlying token the vault never named. +/// +/// Both alternatives fabricate an address, so the strict width-and-padding +/// check is the only reading that cannot. +fn decode_address_word(data: &[u8]) -> Option> { + // The length test runs first, so the two slices are always in bounds. + (data.len() == 32 && data[..12].iter().all(|b| *b == 0)).then(|| data[12..].to_vec()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_padded_word_decodes_to_the_low_twenty_bytes() { + let mut word = [0u8; 32]; + word[12..].copy_from_slice(&[0xab; 20]); + assert_eq!(decode_address_word(&word), Some(vec![0xab; 20])); + } + + #[test] + fn an_empty_return_is_not_an_address() { + // What an EOA, or a self-destructed vault, answers. + assert_eq!(decode_address_word(&[]), None); + } + + #[test] + fn a_word_with_dirty_upper_bytes_is_not_an_address() { + let mut word = [0u8; 32]; + word[0] = 1; + word[12..].copy_from_slice(&[0xab; 20]); + assert_eq!(decode_address_word(&word), None, "masking a packed slot invents an address"); + } +} diff --git a/crates/freeze/src/datasets/erc4626_vault_events.rs b/crates/freeze/src/datasets/erc4626_vault_events.rs new file mode 100644 index 0000000..4dea945 --- /dev/null +++ b/crates/freeze/src/datasets/erc4626_vault_events.rs @@ -0,0 +1,382 @@ +use crate::{ + types::rpc_params::{fixed_from_slice, log_address_matches, topic_matches}, + *, +}; +use alloy::{ + primitives::{B256, U256}, + rpc::types::{Filter, Log, Topic}, + sol_types::SolEvent, +}; +use polars::prelude::*; + +/// Column values for `event_name`, spelled the way the ABI spells them. Every +/// dataset with an `event_name` column uses the ABI casing, so one column name +/// keeps one convention across the crate. +const DEPOSIT: &str = "Deposit"; +const WITHDRAW: &str = "Withdraw"; + +/// columns for ERC-4626 tokenised vault `Deposit` and `Withdraw` events +/// +/// Both events are kept in one table because they are the two halves of the +/// same movement: assets in against shares out, and shares in against assets +/// out. `event_name` says which half a row is. +/// +/// ERC-4626's `Deposit(address,address,uint256,uint256)` is **not** the +/// WETH-style `Deposit(address,uint256)` that [`Erc20WrapperEvents`] collects. +/// The two signatures hash to different topic0 values, so the filters here and +/// there select disjoint sets of logs and a vault deposit can never turn up as +/// a wrapper deposit. Do not dedupe across the two datasets on that assumption. +/// +/// There is deliberately no share-price column. The price is `assets / shares`, +/// a division whose scaling and rounding belong to whoever asks the question, +/// and `shares == 0` would make it undefined — a value this table would then +/// have to invent. +/// +/// Matching happens at the event-signature level with no per-contract +/// configuration, so any contract emitting these exact signatures is indexed +/// whether or not it is a conforming vault. +#[triodion_macros::to_df(Datatype::Erc4626VaultEvents)] +#[derive(Default)] +pub struct Erc4626VaultEvents { + n_rows: u64, + block_number: Vec, + block_hash: Vec>>, + transaction_index: Vec, + log_index: Vec, + transaction_hash: Vec>, + /// the vault contract that emitted the event + vault: Vec>, + /// `"Deposit"` or `"Withdraw"`, spelled the way the ABI spells them so a row + /// joins back to the event definition without a translation table. + /// `"Withdraw"` is a different event from the `"withdrawal"` of + /// [`Erc20WrapperEvents`]. + event_name: Vec, + /// the caller that moved the assets or shares + sender: Vec>, + /// the account whose shares were minted (`Deposit`) or burned (`Withdraw`) + owner: Vec>, + /// who received the assets. `Withdraw` names a receiver; `Deposit` has no + /// receiver argument at all, so the column is null there. + receiver: Vec>>, + /// underlying assets moved, in the vault's asset units + assets: Vec, + /// vault shares moved, in the vault's share units + shares: Vec, + chain_id: Vec, +} + +impl Dataset for Erc4626VaultEvents { + fn aliases() -> Vec<&'static str> { + vec!["erc4626_events"] + } + + fn default_columns() -> Option> { + Some(vec![ + "block_number", + // "block_hash", + "transaction_index", + "log_index", + "transaction_hash", + "vault", + "event_name", + "sender", + "owner", + "receiver", + "assets", + "shares", + "chain_id", + ]) + } + + fn optional_parameters() -> Vec { + // Topic0 is fixed to the two vault signatures. Topic1 is `sender` in + // both events, so filtering on it means one thing. Topic2 and Topic3 + // are deliberately not offered: topic2 is `owner` on a Deposit but + // `receiver` on a Withdraw, so one filter value would silently select + // two different roles. + vec![Dim::Address, Dim::Topic1] + } + + fn use_block_ranges() -> bool { + true + } + + fn default_inner_request_size() -> u64 { + // Same 50-block default as the other log datasets. Vault events are far + // rarer per block than ERC-20 transfers, so this stays well inside any + // node's response limits while keeping request counts predictable + // across log-shaped datasets. + 50 + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for Erc4626VaultEvents { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let mut topics: [Topic; 4] = Default::default(); + // Union filter — eth_getLogs treats a list at one topic position as OR. + topics[0] = + vec![ERC4626::Deposit::SIGNATURE_HASH, ERC4626::Withdraw::SIGNATURE_HASH].into(); + if let Some(sender) = &request.topic1 { + topics[1] = fixed_from_slice::(sender, "topic1")?.into(); + } + let filter = Filter { topics, ..request.ethers_log_filter()? }; + let logs = source.get_logs(&filter).await?; + Ok(logs.into_iter().filter(is_erc4626_vault_event).collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc4626VaultEvents)?; + process_vault_events(response, columns, schema) + } +} + +impl CollectByTransaction for Erc4626VaultEvents { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let logs = source.get_transaction_logs(request.transaction_hash()?).await?; + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_erc4626_vault_event(log) && + log_address_matches(log, &request.address) && + topic_matches(log, 1, &request.topic1) + }) + .collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc4626VaultEvents)?; + process_vault_events(response, columns, schema) + } +} + +/// which of the two ERC-4626 events a log carries +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum VaultEvent { + Deposit, + Withdraw, +} + +/// The event a log carries, or `None` when it is neither. The topic count is +/// checked against the event's own declaration, not against a shared shape: +/// `Deposit` indexes two arguments and `Withdraw` indexes three, so a single +/// count would accept a malformed log of the other kind. +fn vault_event(log: &Log) -> Option { + // Both events carry exactly two non-indexed uint256 words: assets, shares. + if log.data().data.len() != 64 { + return None + } + let topics = log.topics(); + match topics.first() { + Some(t) if *t == ERC4626::Deposit::SIGNATURE_HASH && topics.len() == 3 => { + Some(VaultEvent::Deposit) + } + Some(t) if *t == ERC4626::Withdraw::SIGNATURE_HASH && topics.len() == 4 => { + Some(VaultEvent::Withdraw) + } + _ => None, + } +} + +/// True iff `log` is a well-formed ERC-4626 `Deposit` or `Withdraw`. Shared with +/// the per-transaction path, which receives every log of a transaction and must +/// pick these out itself. +pub(crate) fn is_erc4626_vault_event(log: &Log) -> bool { + vault_event(log).is_some() +} + +/// process logs into columns +fn process_vault_events(logs: Vec, columns: &mut Erc4626VaultEvents, schema: &Table) -> R<()> { + for log in logs.iter() { + // Re-checked rather than assumed: the topic indexing below would panic + // on a log of the wrong shape, and a future caller may reach this loop + // with logs that no filter narrowed first. + let Some(event) = vault_event(log) else { continue }; + let (Some(bn), Some(tx), Some(ti), Some(li)) = + (log.block_number, log.transaction_hash, log.transaction_index, log.log_index) + else { + continue + }; + + let topics = log.topics(); + // Withdraw(sender, receiver, owner) puts owner last; Deposit(sender, + // owner) has no receiver argument at all. Copying owner into receiver + // on a deposit row would invent a role the event never named, and any + // "who was paid out" query would then count deposits as payouts. + let (owner, receiver) = match event { + VaultEvent::Deposit => (topics[2][12..].to_vec(), None), + VaultEvent::Withdraw => (topics[3][12..].to_vec(), Some(topics[2][12..].to_vec())), + }; + let event_name = match event { + VaultEvent::Deposit => DEPOSIT, + VaultEvent::Withdraw => WITHDRAW, + }; + let data = &log.data().data; + + columns.n_rows += 1; + store!(schema, columns, block_number, bn as u32); + store!(schema, columns, block_hash, log.block_hash.map(|hash| hash.to_vec())); + store!(schema, columns, transaction_index, ti as u32); + store!(schema, columns, log_index, li as u32); + store!(schema, columns, transaction_hash, tx.to_vec()); + store!(schema, columns, vault, log.address().to_vec()); + store!(schema, columns, event_name, event_name.to_string()); + // Indexed addresses arrive left-padded to 32 bytes; keep the low 20. + store!(schema, columns, sender, topics[1][12..].to_vec()); + store!(schema, columns, owner, owner); + store!(schema, columns, receiver, receiver); + // `vault_event` proved the data is exactly 64 bytes, so neither slice + // can be out of range. assets first, shares second, as declared. + store!(schema, columns, assets, U256::from_be_slice(&data[..32])); + store!(schema, columns, shares, U256::from_be_slice(&data[32..])); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy::primitives::{Address, Bytes}; + + /// real `Erc4626VaultEvents` schema, so `store!` actually evaluates each column + fn schema() -> Table { + Datatype::Erc4626VaultEvents + .table_schema( + &[U256Type::String], + &ColumnEncoding::Hex, + &None, + &None, + &None, + None, + None, + ) + .expect("Erc4626VaultEvents has a valid default schema") + } + + /// the two non-indexed words: assets then shares + fn data(assets: u64, shares: u64) -> Vec { + let mut out = U256::from(assets).to_be_bytes::<32>().to_vec(); + out.extend_from_slice(&U256::from(shares).to_be_bytes::<32>()); + out + } + + /// rpc `Log` carrying the block/tx/index fields `process_vault_events` requires + fn log_with(topics: Vec, data: Vec) -> Log { + let inner = alloy::primitives::Log::new_unchecked(Address::ZERO, topics, Bytes::from(data)); + Log { + inner, + block_number: Some(1), + transaction_hash: Some(B256::ZERO), + transaction_index: Some(0), + log_index: Some(0), + ..Default::default() + } + } + + fn addr(byte: u8) -> B256 { + B256::left_padding_from(&[byte; 20]) + } + + #[test] + fn erc4626_deposit_does_not_share_topic0_with_the_weth_deposit() { + // The whole reason both datasets can filter on "Deposit" without + // colliding. If this ever fails, the two tables overlap. + assert_ne!(ERC4626::Deposit::SIGNATURE_HASH, ERC20Wrapper::Deposit::SIGNATURE_HASH); + assert_ne!(ERC4626::Withdraw::SIGNATURE_HASH, ERC20Wrapper::Withdrawal::SIGNATURE_HASH); + assert_ne!(ERC4626::Deposit::SIGNATURE_HASH, ERC4626::Withdraw::SIGNATURE_HASH); + } + + #[test] + fn deposit_has_no_receiver() { + let log = + log_with(vec![ERC4626::Deposit::SIGNATURE_HASH, addr(0xaa), addr(0xbb)], data(7, 5)); + let mut columns = Erc4626VaultEvents::default(); + process_vault_events(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.event_name, vec![DEPOSIT.to_string()]); + assert_eq!(columns.sender, vec![vec![0xaau8; 20]]); + assert_eq!(columns.owner, vec![vec![0xbbu8; 20]]); + assert_eq!(columns.receiver, vec![None], "a deposit names no receiver"); + assert_eq!(columns.assets, vec![U256::from(7u64)]); + assert_eq!(columns.shares, vec![U256::from(5u64)]); + } + + #[test] + fn withdraw_reads_owner_from_the_third_indexed_argument() { + // topics: sig, sender, receiver, owner — owner last, not second. + let log = log_with( + vec![ERC4626::Withdraw::SIGNATURE_HASH, addr(0xaa), addr(0xbb), addr(0xcc)], + data(9, 3), + ); + let mut columns = Erc4626VaultEvents::default(); + process_vault_events(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.event_name, vec![WITHDRAW.to_string()]); + assert_eq!(columns.sender, vec![vec![0xaau8; 20]]); + assert_eq!(columns.receiver, vec![Some(vec![0xbbu8; 20])]); + assert_eq!(columns.owner, vec![vec![0xccu8; 20]]); + } + + #[test] + fn zero_shares_with_nonzero_assets_is_a_row() { + // A donation-style deposit: assets go in, no shares come out. Legal, + // and the row must survive so the vault's asset base stays reconcilable. + let log = log_with( + vec![ERC4626::Deposit::SIGNATURE_HASH, addr(0xaa), addr(0xbb)], + data(1_000, 0), + ); + let mut columns = Erc4626VaultEvents::default(); + process_vault_events(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.shares, vec![U256::ZERO]); + assert_eq!(columns.assets, vec![U256::from(1_000u64)]); + } + + #[test] + fn topic_count_is_checked_per_event_not_shared() { + let d = ERC4626::Deposit::SIGNATURE_HASH; + let w = ERC4626::Withdraw::SIGNATURE_HASH; + // Deposit with a Withdraw's topic count, and the reverse. + assert!(!is_erc4626_vault_event(&log_with(vec![d, addr(1), addr(2), addr(3)], data(1, 1)))); + assert!(!is_erc4626_vault_event(&log_with(vec![w, addr(1), addr(2)], data(1, 1)))); + assert!(is_erc4626_vault_event(&log_with(vec![d, addr(1), addr(2)], data(1, 1)))); + assert!(is_erc4626_vault_event(&log_with(vec![w, addr(1), addr(2), addr(3)], data(1, 1)))); + } + + #[test] + fn wrong_data_width_and_foreign_topic0_are_skipped() { + let logs = vec![ + log_with(vec![ERC4626::Deposit::SIGNATURE_HASH, addr(1), addr(2)], vec![0u8; 32]), + log_with(vec![B256::repeat_byte(0xfe), addr(1), addr(2)], data(1, 1)), + log_with(vec![], data(1, 1)), + ]; + let mut columns = Erc4626VaultEvents::default(); + process_vault_events(logs, &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn skips_logs_missing_identity_fields() { + let mut log = + log_with(vec![ERC4626::Deposit::SIGNATURE_HASH, addr(1), addr(2)], data(1, 1)); + log.log_index = None; + let mut columns = Erc4626VaultEvents::default(); + process_vault_events(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } +} diff --git a/crates/freeze/src/datasets/erc721_metadata.rs b/crates/freeze/src/datasets/erc721_metadata.rs index 6418e93..4b08a5e 100644 --- a/crates/freeze/src/datasets/erc721_metadata.rs +++ b/crates/freeze/src/datasets/erc721_metadata.rs @@ -1,8 +1,5 @@ use crate::*; -use alloy::{ - primitives::{Address, Bytes}, - sol_types::SolCall, -}; +use alloy::{primitives::Bytes, sol_types::SolCall}; use polars::prelude::*; use std::collections::HashMap; @@ -87,7 +84,7 @@ impl CollectByTransaction for Erc721Metadata { impl MulticallBatchable for Erc721Metadata { fn calls_for_row(params: &Params, require_success: bool) -> R> { - let target = Address::from_slice(¶ms.address()?); + let target = params.ethers_address()?; let allow_failure = !require_success; Ok(vec![ Multicall3::Call3 { diff --git a/crates/freeze/src/datasets/erc721_transfers.rs b/crates/freeze/src/datasets/erc721_transfers.rs index a9bc543..78a0dff 100644 --- a/crates/freeze/src/datasets/erc721_transfers.rs +++ b/crates/freeze/src/datasets/erc721_transfers.rs @@ -1,4 +1,7 @@ -use crate::*; +use crate::{ + types::rpc_params::{address_topic_matches, log_address_matches}, + *, +}; use alloy::{ primitives::U256, rpc::types::{Filter, Log, Topic}, @@ -96,7 +99,21 @@ impl CollectByTransaction for Erc721Transfers { async fn extract(request: Params, source: Arc, _: Arc) -> R { let logs = source.get_transaction_logs(request.transaction_hash()?).await?; - Ok(logs.into_iter().filter(is_erc721_transfer).collect()) + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_erc721_transfer(log) && + log_address_matches(log, &request.address) && + address_topic_matches(log, 1, &request.from_address) && + address_topic_matches(log, 2, &request.to_address) + }) + .collect()) } fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { diff --git a/crates/freeze/src/datasets/erc777_transfers.rs b/crates/freeze/src/datasets/erc777_transfers.rs new file mode 100644 index 0000000..fd1a035 --- /dev/null +++ b/crates/freeze/src/datasets/erc777_transfers.rs @@ -0,0 +1,490 @@ +use crate::{ + types::rpc_params::{fixed_from_slice, log_address_matches, topic_matches}, + *, +}; +use alloy::{ + primitives::{B256, U256}, + rpc::types::{Filter, Log, Topic}, + sol_types::SolEvent, +}; +use polars::prelude::*; + +/// columns for erc777_transfers +/// +/// One row per ERC-777 `Sent`, `Minted` or `Burned` event. +/// +/// # THIS DATASET DOUBLE-COUNTS AGAINST `erc20_transfers`. DO NOT UNION THEM. +/// +/// ERC-777 is a superset of ERC-20, and a compliant ERC-777 token emits an +/// ERC-20 `Transfer` **for the same movement** alongside every `Sent`, +/// `Minted` and `Burned`. Both logs are real, both are in the receipt, and +/// both are collected — `Transfer` by [`Erc20Transfers`], the richer one here. +/// They are two views of one movement, not two movements. +/// +/// So `SELECT ... FROM erc20_transfers UNION ALL SELECT ... FROM +/// erc777_transfers` reports every ERC-777 movement twice, and every volume, +/// count and balance derived from it is wrong by exactly the ERC-777 share of +/// the chain. Pick one table per token: this one when the operator or the +/// data payloads matter, `erc20_transfers` when only the movement does. +/// If you must have one table, anti-join on +/// `(transaction_hash, block_number, erc777)` — never on `log_index`, because +/// the mirrored pair are two different logs with two different indices. +/// +/// The mirroring is a convention, not a consensus rule. A token can emit one +/// without the other, so neither table is a strict subset of the other. +#[triodion_macros::to_df(Datatype::Erc777Transfers)] +#[derive(Default)] +pub struct Erc777Transfers { + n_rows: u64, + block_number: Vec, + block_hash: Vec>>, + transaction_index: Vec, + log_index: Vec, + transaction_hash: Vec>, + erc777: Vec>, + // The Solidity identifier as written in the standard — "Sent", "Minted", + // "Burned" — so the value joins straight to an ABI without a translation + // table. Which of the three it is decides which of the address columns + // below is null. + event_name: Vec, + // The party that executed the move. This is what ERC-777 adds over + // ERC-20: an authorised operator can move a holder's tokens with no + // allowance and no call from the holder. Always present — on an ordinary + // self-initiated send the holder is its own operator, so a naive + // COUNT(DISTINCT operator) counts every ordinary sender as an operator. + // `is_operator_send` beside it separates the two. + operator: Vec>, + // Null on `Minted`. A mint has no payer, and ERC-777 does not use the + // ERC-20 zero-address convention for mints, so writing 0x0 here would + // invent a party that the event never named. + from_address: Vec>>, + // Null on `Burned`, for the same reason: a burn names no recipient. + to_address: Vec>>, + amount: Vec, + // Non-indexed dynamic `bytes` from the log body, ABI-decoded. Empty is a + // real, common value here — most sends carry no payload — and it is + // stored as zero-length bytes, not as a null. Null would mean the field + // does not exist, and it always exists on all three events. + data: Vec>, + // The operator's own payload, distinct from the holder's `data`. Same + // rule: empty bytes, never null. + operator_data: Vec>, + // Derived, not reported: `operator != from_address`. True when a third + // party moved someone else's tokens, which is the case that has no ERC-20 + // equivalent. Null on `Minted`, where there is no `from` to compare + // against — false would claim a self-send that the event never described. + is_operator_send: Vec>, + chain_id: Vec, +} + +impl Dataset for Erc777Transfers { + fn default_columns() -> Option> { + Some(vec![ + "block_number", + // "block_hash", + "transaction_index", + "log_index", + "transaction_hash", + "erc777", + "event_name", + "operator", + "from_address", + "to_address", + "amount", + "data", + "operator_data", + "is_operator_send", + "chain_id", + ]) + } + + fn optional_parameters() -> Vec { + // Topic0 is fixed to the three ERC-777 signatures, so it is not + // offered. The remaining topics are exposed by position, not by name, + // because their meaning is not the same across the three events: + // topic1 = operator, on all three; + // topic2 = `from` on Sent and Burned, but `to` on Minted; + // topic3 = `to`, on Sent only. + // A `--from-address` dim would therefore have to mean topic2, and + // would silently match Minted recipients too. Filtering by topic3 also + // drops every Minted and Burned row, because those logs have no + // topic3 for the node to match against. + vec![Dim::Address, Dim::Topic1, Dim::Topic2, Dim::Topic3] + } + + fn use_block_ranges() -> bool { + true + } + + fn default_inner_request_size() -> u64 { + // The same 50 as every other log dataset, and for the reason + // `proxy_upgrades` gives: the dispatcher applies + // `max(user, dataset_default)`, so this value is a floor and not a + // default. A wider one would fetch these rare logs in fewer requests, + // but `--inner-request-size` could then only raise it, never lower it, + // and an endpoint that caps the block span of `eth_getLogs` would + // reject every request with no flag left to fix it. + 50 + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for Erc777Transfers { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let mut topics: [Topic; 4] = Default::default(); + // Union filter — eth_getLogs treats a list at one topic position as OR. + topics[0] = vec![ + ERC777::Sent::SIGNATURE_HASH, + ERC777::Minted::SIGNATURE_HASH, + ERC777::Burned::SIGNATURE_HASH, + ] + .into(); + // Topic dims are the full 32-byte word, as everywhere else in the repo. + // Left-padding a 20-byte address here would be dead code: the struct + // update below evaluates `ethers_log_filter()`, which runs + // `fixed_from_slice::` over the same raw dims and errors on + // anything that is not exactly 32 bytes, so a short value never + // survives to be padded. + for (position, dim) in [(1, &request.topic1), (2, &request.topic2), (3, &request.topic3)] { + if let Some(bytes) = dim { + topics[position] = + fixed_from_slice::(bytes, &format!("topic{position}"))?.into(); + } + } + let filter = Filter { topics, ..request.ethers_log_filter()? }; + let logs = source.get_logs(&filter).await?; + Ok(logs.into_iter().filter(is_erc777_transfer_event).collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc777Transfers)?; + process_erc777_transfers(response, columns, schema) + } +} + +impl CollectByTransaction for Erc777Transfers { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let logs = source.get_transaction_logs(request.transaction_hash()?).await?; + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + is_erc777_transfer_event(log) && + log_address_matches(log, &request.address) && + topic_matches(log, 1, &request.topic1) && + topic_matches(log, 2, &request.topic2) && + topic_matches(log, 3, &request.topic3) + }) + .collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::Erc777Transfers)?; + process_erc777_transfers(response, columns, schema) + } +} + +/// True iff `log` carries one of the three ERC-777 movement signatures *and* +/// the topic count that signature implies: Sent indexes three addresses, +/// Minted and Burned index two. +/// +/// The arity check is what stops a contract that reuses one of these topic0 +/// values with a different argument list from being decoded as ERC-777. The +/// by-transaction path also needs the signature check, because it receives +/// every log in the transaction with no node-side filtering. +/// +/// Written to stay reusable by a coalesced [`crate::LogEvents`] fan-out, which this datatype is NOT +/// a member of today (see `MultiDatatype::LogEvents`). +pub(crate) fn is_erc777_transfer_event(log: &Log) -> bool { + match log.topics().first() { + Some(topic0) if *topic0 == ERC777::Sent::SIGNATURE_HASH => log.topics().len() == 4, + Some(topic0) if *topic0 == ERC777::Minted::SIGNATURE_HASH => log.topics().len() == 3, + Some(topic0) if *topic0 == ERC777::Burned::SIGNATURE_HASH => log.topics().len() == 3, + _ => false, + } +} + +/// One decoded ERC-777 movement, with the two absences made explicit. +struct DecodedErc777 { + event_name: &'static str, + operator: Vec, + from_address: Option>, + to_address: Option>, + amount: U256, + data: Vec, + operator_data: Vec, +} + +/// Decode one log, or `None` if it is not a well-formed ERC-777 movement. +/// +/// `data` and `operatorData` are non-indexed dynamic `bytes`: the log body +/// holds offsets, then lengths, then padded content, and the two payloads can +/// be any length in any order of appearance. Slicing fixed windows out of the +/// body silently mangles them, so the generated ABI decoder does the work and +/// a body it rejects becomes a skipped row rather than a wrong one. +fn decode_erc777_event(log: &Log) -> Option { + let topic0 = *log.topics().first()?; + let body = &log.inner.data; + + if topic0 == ERC777::Sent::SIGNATURE_HASH { + let event = ERC777::Sent::decode_log_data(body).ok()?; + Some(DecodedErc777 { + event_name: "Sent", + operator: event.operator.to_vec(), + from_address: Some(event.from.to_vec()), + to_address: Some(event.to.to_vec()), + amount: event.amount, + data: event.data.to_vec(), + operator_data: event.operatorData.to_vec(), + }) + } else if topic0 == ERC777::Minted::SIGNATURE_HASH { + let event = ERC777::Minted::decode_log_data(body).ok()?; + Some(DecodedErc777 { + event_name: "Minted", + operator: event.operator.to_vec(), + // No sender exists. See the column comment. + from_address: None, + to_address: Some(event.to.to_vec()), + amount: event.amount, + data: event.data.to_vec(), + operator_data: event.operatorData.to_vec(), + }) + } else if topic0 == ERC777::Burned::SIGNATURE_HASH { + let event = ERC777::Burned::decode_log_data(body).ok()?; + Some(DecodedErc777 { + event_name: "Burned", + operator: event.operator.to_vec(), + from_address: Some(event.from.to_vec()), + // No recipient exists. See the column comment. + to_address: None, + amount: event.amount, + data: event.data.to_vec(), + operator_data: event.operatorData.to_vec(), + }) + } else { + None + } +} + +/// process logs into columns +fn process_erc777_transfers( + logs: Vec, + columns: &mut Erc777Transfers, + schema: &Table, +) -> R<()> { + for log in logs.iter() { + // Both callers pre-filter, but the coalesced LogEvents extractor may + // hand this loop mixed logs. Re-checking here keeps a foreign log a + // skipped row instead of a mis-tagged one. + if !is_erc777_transfer_event(log) { + continue; + } + let (Some(bn), Some(tx), Some(ti), Some(li)) = + (log.block_number, log.transaction_hash, log.transaction_index, log.log_index) + else { + continue; + }; + let Some(event) = decode_erc777_event(log) else { continue }; + + // Undefined rather than false when there is no counterparty to compare. + let is_operator_send = event.from_address.as_ref().map(|from| *from != event.operator); + + columns.n_rows += 1; + store!(schema, columns, block_number, bn as u32); + store!(schema, columns, block_hash, log.block_hash.map(|bh| bh.to_vec())); + store!(schema, columns, transaction_index, ti as u32); + store!(schema, columns, log_index, li as u32); + store!(schema, columns, transaction_hash, tx.to_vec()); + store!(schema, columns, erc777, log.address().to_vec()); + store!(schema, columns, event_name, event.event_name.to_string()); + store!(schema, columns, operator, event.operator); + store!(schema, columns, from_address, event.from_address); + store!(schema, columns, to_address, event.to_address); + store!(schema, columns, amount, event.amount); + store!(schema, columns, data, event.data); + store!(schema, columns, operator_data, event.operator_data); + store!(schema, columns, is_operator_send, is_operator_send); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy::primitives::{Address, Bytes, B256}; + + /// real `Erc777Transfers` schema, so `store!` actually evaluates each column + fn schema() -> Table { + Datatype::Erc777Transfers + .table_schema( + &[U256Type::String], + &ColumnEncoding::Hex, + &None, + &None, + &None, + None, + None, + ) + .expect("Erc777Transfers has a valid default schema") + } + + fn addr(byte: u8) -> Address { + Address::repeat_byte(byte) + } + + /// rpc `Log` around an encoded event, carrying the identity fields the + /// processor requires + fn rpc_log(data: alloy::primitives::LogData) -> Log { + Log { + inner: alloy::primitives::Log { address: addr(0x11), data }, + block_number: Some(1), + transaction_hash: Some(B256::ZERO), + transaction_index: Some(0), + log_index: Some(0), + ..Default::default() + } + } + + fn sent(operator: Address, from: Address, to: Address, data: Vec) -> Log { + rpc_log( + ERC777::Sent { + operator, + from, + to, + amount: U256::from(7u64), + data: Bytes::from(data), + operatorData: Bytes::new(), + } + .encode_log_data(), + ) + } + + #[test] + fn mint_has_no_sender_and_burn_has_no_recipient() { + // The whole point of the two Option columns: an absent party must be + // null, never the zero address. + let minted = rpc_log( + ERC777::Minted { + operator: addr(0xaa), + to: addr(0xbb), + amount: U256::from(1u64), + data: Bytes::new(), + operatorData: Bytes::new(), + } + .encode_log_data(), + ); + let burned = rpc_log( + ERC777::Burned { + operator: addr(0xaa), + from: addr(0xcc), + amount: U256::from(2u64), + data: Bytes::new(), + operatorData: Bytes::new(), + } + .encode_log_data(), + ); + + let mut columns = Erc777Transfers::default(); + process_erc777_transfers(vec![minted, burned], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 2); + assert_eq!(columns.event_name, vec!["Minted".to_string(), "Burned".to_string()]); + assert_eq!(columns.from_address, vec![None, Some(vec![0xccu8; 20])]); + assert_eq!(columns.to_address, vec![Some(vec![0xbbu8; 20]), None]); + // Minted has no `from`, so the comparison is undefined, not false. + assert_eq!(columns.is_operator_send, vec![None, Some(true)]); + } + + #[test] + fn self_initiated_send_is_not_an_operator_send() { + let logs = vec![ + sent(addr(0xaa), addr(0xaa), addr(0xbb), vec![]), + sent(addr(0x99), addr(0xaa), addr(0xbb), vec![]), + ]; + let mut columns = Erc777Transfers::default(); + process_erc777_transfers(logs, &mut columns, &schema()).unwrap(); + assert_eq!(columns.is_operator_send, vec![Some(false), Some(true)]); + } + + #[test] + fn dynamic_payloads_round_trip_and_empty_is_not_null() { + // `data` is dynamic bytes in the body; a fixed-window slice would not + // survive a payload that is not a multiple of 32 bytes. + let payload = vec![0xde, 0xad, 0xbe, 0xef, 0x01]; + let logs = vec![ + sent(addr(0xaa), addr(0xaa), addr(0xbb), payload.clone()), + sent(addr(0xaa), addr(0xaa), addr(0xbb), vec![]), + ]; + let mut columns = Erc777Transfers::default(); + process_erc777_transfers(logs, &mut columns, &schema()).unwrap(); + assert_eq!(columns.data, vec![payload, Vec::::new()]); + assert_eq!(columns.operator_data, vec![Vec::::new(), Vec::::new()]); + } + + #[test] + fn predicate_requires_the_arity_the_signature_implies() { + let ok = sent(addr(0xaa), addr(0xaa), addr(0xbb), vec![]); + assert!(is_erc777_transfer_event(&ok)); + + // Sent topic0 with only three topics is not an ERC-777 Sent. + let truncated = rpc_log(alloy::primitives::LogData::new_unchecked( + vec![ERC777::Sent::SIGNATURE_HASH, B256::ZERO, B256::ZERO], + ok.inner.data.data.clone(), + )); + assert!(!is_erc777_transfer_event(&truncated)); + + let foreign = rpc_log(alloy::primitives::LogData::new_unchecked( + vec![B256::repeat_byte(0xfe), B256::ZERO, B256::ZERO, B256::ZERO], + ok.inner.data.data.clone(), + )); + assert!(!is_erc777_transfer_event(&foreign)); + } + + #[test] + fn undecodable_body_is_skipped_not_panicking() { + // Right signature and arity, truncated body: skip the row. + let malformed = rpc_log(alloy::primitives::LogData::new_unchecked( + vec![ERC777::Sent::SIGNATURE_HASH, B256::ZERO, B256::ZERO, B256::ZERO], + Bytes::from(vec![0u8; 8]), + )); + assert!(is_erc777_transfer_event(&malformed)); + + let mut columns = Erc777Transfers::default(); + process_erc777_transfers(vec![malformed], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn skips_logs_missing_identity_fields() { + let mut log = sent(addr(0xaa), addr(0xaa), addr(0xbb), vec![]); + log.log_index = None; + let mut columns = Erc777Transfers::default(); + process_erc777_transfers(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn the_three_signatures_are_distinct() { + let sig_sent = ERC777::Sent::SIGNATURE_HASH; + let sig_minted = ERC777::Minted::SIGNATURE_HASH; + let sig_burned = ERC777::Burned::SIGNATURE_HASH; + assert_ne!(sig_sent, sig_minted); + assert_ne!(sig_sent, sig_burned); + assert_ne!(sig_minted, sig_burned); + // The mirrored ERC-20 Transfer is a different log entirely, which is + // why unioning the two datasets double-counts. + assert_ne!(sig_sent, ERC20::Transfer::SIGNATURE_HASH); + } +} diff --git a/crates/freeze/src/datasets/eth_calls.rs b/crates/freeze/src/datasets/eth_calls.rs index 3e656bd..8bfa08f 100644 --- a/crates/freeze/src/datasets/eth_calls.rs +++ b/crates/freeze/src/datasets/eth_calls.rs @@ -1,6 +1,6 @@ use crate::*; use alloy::{ - primitives::{keccak256, Address, Bytes, TxKind}, + primitives::{keccak256, Bytes, TxKind}, rpc::types::{TransactionInput, TransactionRequest}, }; use polars::prelude::*; @@ -78,7 +78,7 @@ impl CollectByTransaction for EthCalls { impl MulticallBatchable for EthCalls { fn calls_for_row(params: &Params, require_success: bool) -> R> { Ok(vec![Multicall3::Call3 { - target: Address::from_slice(¶ms.contract()?), + target: params.ethers_contract()?, allowFailure: !require_success, callData: Bytes::from(params.call_data()?), }]) diff --git a/crates/freeze/src/datasets/logs.rs b/crates/freeze/src/datasets/logs.rs index 7d94368..1158425 100644 --- a/crates/freeze/src/datasets/logs.rs +++ b/crates/freeze/src/datasets/logs.rs @@ -1,4 +1,7 @@ -use crate::*; +use crate::{ + types::rpc_params::{log_address_matches, topic_matches}, + *, +}; use alloy::{ dyn_abi::{DynSolType, DynSolValue, EventExt}, rpc::types::Log, @@ -87,7 +90,23 @@ impl CollectByTransaction for Logs { type Response = Vec; async fn extract(request: Params, source: Arc, _: Arc) -> R { - source.get_transaction_logs(request.transaction_hash()?).await + let logs = source.get_transaction_logs(request.transaction_hash()?).await?; + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| { + log_address_matches(log, &request.address) && + topic_matches(log, 0, &request.topic0) && + topic_matches(log, 1, &request.topic1) && + topic_matches(log, 2, &request.topic2) && + topic_matches(log, 3, &request.topic3) + }) + .collect()) } fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { diff --git a/crates/freeze/src/datasets/mod.rs b/crates/freeze/src/datasets/mod.rs index 5a7153a..c9c238d 100644 --- a/crates/freeze/src/datasets/mod.rs +++ b/crates/freeze/src/datasets/mod.rs @@ -2,6 +2,8 @@ pub mod access_lists; /// address appearances pub mod address_appearances; +/// ERC-721 / ERC-1155 ApprovalForAll events +pub mod approvals_for_all; /// authorizations pub mod authorizations; /// balance diffs @@ -24,11 +26,19 @@ pub mod code_reads; pub mod codes; /// consolidation requests pub mod consolidation_requests; +/// ERC-165 interface probing +pub mod contract_interfaces; /// contracts pub mod contracts; /// deposit requests pub mod deposit_requests; -/// erc20 approval +/// ERC-1155 URI events +pub mod erc1155_metadata; +/// ERC-1155 transfers +pub mod erc1155_transfers; +/// erc20 allowances +pub mod erc20_allowances; +/// erc20 approvals pub mod erc20_approvals; /// erc20 balances pub mod erc20_balances; @@ -40,10 +50,18 @@ pub mod erc20_supplies; pub mod erc20_transfers; /// erc20 wrapper events (Deposit / Withdrawal) pub mod erc20_wrapper_events; +/// ERC-2612 permit nonces +pub mod erc2612_nonces; +/// ERC-4626 vault metadata +pub mod erc4626_metadata; +/// ERC-4626 vault events +pub mod erc4626_vault_events; /// erc721 metadata pub mod erc721_metadata; /// erc721 transfers pub mod erc721_transfers; +/// ERC-777 transfers +pub mod erc777_transfers; /// eth calls pub mod eth_calls; /// four byte counts @@ -72,6 +90,10 @@ pub mod nonce_diffs; pub mod nonce_reads; /// nonces pub mod nonces; +/// ERC-1967 proxy slots +pub mod proxy_slots; +/// ERC-1967 proxy upgrade events +pub mod proxy_upgrades; /// slots pub mod slots; /// storage diffs @@ -90,9 +112,9 @@ pub mod vm_traces; pub mod withdrawal_requests; /// withdrawals pub mod withdrawals; - pub use access_lists::*; pub use address_appearances::*; +pub use approvals_for_all::*; pub use authorizations::*; pub use balance_diffs::*; pub use balance_reads::*; @@ -103,16 +125,24 @@ pub use code_diffs::*; pub use code_reads::*; pub use codes::*; pub use consolidation_requests::*; +pub use contract_interfaces::*; pub use contracts::*; pub use deposit_requests::*; +pub use erc1155_metadata::*; +pub use erc1155_transfers::*; +pub use erc20_allowances::*; pub use erc20_approvals::*; pub use erc20_balances::*; pub use erc20_metadata::*; pub use erc20_supplies::*; pub use erc20_transfers::*; pub use erc20_wrapper_events::*; +pub use erc2612_nonces::*; +pub use erc4626_metadata::*; +pub use erc4626_vault_events::*; pub use erc721_metadata::*; pub use erc721_transfers::*; +pub use erc777_transfers::*; pub use eth_calls::*; pub use four_byte_counts::*; pub use geth_balance_diffs::*; @@ -127,6 +157,8 @@ pub use native_transfers::*; pub use nonce_diffs::*; pub use nonce_reads::*; pub use nonces::*; +pub use proxy_slots::*; +pub use proxy_upgrades::*; pub use slots::*; pub use storage_diffs::*; pub use storage_reads::*; diff --git a/crates/freeze/src/datasets/nonces.rs b/crates/freeze/src/datasets/nonces.rs index b9de231..8e08fa3 100644 --- a/crates/freeze/src/datasets/nonces.rs +++ b/crates/freeze/src/datasets/nonces.rs @@ -1,6 +1,10 @@ use crate::*; -use alloy::primitives::Address; +use alloy::{ + eips::BlockNumberOrTag, + primitives::{Address, U64}, +}; use polars::prelude::*; +use std::collections::HashMap; /// columns for balances #[triodion_macros::to_df(Datatype::Nonces)] @@ -45,12 +49,58 @@ impl CollectByBlock for Nonces { let schema = query.schemas.get_schema(&Datatype::Nonces)?; process_nonce(columns, response, schema) } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.batch_rpc_calls { + rpc_batch_collect_by_block::(partition, source, query, inner_request_size).await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } } impl CollectByTransaction for Nonces { type Response = (); } +/// Nonces batch at the transport and nowhere else. +/// +/// This is the one account field the state-override trick cannot reach. The EVM +/// exposes an account's balance (`BALANCE`), its code (`EXTCODESIZE`, +/// `EXTCODEHASH`, `EXTCODECOPY`) and — from inside it — its storage (`SLOAD`), +/// but there is no opcode that reads another account's nonce. No injected +/// bytecode batches this, on any node, at any block. `eth_getProof` returns a +/// nonce, but one account per call, so it buys nothing here either. +/// +/// So the only lever is JSON-RPC batching: same request count reduction, no +/// requirements on the node beyond JSON-RPC itself. +impl RpcBatchable for Nonces { + type Param = (Address, BlockNumberOrTag); + type Item = U64; + + fn method() -> &'static str { + "eth_getTransactionCount" + } + + fn param_for_row(params: &Params) -> R { + // `ethers_address`, never `Address::from_slice`, which PANICS on a width + // mismatch — and `--address` is hex-decoded with no length check. This + // runs once per row of a whole chunk, so a panic would abort a hundred + // rows where the per-row path aborted one. As an error the chunk demotes + // and the per-row path reports that address as it always did. + Ok((params.ethers_address()?, BlockNumberOrTag::Number(params.block_number()?))) + } + + fn decode_row(params: &Params, item: Self::Item) -> R { + Ok((params.block_number()? as u32, None, params.address()?, item.to::())) + } +} + fn process_nonce(columns: &mut Nonces, data: BlockTxAddressOutput, schema: &Table) -> R<()> { let (block, _tx, address, output) = data; columns.n_rows += 1; diff --git a/crates/freeze/src/datasets/proxy_slots.rs b/crates/freeze/src/datasets/proxy_slots.rs new file mode 100644 index 0000000..2052039 --- /dev/null +++ b/crates/freeze/src/datasets/proxy_slots.rs @@ -0,0 +1,191 @@ +use crate::*; +use alloy::primitives::{b256, B256, U256}; +use polars::prelude::*; + +/// ERC-1967 implementation slot. +/// +/// `keccak256("eip1967.proxy.implementation") - 1`. The `- 1` puts the slot +/// outside the image of `keccak256`, so no mapping or array can ever be laid +/// out on top of it. +const ERC1967_IMPLEMENTATION_SLOT: B256 = + b256!("360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc"); + +/// ERC-1967 admin slot: `keccak256("eip1967.proxy.admin") - 1`. +const ERC1967_ADMIN_SLOT: B256 = + b256!("b53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d6103"); + +/// ERC-1967 beacon slot: `keccak256("eip1967.proxy.beacon") - 1`. +const ERC1967_BEACON_SLOT: B256 = + b256!("a3f0ad74e5423aebfd80d3ef4346578335a9a72aeaee59ff6cb3582b35133d50"); + +// The fourth ERC-1967 slot, rollback +// (`0x4910fdfa16fed3260ed0e7147f7cc6da11a60208b5b9406d12a635614ffd9143`), is +// deliberately not read. It is written and cleared inside a single upgrade +// transaction, so any read at a block boundary sees it unset. A column of it +// would be all nulls and would invite the reading "no proxy here ever rolled +// back", which the data cannot support. Do not "fix" this omission. + +/// columns for proxy_slots +/// +/// One row per (block, address), holding the three ERC-1967 proxy slots read +/// with `eth_getStorageAt`. +/// +/// A null means the slot is unset, i.e. the contract is not that kind of proxy +/// at that block. Every ordinary (non-proxy) contract answers null in all three +/// columns. +/// +/// `implementation` set with `beacon` null is a standard ERC-1967 proxy: the +/// code it delegates to is in `implementation`. `beacon` set is a beacon proxy, +/// and its real implementation is held by the beacon contract, reachable only +/// through `IBeacon.implementation()` on that address — it is not in this row +/// and must not be inferred from it. The two columns are separate precisely so +/// no single column has to mean both things. +#[triodion_macros::to_df(Datatype::ProxySlots)] +#[derive(Default)] +pub struct ProxySlots { + n_rows: u64, + block_number: Vec, + address: Vec>, + implementation: Vec>>, + admin: Vec>>, + beacon: Vec>>, + chain_id: Vec, +} + +impl Dataset for ProxySlots { + fn default_columns() -> Option> { + Some(vec!["block_number", "address", "implementation", "admin", "beacon", "chain_id"]) + } + + fn default_sort() -> Option> { + Some(vec!["block_number", "address"]) + } + + fn default_blocks() -> Option { + Some("latest".to_string()) + } + + fn required_parameters() -> Vec { + vec![Dim::Address] + } + + fn aliases() -> Vec<&'static str> { + vec!["erc1967_slots"] + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for ProxySlots { + type Response = (u32, Vec, Option>, Option>, Option>); + + async fn extract(request: Params, source: Arc, query: Arc) -> R { + let address = request.ethers_address()?; + let block_number = request.block_number()? as u32; + + // `from_be_bytes`, not `from_be_slice`: the latter panics when the slice + // is not at most 32 bytes. All three are 32-byte constants, so neither + // form can fail today — the total one is used so nothing is left to + // inherit if one of these ever becomes a runtime value. + let slots = [ + U256::from_be_bytes(ERC1967_IMPLEMENTATION_SLOT.0), + U256::from_be_bytes(ERC1967_ADMIN_SLOT.0), + U256::from_be_bytes(ERC1967_BEACON_SLOT.0), + ]; + + // Unlike an `eth_call` read of a getter, a storage read has no + // contract-level failure to fold into a null: an address with no code + // answers a zero word exactly as an unset slot does. So every error + // here is a node error and must propagate, or the chunk would be + // written as nulls. Neither path below ever turns a *batching* failure + // into an error, for exactly that reason. + let words = if query.batch_state_reads { + // Three slots of one contract, which is one state-override call + // rather than three `eth_getStorageAt`s. This dataset cannot use the + // `StateOverrideBatchable` runner — that batches one word per *row*, + // and a row here is three — so it reads through + // `read_storage_slots`, which carries the same demotion policy at + // the `Source` level and shares the same run-level verdict. + // + // Cross-address batching is not available to it either: `SLOAD` + // reads only the storage of the contract currently executing, so the + // extractor must be injected at each address in turn. The win is + // 3 -> 1 requests per row, and there is no larger group to find. + source.read_storage_slots(address, &slots, block_number.into()).await? + } else { + // `--no-batch-state-reads`. This is not the demotion path — that one + // lives inside `read_storage_slots` and is reached by failure, never + // by choice. It is the plain read the operator asked for, and the + // same three requests this dataset made before batching existed. + // + // Every other state-read dataset routes on this flag in + // `collect_by_block`; this one cannot, because the flag has to be + // read where the three slots are, so it routes here instead. A + // dataset that ignored the flag would make the switch a lie. + // + // All three requests go out at once, so turning batching off costs + // requests, not wall-clock — the same property `per_row` gives the + // runner. + let reads = + slots.iter().map(|slot| source.get_storage_at(address, *slot, block_number.into())); + futures::future::try_join_all(reads) + .await? + .into_iter() + // The same 32 big-endian bytes the batched path returns, so + // both routes write byte-identical cells. + .map(|word| B256::from(word.to_be_bytes::<32>())) + .collect() + }; + + // One word per requested slot, in order, on both paths. A different + // count is a bug in this crate, not a property of the chain, so it must + // not become three nulls — that would read as "not a proxy". + let [implementation, admin, beacon]: [B256; 3] = words.try_into().map_err(|_| { + err("state read returned the wrong number of words for the ERC-1967 slots") + })?; + + Ok(( + block_number, + request.address()?, + word_as_address(implementation), + word_as_address(admin), + word_as_address(beacon), + )) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::ProxySlots)?; + let (block, address, implementation, admin, beacon) = response; + columns.n_rows += 1; + store!(schema, columns, block_number, block); + store!(schema, columns, address, address); + store!(schema, columns, implementation, implementation); + store!(schema, columns, admin, admin); + store!(schema, columns, beacon, beacon); + Ok(()) + } +} + +// Storage is a property of a block, not of a transaction. `-t` would have to +// mean "the slots as of the block that contains this transaction", which is +// the same row for every transaction in that block and is already reachable by +// asking for the block. +impl CollectByTransaction for ProxySlots { + type Response = (); +} + +/// The address held in an ERC-1967 slot word, or `None` when the slot is unset. +fn word_as_address(word: B256) -> Option> { + // A proxy itself reads these slots as `address(uint160(uint256(word)))`, + // which keeps the low 20 bytes and drops the rest. Mirror that truncation + // instead of rejecting a word whose high 12 bytes are dirty. + let address = &word.0[12..]; + + // Zero means the slot was never written, so this contract is not that kind + // of proxy. Storing 0x00..00 would instead assert that it delegates to the + // zero address — a live claim about a live proxy — and every non-proxy + // contract in the file would then join against the zero-address row. + (!address.iter().all(|byte| *byte == 0)).then(|| address.to_vec()) +} diff --git a/crates/freeze/src/datasets/proxy_upgrades.rs b/crates/freeze/src/datasets/proxy_upgrades.rs new file mode 100644 index 0000000..3c96dda --- /dev/null +++ b/crates/freeze/src/datasets/proxy_upgrades.rs @@ -0,0 +1,512 @@ +use crate::{types::rpc_params::log_address_matches, *}; +use alloy::{ + rpc::types::{Filter, Log, Topic}, + sol_types::SolEvent, +}; +use polars::prelude::*; + +/// Column values for `event_name`, spelled the way the ABI spells them so a row +/// joins back to the event definition without a translation table. +const UPGRADED: &str = "Upgraded"; +const BEACON_UPGRADED: &str = "BeaconUpgraded"; +const ADMIN_CHANGED: &str = "AdminChanged"; + +/// `AdminChanged` carries two non-indexed `address` arguments, so its data is +/// exactly two ABI words. +const ADMIN_CHANGED_DATA_LEN: usize = 64; + +/// One ABI word, which is how a non-indexed `address` argument is encoded. +const ADDRESS_WORD_LEN: usize = 32; + +/// columns for ERC-1967 proxy upgrade events +/// +/// One row per emitted event, over all three ERC-1967 events: +/// `Upgraded(address indexed implementation)`, +/// `BeaconUpgraded(address indexed beacon)` and +/// `AdminChanged(address previousAdmin, address newAdmin)`. +/// +/// The columns an event does not carry are null, not zero. The zero address is +/// a real value here and must stay distinguishable from an absence: a proxy +/// upgraded to `0x0` is bricked, and an admin changed to `0x0` has been +/// renounced. Three separate tables would each be near-empty and would have to +/// be merged again before "what changed on this proxy, and in what order" could +/// be asked, so this is one table. +/// +/// This is the event-side view of a proxy. `proxy_slots` reads the ERC-1967 +/// storage slots and answers "what is the implementation at block N"; this +/// dataset answers "when did it change, and in which transaction". Join +/// `proxy_upgrades.proxy_address` to `proxy_slots.address`. Neither replaces +/// the other: +/// +/// - These events are a convention, not an EVM rule. A proxy can write the implementation slot and +/// emit nothing, and a beacon that swaps its own implementation moves every proxy behind it with +/// no event at any of them. No rows here is not evidence that nothing changed — only a slot read +/// is. +/// - A slot read cannot say how many times the slot changed between two blocks, or which +/// transaction changed it. +#[triodion_macros::to_df(Datatype::ProxyUpgrades)] +#[derive(Default)] +pub struct ProxyUpgrades { + n_rows: u64, + block_number: Vec, + block_hash: Vec>>, + transaction_index: Vec, + log_index: Vec, + transaction_hash: Vec>, + /// the contract that emitted the event: the proxy itself, never the + /// implementation it points at + proxy_address: Vec>, + /// `Upgraded`, `BeaconUpgraded` or `AdminChanged` + event_name: Vec, + /// `Upgraded` rows only + implementation: Vec>>, + /// `BeaconUpgraded` rows only. The implementation behind the beacon is not + /// in this log and must be read from the beacon contract. + beacon: Vec>>, + /// `AdminChanged` rows only + previous_admin: Vec>>, + /// `AdminChanged` rows only + new_admin: Vec>>, + chain_id: Vec, +} + +impl Dataset for ProxyUpgrades { + fn aliases() -> Vec<&'static str> { + vec!["erc1967_events"] + } + + fn default_columns() -> Option> { + Some(vec![ + "block_number", + // "block_hash", + "transaction_index", + "log_index", + "transaction_hash", + "proxy_address", + "event_name", + "implementation", + "beacon", + "previous_admin", + "new_admin", + "chain_id", + ]) + } + + fn optional_parameters() -> Vec { + // Topic0 is fixed to the three ERC-1967 signatures. Topic1 is + // deliberately not offered: it is the implementation on `Upgraded`, the + // beacon on `BeaconUpgraded`, and does not exist on `AdminChanged`, so + // any `--topic1` value would silently delete every `AdminChanged` row + // and mean two different things across the rows it kept. + vec![Dim::Address] + } + + fn use_block_ranges() -> bool { + true + } + + fn default_inner_request_size() -> u64 { + // The same 50 as the other log datasets. A larger value would fetch + // these rare events in fewer requests, but the dispatcher applies + // `max(user, dataset_default)`, so a high default becomes a floor the + // user cannot lower on a strict RPC endpoint. + 50 + } + + fn arg_aliases() -> Option> { + Some([(Dim::Contract, Dim::Address)].into_iter().collect()) + } +} + +impl CollectByBlock for ProxyUpgrades { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let mut topics: [Topic; 4] = Default::default(); + // A list at a topic position is OR in `eth_getLogs`, and alloy's `Topic` + // is built from a `Vec` for exactly that. Assigning a single hash + // instead would drop the other two events without any error. + topics[0] = vec![ + ERC1967::Upgraded::SIGNATURE_HASH, + ERC1967::BeaconUpgraded::SIGNATURE_HASH, + ERC1967::AdminChanged::SIGNATURE_HASH, + ] + .into(); + let filter = Filter { topics, ..request.ethers_log_filter()? }; + let logs = source.get_logs(&filter).await?; + // The node matched topic0 only; this also checks the shape each event + // must have before its arguments can be read. + Ok(logs.into_iter().filter(is_proxy_upgrade).collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::ProxyUpgrades)?; + process_proxy_upgrades(response, columns, schema) + } +} + +impl CollectByTransaction for ProxyUpgrades { + type Response = Vec; + + async fn extract(request: Params, source: Arc, _: Arc) -> R { + let logs = source.get_transaction_logs(request.transaction_hash()?).await?; + // The dims never reach the node on this path: `--txs` asks for one + // transaction's whole receipt, so the narrowing the by-block path pins + // into the `eth_getLogs` filter has to be re-applied here. Without it a + // dim is accepted, counted into the partition set, printed in the run + // summary, and then ignored — the run returns rows it was asked to + // exclude and says nothing. + Ok(logs + .into_iter() + .filter(|log| is_proxy_upgrade(log) && log_address_matches(log, &request.address)) + .collect()) + } + + fn transform(response: Self::Response, columns: &mut Self, query: &Arc) -> R<()> { + let schema = query.schemas.get_schema(&Datatype::ProxyUpgrades)?; + process_proxy_upgrades(response, columns, schema) + } +} + +/// One decoded ERC-1967 event. The variants carry only the arguments their event +/// actually has, so the null columns of a row are decided here rather than by a +/// default filled in later. +enum ProxyUpgradeEvent { + Upgraded { implementation: Vec }, + BeaconUpgraded { beacon: Vec }, + AdminChanged { previous_admin: Vec, new_admin: Vec }, +} + +/// True iff `log` is one of the three ERC-1967 events and has the shape that +/// event's arguments live in. Written to stay reusable by a coalesced [`crate::LogEvents`] fan-out, +/// which this datatype is NOT a member of today (see `MultiDatatype::LogEvents`). +pub(crate) fn is_proxy_upgrade(log: &Log) -> bool { + let Some(topic0) = log.topics().first() else { return false }; + if *topic0 == ERC1967::Upgraded::SIGNATURE_HASH || + *topic0 == ERC1967::BeaconUpgraded::SIGNATURE_HASH + { + // Two encodings of the same event, both live on mainnet. + // + // ERC-1967 and OpenZeppelin >= 3.x declare the address argument + // `indexed`, so it is topics[1] and the data is empty. The older + // zos-lib `AdminUpgradeabilityProxy` declares `Upgraded(address + // implementation)` with no `indexed`, so the log has one topic and the + // address sits in a single 32-byte data word. + // + // `indexed` is not part of the signature preimage, so both hash to the + // same topic0 and the node returns both for the filter in `extract`. + // Accepting only the topic form silently dropped every upgrade of a + // zos-lib proxy — USDC's is one — and reported the proxy as never + // upgraded. + // + // Extra data alongside a present topic1 is tolerated rather than + // rejected: the argument is still unambiguously in the topic, and + // dropping the row would lose a real upgrade over padding nobody + // reads. The width must be exact in the data form, because there the + // padding is the value. + log.topics().len() >= 2 || log.data().data.len() == ADDRESS_WORD_LEN + } else if *topic0 == ERC1967::AdminChanged::SIGNATURE_HASH { + // The opposite case, and the one that is easy to get backwards. + // `AdminChanged` indexes neither argument, so the log has one topic and + // both admins sit in the data. The width must be exact — the ABI decode + // below reads two words, and a different width means some other event + // collided with this signature. + log.data().data.len() == ADMIN_CHANGED_DATA_LEN + } else { + false + } +} + +/// The sole `address` argument of `Upgraded` or `BeaconUpgraded`, from whichever +/// of the two encodings this log uses. +/// +/// A present topic1 wins: the argument is `indexed`, so the topic is the value +/// and any data is padding. Otherwise the argument is non-indexed and the data +/// holds it as one ABI word. +/// +/// The high 12 bytes of that word must be zero. An `address` is ABI-encoded +/// left-padded with zeros, so a non-zero prefix means the 32 bytes are not an +/// address — some other event collided with this signature — and inventing an +/// address from its low 20 bytes would be worse than skipping the row. +fn address_argument(log: &Log) -> Option> { + if let Some(topic) = log.topics().get(1) { + return Some(topic[12..].to_vec()) + } + let word = log.data().data.as_ref(); + if word.len() != ADDRESS_WORD_LEN || word[..12].iter().any(|byte| *byte != 0) { + return None + } + Some(word[12..].to_vec()) +} + +/// Decode one log, or `None` if it is not a well-formed ERC-1967 event. +fn decode_proxy_upgrade(log: &Log) -> Option { + let topic0 = *log.topics().first()?; + if topic0 == ERC1967::Upgraded::SIGNATURE_HASH { + let implementation = address_argument(log)?; + Some(ProxyUpgradeEvent::Upgraded { implementation }) + } else if topic0 == ERC1967::BeaconUpgraded::SIGNATURE_HASH { + let beacon = address_argument(log)?; + Some(ProxyUpgradeEvent::BeaconUpgraded { beacon }) + } else if topic0 == ERC1967::AdminChanged::SIGNATURE_HASH { + if log.data().data.len() != ADMIN_CHANGED_DATA_LEN { + return None + } + // Read from the data, not the topics. `decode_log_data` re-checks the + // signature and topic count, and a failure here skips the row. + let decoded = ERC1967::AdminChanged::decode_log_data(log.data()).ok()?; + Some(ProxyUpgradeEvent::AdminChanged { + previous_admin: decoded.previousAdmin.to_vec(), + new_admin: decoded.newAdmin.to_vec(), + }) + } else { + None + } +} + +/// process logs into columns +fn process_proxy_upgrades(logs: Vec, columns: &mut ProxyUpgrades, schema: &Table) -> R<()> { + for log in logs.iter() { + // Decode before anything is stored. A caller that reaches here with + // mixed logs — a future coalesced [`crate::LogEvents`] fan-out, say — + // skips them here instead of writing a row of nulls under a real + // block number, which would read as "an upgrade with no argument". + let Some(event) = decode_proxy_upgrade(log) else { continue }; + + // Each arm names every column the event does not have. An `Upgraded` + // row has no admin at all, which is not the same as an admin of zero. + let (event_name, implementation, beacon, previous_admin, new_admin) = match event { + ProxyUpgradeEvent::Upgraded { implementation } => { + (UPGRADED, Some(implementation), None, None, None) + } + ProxyUpgradeEvent::BeaconUpgraded { beacon } => { + (BEACON_UPGRADED, None, Some(beacon), None, None) + } + ProxyUpgradeEvent::AdminChanged { previous_admin, new_admin } => { + (ADMIN_CHANGED, None, None, Some(previous_admin), Some(new_admin)) + } + }; + + if let (Some(bn), Some(tx), Some(ti), Some(li)) = + (log.block_number, log.transaction_hash, log.transaction_index, log.log_index) + { + columns.n_rows += 1; + store!(schema, columns, block_number, bn as u32); + store!(schema, columns, block_hash, log.block_hash.map(|bh| bh.to_vec())); + store!(schema, columns, transaction_index, ti as u32); + store!(schema, columns, log_index, li as u32); + store!(schema, columns, transaction_hash, tx.to_vec()); + store!(schema, columns, proxy_address, log.address().to_vec()); + store!(schema, columns, event_name, event_name.to_string()); + store!(schema, columns, implementation, implementation); + store!(schema, columns, beacon, beacon); + store!(schema, columns, previous_admin, previous_admin); + store!(schema, columns, new_admin, new_admin); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy::primitives::{Address, Bytes, B256}; + + /// real `ProxyUpgrades` schema, so `store!` actually evaluates each column + fn schema() -> Table { + Datatype::ProxyUpgrades + .table_schema( + &[U256Type::String], + &ColumnEncoding::Hex, + &None, + &None, + &None, + None, + None, + ) + .expect("ProxyUpgrades has a valid default schema") + } + + /// rpc `Log` carrying the block/tx/index fields `process_proxy_upgrades` needs + fn log_with(topics: Vec, data: Vec) -> Log { + let inner = alloy::primitives::Log::new_unchecked(Address::ZERO, topics, Bytes::from(data)); + Log { + inner, + block_number: Some(1), + transaction_hash: Some(B256::ZERO), + transaction_index: Some(0), + log_index: Some(0), + ..Default::default() + } + } + + /// the two ABI words an `AdminChanged` log carries in its data + fn admin_changed_data(previous: [u8; 20], new: [u8; 20]) -> Vec { + let mut data = Vec::with_capacity(ADMIN_CHANGED_DATA_LEN); + data.extend_from_slice(B256::left_padding_from(&previous).as_slice()); + data.extend_from_slice(B256::left_padding_from(&new).as_slice()); + data + } + + #[test] + fn the_three_signatures_are_distinct() { + let upgraded = ERC1967::Upgraded::SIGNATURE_HASH; + let beacon = ERC1967::BeaconUpgraded::SIGNATURE_HASH; + let admin = ERC1967::AdminChanged::SIGNATURE_HASH; + assert_ne!(upgraded, beacon); + assert_ne!(upgraded, admin); + assert_ne!(beacon, admin); + } + + #[test] + fn admin_changed_is_read_from_data_not_topics() { + // The headline trap: neither argument is indexed. A topics-based read + // would find nothing and index past the end of a one-topic log. + let log = log_with( + vec![ERC1967::AdminChanged::SIGNATURE_HASH], + admin_changed_data([0x11; 20], [0x22; 20]), + ); + assert_eq!(log.topics().len(), 1, "AdminChanged has no indexed argument"); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.event_name, vec![ADMIN_CHANGED.to_string()]); + assert_eq!(columns.previous_admin, vec![Some(vec![0x11u8; 20])]); + assert_eq!(columns.new_admin, vec![Some(vec![0x22u8; 20])]); + // Null, not the zero address: this event carries no implementation. + assert_eq!(columns.implementation, vec![None]); + assert_eq!(columns.beacon, vec![None]); + } + + #[test] + fn upgraded_is_read_from_topic1() { + let implementation = B256::left_padding_from(&[0xab; 20]); + let log = log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH, implementation], vec![]); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.event_name, vec![UPGRADED.to_string()]); + assert_eq!(columns.implementation, vec![Some(vec![0xabu8; 20])]); + assert_eq!(columns.beacon, vec![None]); + assert_eq!(columns.previous_admin, vec![None]); + assert_eq!(columns.new_admin, vec![None]); + } + + #[test] + fn beacon_upgraded_fills_beacon_and_leaves_implementation_null() { + // A beacon proxy's implementation lives on the beacon, not in this log. + // Filling it with the beacon address would make the two indistinguishable. + let beacon = B256::left_padding_from(&[0xcd; 20]); + let log = log_with(vec![ERC1967::BeaconUpgraded::SIGNATURE_HASH, beacon], vec![]); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.event_name, vec![BEACON_UPGRADED.to_string()]); + assert_eq!(columns.beacon, vec![Some(vec![0xcdu8; 20])]); + assert_eq!(columns.implementation, vec![None]); + } + + #[test] + fn upgraded_with_no_topic1_and_no_data_is_skipped_not_panicking() { + // One topic and an empty body names no address in either encoding, so + // there is nothing to report and nothing to panic on. + let malformed = log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH], vec![]); + assert!(!is_proxy_upgrade(&malformed)); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![malformed], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn upgraded_is_read_from_the_data_when_the_argument_is_not_indexed() { + // The zos-lib `AdminUpgradeabilityProxy` form: `Upgraded(address + // implementation)` with no `indexed`. `indexed` is not in the signature + // preimage, so this log carries the same topic0 as the ERC-1967 form + // and the node returns it for the same filter. Reading only topics[1] + // dropped every one of these, USDC's proxy included. + let mut word = vec![0u8; 32]; + word[12..].copy_from_slice(&[0xab; 20]); + let log = log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH], word); + assert!(is_proxy_upgrade(&log)); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.event_name, vec![UPGRADED.to_string()]); + assert_eq!(columns.implementation, vec![Some(vec![0xabu8; 20])]); + assert_eq!(columns.beacon, vec![None]); + } + + #[test] + fn a_data_word_that_is_not_an_address_is_skipped() { + // An `address` is ABI-encoded left-padded with zeros. A non-zero high + // prefix means these 32 bytes are something else that collided with + // this signature, and its low 20 bytes are not an implementation. + let log = log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH], vec![0xff; 32]); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn admin_changed_with_wrong_data_width_is_skipped() { + let short = log_with(vec![ERC1967::AdminChanged::SIGNATURE_HASH], vec![0u8; 32]); + assert!(!is_proxy_upgrade(&short)); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![short], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn indexed_events_tolerate_trailing_data() { + // The argument is in the topic, so unexpected data does not make the + // row ambiguous and must not delete a real upgrade. + let implementation = B256::left_padding_from(&[0x07; 20]); + let log = log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH, implementation], vec![0xff; 32]); + assert!(is_proxy_upgrade(&log)); + + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.implementation, vec![Some(vec![0x07u8; 20])]); + } + + #[test] + fn foreign_topic0_is_skipped() { + let logs = vec![ + log_with(vec![B256::repeat_byte(0xfe), B256::ZERO], vec![]), + log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH, B256::ZERO], vec![]), + ]; + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(logs, &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 1); + assert_eq!(columns.event_name, vec![UPGRADED.to_string()]); + } + + #[test] + fn skips_logs_missing_identity_fields() { + let mut log = log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH, B256::ZERO], vec![]); + log.log_index = None; + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.n_rows, 0); + } + + #[test] + fn a_zero_implementation_is_a_value_not_an_absence() { + // A proxy upgraded to 0x0 is bricked. That must not read as "unknown". + let log = log_with(vec![ERC1967::Upgraded::SIGNATURE_HASH, B256::ZERO], vec![]); + let mut columns = ProxyUpgrades::default(); + process_proxy_upgrades(vec![log], &mut columns, &schema()).unwrap(); + assert_eq!(columns.implementation, vec![Some(vec![0u8; 20])]); + } +} diff --git a/crates/freeze/src/datasets/slots.rs b/crates/freeze/src/datasets/slots.rs index 512b268..4a5be3c 100644 --- a/crates/freeze/src/datasets/slots.rs +++ b/crates/freeze/src/datasets/slots.rs @@ -1,6 +1,7 @@ use crate::*; -use alloy::primitives::{Address, U256}; +use alloy::primitives::{Address, B256, U256}; use polars::prelude::*; +use std::collections::HashMap; /// columns for balances #[triodion_macros::to_df(Datatype::Slots)] @@ -59,12 +60,70 @@ impl CollectByBlock for Slots { let schema = query.schemas.get_schema(&Datatype::Slots)?; process_slot(columns, response, schema) } + + async fn collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, + ) -> R> { + if query.batch_state_reads { + state_override_collect_by_block::(partition, source, query, inner_request_size) + .await + } else { + default_collect_by_block::(partition, source, query, inner_request_size).await + } + } } impl CollectByTransaction for Slots { type Response = (); } +/// Read whole contracts' worth of slots per request instead of one slot per +/// request. +/// +/// Every row of this dataset is a `(block, contract, slot)` triple, and the +/// per-row path spends one `eth_getStorageAt` on each. Grouping by +/// `(block, contract)` and overriding that contract's code with the `SLOAD` +/// extractor turns a group into a single `eth_call` — which is the entire point +/// of the technique: there is no aggregator contract for storage, because a +/// deployed contract cannot read another's slots, but bytecode *injected into* +/// the target can read all of them. +impl StateOverrideBatchable for Slots { + fn reader() -> StateReader { + StateReader::Storage + } + + fn target(params: &Params) -> R
{ + // `Dim::Contract` is aliased to `Dim::Address` for this dataset, so the + // contract under inspection arrives as `address`. + // + // `ethers_address` rather than `Address::from_slice`, which panics on a + // width mismatch — and `--address` is hex-decoded without a length + // check. This runs in the grouping loop, not in a worker task, so a + // panic here would take the whole freeze down instead of one row. As an + // error it makes the row ineligible, and the per-call path reports the + // bad argument exactly as it did before batching existed. + params.ethers_address() + } + + fn input_word(params: &Params) -> R { + // `U256::from_be_slice` PANICS above 32 bytes, and a slot arrives from + // the command line, so an over-long `--slot` would abort a worker task + // rather than report a bad argument. The checked form turns it into an + // error, which the runner treats as "not batchable" and sends down the + // per-call path, where the same value is rejected with a real message. + U256::try_from_be_slice(¶ms.slot()?).ok_or_else(|| err("slot does not fit in 32 bytes")) + } + + fn decode_row(params: &Params, value: B256) -> R { + // `B256::to_vec` is the same 32 big-endian bytes `U256::to_vec_u8` + // produces on the per-row path, so both paths write identical cells. + Ok((params.block_number()? as u32, None, params.address()?, params.slot()?, value.to_vec())) + } +} + fn process_slot(columns: &mut Slots, data: BlockTxAddressOutput, schema: &Table) -> R<()> { let (block, _tx, address, slot, output) = data; columns.n_rows += 1; diff --git a/crates/freeze/src/multi_datasets/log_events.rs b/crates/freeze/src/multi_datasets/log_events.rs index 619b2d7..d0a99bb 100644 --- a/crates/freeze/src/multi_datasets/log_events.rs +++ b/crates/freeze/src/multi_datasets/log_events.rs @@ -62,7 +62,10 @@ use crate::{ erc20_approvals::is_erc20_approval, erc20_transfers::is_erc20_transfer, erc20_wrapper_events::is_wrapper_event, erc721_transfers::is_erc721_transfer, }, - types::{collection::*, rpc_params::fixed_from_slice}, + types::{ + collection::*, + rpc_params::{address_topic_matches, fixed_from_slice, topic_matches}, + }, *, }; use alloy::{ @@ -242,35 +245,6 @@ fn narrow_agreed_topics(topics: &mut [Topic; 4], request: &Params, active: &Hash } } -/// True iff `log`'s topic at `idx` equals `want` (`None` ⇒ unconstrained). -/// -/// For the `--topic0..3` dims only, where the scalar path compares the user's -/// bytes to a topic verbatim. A non-32-byte value cannot equal any topic and so -/// matches nothing here — but it never reaches this point, because -/// `build_union_filter`'s `request.ethers_log_filter()?` already rejected it. -fn topic_matches(log: &Log, idx: usize, want: &Option>) -> bool { - match want { - None => true, - Some(bytes) => log.topics().get(idx).is_some_and(|t| t.as_slice() == bytes.as_slice()), - } -} - -/// Like [`topic_matches`], but for the address-shaped dims (`--from-address`, -/// `--to-address`), which the CLI stores at whatever width the user typed. -/// -/// Compares against the left-padded 32-byte form, which is exactly what the -/// scalar extractors put into their RPC filter via [`address_dim_as_topic`]. A -/// value wider than 32 bytes matches nothing; the scalar path errors on the same -/// input, so neither returns rows. -fn address_topic_matches(log: &Log, idx: usize, want: &Option>) -> bool { - match want { - None => true, - Some(bytes) => { - address_dim_as_topic(bytes).is_some_and(|t| log.topics().get(idx) == Some(&t)) - } - } -} - // --- full by-block predicates: each scalar dataset's shape+signature predicate // (imported from its own module — the single source of truth) plus the // per-position narrowing that dataset applies at the RPC. The union filter @@ -489,6 +463,9 @@ mod tests { multicall: false, multicall_batch_size: 0, multicall_require_success: false, + batch_state_reads: false, + batch_rpc_calls: false, + state_override_batch_size: 0, }) } diff --git a/crates/freeze/src/types/datatypes/scalar.rs b/crates/freeze/src/types/datatypes/scalar.rs index 4a51997..7455e14 100644 --- a/crates/freeze/src/types/datatypes/scalar.rs +++ b/crates/freeze/src/types/datatypes/scalar.rs @@ -5,6 +5,7 @@ use std::collections::HashMap; define_datatypes!( AccessLists, AddressAppearances, + ApprovalsForAll, Authorizations, BalanceDiffs, BalanceReads, @@ -15,16 +16,24 @@ define_datatypes!( CodeReads, Codes, ConsolidationRequests, + ContractInterfaces, Contracts, DepositRequests, + Erc1155Metadata, + Erc1155Transfers, + Erc20Allowances, Erc20Balances, Erc20Metadata, Erc20Supplies, Erc20Transfers, Erc20Approvals, Erc20WrapperEvents, + Erc2612Nonces, + Erc4626Metadata, + Erc4626VaultEvents, Erc721Metadata, Erc721Transfers, + Erc777Transfers, EthCalls, FourByteCounts, GethCalls, @@ -39,6 +48,8 @@ define_datatypes!( NonceDiffs, NonceReads, Nonces, + ProxySlots, + ProxyUpgrades, Slots, StorageDiffs, StorageReads, diff --git a/crates/freeze/src/types/errors.rs b/crates/freeze/src/types/errors.rs index 8cc60de..89bf5a9 100644 --- a/crates/freeze/src/types/errors.rs +++ b/crates/freeze/src/types/errors.rs @@ -176,6 +176,41 @@ pub enum CallOutcome { NodeFailed, } +/// JSON-RPC error code EIP-1474 reserves for `execution reverted`. +const EXECUTION_REVERTED_CODE: i64 = 3; + +/// Lowercase fragments of the EVM's own execution-failure messages. +/// +/// Every one of these is the EVM refusing to finish running *the contract*, so +/// the node did its job and the address has no answer to give. That is a +/// [`CallOutcome::ContractRefused`], and a null is the honest cell for it. +/// +/// Taken from geth's `vm.Err*` set, which reth and erigon reproduce and which +/// providers pass through. The list was four fragments and covered a third of +/// that set; the rest were being classified as node failures and were failing +/// whole chunks. `contract_interfaces` is the most exposed dataset, because it +/// aims `eth_call` at arbitrary addresses holding arbitrary bytecode. +/// +/// Anything not listed here still defaults to [`CallOutcome::NodeFailed`]. That +/// direction is the safe one: an unrecognised error surfaces loudly instead of +/// being laundered into a null. +const EVM_FAILURE_MESSAGES: &[&str] = &[ + "revert", + "invalid opcode", + "invalid jump", + "out of gas", + "stack underflow", + "stack limit reached", + "return data out of bounds", + "max call depth exceeded", + "gas uint64 overflow", + "nonce uint64 overflow", + "write protection", + "invalid code", + "max code size exceeded", + "contract address collision", +]; + impl CollectError { /// Classify this error as a contract-level refusal or a node-level failure. /// @@ -195,15 +230,22 @@ impl CollectError { return CallOutcome::NodeFailed } - // Geth, reth, erigon, nethermind and every provider that proxies them - // report contract-level failures through the message, not the code - // (the code is an un-namespaced -32000 on most of them). Matching on - // the message is unlovely but it is the only portable signal. + // EIP-1474 reserves code 3 for "execution reverted", and geth uses it + // for exactly that, carrying the revert blob in `data`. It is the one + // structured signal available here, so it is checked before the + // message: a provider that rewords the message still sets the code. + if payload.code == EXECUTION_REVERTED_CODE { + return CallOutcome::ContractRefused + } + + // Otherwise the message is the only portable signal. Geth, reth, + // erigon, nethermind and every provider that proxies them report + // contract-level failures through it, not the code (which is an + // un-namespaced -32000 on most of them). Matching on a message is + // unlovely, but there is nothing else to match on. let message = payload.message.to_ascii_lowercase(); - let contract_refused = message.contains("revert") || - message.contains("invalid opcode") || - message.contains("out of gas") || - message.contains("invalid jump"); + let contract_refused = + EVM_FAILURE_MESSAGES.iter().any(|fragment| message.contains(fragment)); if contract_refused { CallOutcome::ContractRefused @@ -275,6 +317,42 @@ mod tests { ); } + #[test] + fn the_reserved_revert_code_is_enough_on_its_own() { + // EIP-1474 reserves 3 for execution-reverted. A provider that rewords + // the message still sets the code, so the code is checked first. + assert_eq!( + error_resp(3, "VM execution error.").call_outcome(), + CallOutcome::ContractRefused + ); + } + + #[test] + fn every_evm_execution_failure_is_a_contract_refusal() { + // The EVM refusing to finish running the contract, in geth's own + // wording. Each of these used to be classified as a node failure and + // failed the whole chunk instead of writing one null row. They arrive + // under -32000, so only the message distinguishes them. + for message in [ + "stack underflow (0 <=> 2)", + "stack limit reached 1024 (1024)", + "return data out of bounds", + "max call depth exceeded", + "gas uint64 overflow", + "nonce uint64 overflow", + "write protection", + "invalid code: must not begin with 0xef", + "max code size exceeded", + "contract address collision", + ] { + assert_eq!( + error_resp(-32000, message).call_outcome(), + CallOutcome::ContractRefused, + "{message}" + ); + } + } + #[test] fn pruned_archive_state_is_a_node_failure() { // This is the regression that motivated the classifier: a non-archive diff --git a/crates/freeze/src/types/mod.rs b/crates/freeze/src/types/mod.rs index fe7e2fa..46c191b 100644 --- a/crates/freeze/src/types/mod.rs +++ b/crates/freeze/src/types/mod.rs @@ -48,6 +48,12 @@ pub mod signatures; /// Multicall3 helpers pub mod multicall; +/// bulk state reads via `eth_call` state overrides +pub mod state_override; + +/// many identical JSON-RPC calls per HTTP request +pub mod rpc_batch; + /// error specifications pub mod errors; /// type specifications for output data formats @@ -76,8 +82,13 @@ pub use dataframes::*; pub use datatypes::*; pub use files::{ColumnEncoding, FileFormat, FileOutput, SubDir}; pub use queries::{Query, QueryLabels, TimeDimension}; +pub use rpc_batch::{rpc_batch_collect_by_block, RpcBatchable, DEFAULT_RPC_BATCH_ROWS}; pub use schemas::{ColumnType, SchemaFunctions, Schemas, Table, U256Type}; pub use sources::{Fetcher, RateLimiter, Source, SourceLabels}; +pub use state_override::{ + override_unavailable, state_override_collect_by_block, OverrideSupport, StateOverrideBatchable, + StateReader, DEFAULT_STATE_OVERRIDE_BATCH_SIZE, SCRATCH_ADDRESS, +}; // pub(crate) use summaries::FreezeSummaryAgg; // pub use summaries::{FreezeChunkSummary, FreezeSummary}; pub use summaries::{print_all_datasets, print_dataset_info, FreezeSummary}; diff --git a/crates/freeze/src/types/multicall.rs b/crates/freeze/src/types/multicall.rs index bb4b20e..fa71cf2 100644 --- a/crates/freeze/src/types/multicall.rs +++ b/crates/freeze/src/types/multicall.rs @@ -356,7 +356,7 @@ where /// Anything else — pruned state, an unsupported method, an auth rejection, a /// rate limit — is a property of the request, not of its length, and halving /// only multiplies the failed requests. -fn batch_may_shrink_to_fit(error: &CollectError) -> bool { +pub(crate) fn batch_may_shrink_to_fit(error: &CollectError) -> bool { let CollectError::ProviderError(rpc_err) = error else { // Not a provider failure at all (a decode mismatch, a short result // slice): a smaller batch is worth one attempt, matching the previous @@ -429,9 +429,20 @@ where Err(e) if current.len() > 1 => return Err(e), + // `--multicall-require-success` asks for the whole batch to fail + // when any inner call reverts, and the arm above honours that for + // batches of two or more. A singleton must fail the same way. + // Demoting it to `D::extract` would not: `extract` folds a + // contract-level refusal into a null by design, so the flag became + // a silent no-op on exactly the shape that produces singletons — + // rows are grouped by block above, so one address over a block + // range gives one row per group and *every* batch is a singleton. + Err(e) if require_success => return Err(e), + Err(_) => { - // Singleton batch failed. Fall through to the per-call path so - // the row is decoded by the dataset's own `extract` — which now + // Singleton batch failed, and the caller did not ask for + // batch-level strictness. Fall through to the per-call path so + // the row is decoded by the dataset's own `extract` — which // classifies the failure itself: a contract-level refusal // becomes a null, a node-level failure propagates. let p = current.into_iter().next().expect("len checked above"); diff --git a/crates/freeze/src/types/queries.rs b/crates/freeze/src/types/queries.rs index ae49680..bfaedfe 100644 --- a/crates/freeze/src/types/queries.rs +++ b/crates/freeze/src/types/queries.rs @@ -34,6 +34,42 @@ pub struct Query { /// [`DEFAULT_MULTICALL_BATCH_SIZE`](crate::DEFAULT_MULTICALL_BATCH_SIZE) /// when constructed via the CLI / Python bindings. pub multicall_batch_size: u32, + /// Read storage slots / balances in bulk through `eth_call` state overrides. + /// + /// When true, datasets that implement + /// [`StateOverrideBatchable`](crate::types::state_override::StateOverrideBatchable) + /// group their rows by `(block, contract)` and read each group with one + /// `eth_call` carrying an extractor-bytecode `code` override, instead of one + /// `eth_getStorageAt` / `eth_getBalance` per row. Any batch that fails — + /// including on an endpoint that rejects or ignores overrides — degrades to + /// the per-row path, so leaving this on cannot change results, only the + /// number of requests it takes to get them. + pub batch_state_reads: bool, + /// Words per state-override extractor call. + /// + /// Only meaningful when [`batch_state_reads`](Self::batch_state_reads) is true. + /// Defaults to + /// [`DEFAULT_STATE_OVERRIDE_BATCH_SIZE`](crate::types::state_override::DEFAULT_STATE_OVERRIDE_BATCH_SIZE) + /// when zero. + pub state_override_batch_size: u32, + /// Send many identical JSON-RPC calls per HTTP request. + /// + /// When true, datasets that implement + /// [`RpcBatchable`](crate::types::rpc_batch::RpcBatchable) pack their rows + /// into JSON-RPC batch envelopes — one HTTP request carrying N embedded + /// calls — instead of one request per row. `blocks`, `nonces` and `codes` + /// take this route. + /// + /// Deliberately separate from + /// [`batch_state_reads`](Self::batch_state_reads). A state override needs a + /// node that honours the third `eth_call` parameter, which not every + /// endpoint does; a JSON-RPC batch needs nothing beyond JSON-RPC itself. An + /// operator who distrusts an endpoint's override support has no reason to + /// give up plain batching as well. + /// + /// A batch that fails falls back to the per-row path, so this changes how + /// many requests the results cost, never the results. + pub batch_rpc_calls: bool, /// When true, mark the entire Multicall3 batch as failed if any inner call reverts. /// /// When false (the default), inner reverts are returned as `None` in diff --git a/crates/freeze/src/types/rpc_batch.rs b/crates/freeze/src/types/rpc_batch.rs new file mode 100644 index 0000000..37b7cfc --- /dev/null +++ b/crates/freeze/src/types/rpc_batch.rs @@ -0,0 +1,185 @@ +//! The universal batching floor: many identical JSON-RPC calls per HTTP request. +//! +//! [State overrides](crate::types::state_override) are the fastest way to read +//! bulk state, and [Multicall3](crate::types::multicall) the fastest way to +//! batch contract calls — but both ask something of the node. Overrides need an +//! endpoint that honours the third `eth_call` parameter; Multicall3 needs the +//! aggregator deployed at the block being read. Plenty of endpoints offer +//! neither, and some methods have no override path at all no matter how +//! cooperative the node is. +//! +//! `eth_getTransactionCount` is the clean example. There is no `NONCE` opcode: +//! the EVM exposes an account's balance (`BALANCE`), its code (`EXTCODECOPY`, +//! `EXTCODESIZE`, `EXTCODEHASH`) and its storage (`SLOAD` from inside it), but +//! nothing reads another account's nonce. No injected bytecode can batch nonce +//! reads, on any node, ever. The only lever left is the transport. +//! +//! JSON-RPC has always allowed an array of requests in one HTTP body, and +//! [`Source::send_batch`] already implements it with the size negotiation real +//! providers demand (OP Mainnet answers `413` above ten calls; Base returns +//! `-32014 "maximum 10 calls in 1 batch"`). It needs nothing from the node +//! beyond JSON-RPC itself, so it is the rung every dataset can stand on. Dedaub +//! measured batched `eth_getStorageAt` at 57× sequential — most of the win, +//! with none of the requirements. +//! +//! This module turns that into an opt-in for `CollectByBlock` datasets, the +//! same shape [`MulticallBatchable`](crate::types::multicall::MulticallBatchable) +//! and [`StateOverrideBatchable`](crate::types::state_override::StateOverrideBatchable) +//! have: say what one row's params look like on the wire and how to read one +//! result back, and [`rpc_batch_collect_by_block`] does the rest. + +use crate::{CollectByBlock, CollectError, Datatype, Params, Partition, Query, Source}; +use polars::prelude::*; +use std::{collections::HashMap, sync::Arc}; +use tokio::sync::mpsc; + +type R = ::core::result::Result; + +/// Rows per HTTP request before the runner starts a second one. +/// +/// [`Source::send_batch`] negotiates downward on its own when a provider +/// refuses a batch this size, so this is a starting point rather than a limit. +/// It matches the batch size the receipt and block helpers already use, which +/// keeps one number to reason about across the tool. +pub const DEFAULT_RPC_BATCH_ROWS: usize = 100; + +/// Trait for `CollectByBlock` datasets whose per-row RPC can be batched at the +/// transport level. +/// +/// Every row must map to the *same* JSON-RPC method — that is what lets the +/// calls share one HTTP body — differing only in parameters. +pub trait RpcBatchable: CollectByBlock { + /// The parameter tuple for one row, serialized as this method's `params`. + /// + /// Typically `(Address, BlockNumberOrTag)`. A tuple serializes to a JSON + /// array, which is the shape JSON-RPC wants. + type Param: alloy::rpc::json_rpc::RpcSend + Send + Sync + 'static; + + /// The result type for one row, deserialized from this method's `result`. + type Item: alloy::rpc::json_rpc::RpcRecv + Send + 'static; + + /// The JSON-RPC method every row in a batch invokes. + fn method() -> &'static str; + + /// Build one row's parameters. + /// + /// # Errors + /// Returns `Err` when the row is missing a required parameter; the runner + /// routes that row to the per-call path rather than failing the partition. + fn param_for_row(params: &Params) -> R; + + /// Turn one row's result into the dataset's response. + /// + /// # Errors + /// Returns `Err` only for unrecoverable decoding bugs. + fn decode_row(params: &Params, item: Self::Item) -> R; + + /// Rows per HTTP request for this dataset. Override when responses are + /// large enough that a hundred of them is an unwieldy body — `eth_getCode` + /// against a partition of large contracts, say. + fn default_rpc_batch_rows() -> usize { + DEFAULT_RPC_BATCH_ROWS + } +} + +/// JSON-RPC-batched collection for `D: RpcBatchable`. +/// +/// Chunks the partition's rows, sends each chunk as one HTTP request, and +/// spawns the chunks concurrently so the node sees the same parallelism it does +/// today with a fraction of the requests. A chunk that fails falls back to +/// [`CollectByBlock::extract`] per row, so a provider that rejects batching +/// outright still produces the same output the per-row path would. +/// +/// Unlike the state-override runner there is no grouping by block or contract: +/// every row carries its own parameters, including its own block, so any rows +/// may share a request. +/// +/// # Errors +/// Returns `Err` only for unrecoverable conditions — mpsc send failure, a +/// dataset `transform` failure, or a per-row fallback that itself failed. +pub async fn rpc_batch_collect_by_block( + partition: Partition, + source: Arc, + query: Arc, + inner_request_size: Option, +) -> R> +where + D: RpcBatchable + Send + Sync + 'static, + D::Response: Send + 'static, +{ + let (sender, receiver) = mpsc::channel(1); + let chain_id = source.chain_id; + let rows_per_request = D::default_rpc_batch_rows().max(1); + + let all_rows = partition.param_sets(inner_request_size)?; + let mut handles = Vec::new(); + + for chunk in all_rows.chunks(rows_per_request) { + let chunk = chunk.to_vec(); + let sender = sender.clone(); + let source = source.clone(); + let query = query.clone(); + let handle = tokio::task::spawn(async move { + let responses = batch_with_fallback::(chunk, &source, query).await?; + for resp in responses { + sender + .send(Ok(resp)) + .await + .map_err(|_| CollectError::CollectError("mpsc send failed".to_string()))?; + } + Ok::<(), CollectError>(()) + }); + handles.push(handle); + } + + drop(sender); + + let columns = ::transform_channel(receiver, &query).await?; + crate::collect_generic::join_partition_handles(handles).await?; + columns.create_dfs(&query.schemas, chain_id) +} + +/// Send one chunk as a batch, falling back to per-row extraction if it fails. +/// +/// There is deliberately no halving here — [`Source::send_batch`] already does +/// that, and it does it better, because it learns the provider's ceiling once +/// and keeps it for the rest of the call. A failure that reaches this function +/// has already survived that negotiation, so it is not about size, and the only +/// useful response left is to stop batching these rows. +async fn batch_with_fallback( + rows: Vec, + source: &Arc, + query: Arc, +) -> R> +where + D: RpcBatchable, +{ + // A row missing a parameter cannot be encoded, so the whole chunk goes to + // the per-row path where the dataset's own accessor reports which field is + // absent — a better error than "batch encoding failed". + let params: Result, CollectError> = rows.iter().map(D::param_for_row).collect(); + + if let Ok(params) = params { + match source.send_batch::(D::method(), ¶ms).await { + Ok(items) if items.len() == rows.len() => { + return rows.iter().zip(items).map(|(p, i)| D::decode_row(p, i)).collect() + } + // A short or long result array means the responses cannot be + // matched to their rows. Falling back re-reads them one at a time + // rather than pairing them up wrongly. + Ok(_) | Err(_) => {} + } + } + + // Every row goes in flight at once rather than being awaited in sequence, + // matching `state_override::per_row`. `Source`'s semaphore is what caps the + // concurrency either way, so demoting a batch costs requests, not + // serialisation. A sequential loop here would make an endpoint that refuses + // batching *slower than before batching existed*: `codes` batches 50 rows + // and `nonces` 100, so the run's in-flight requests would drop from one per + // row to one per chunk. + futures::future::try_join_all( + rows.into_iter().map(|p| D::extract(p, source.clone(), query.clone())), + ) + .await +} diff --git a/crates/freeze/src/types/rpc_params.rs b/crates/freeze/src/types/rpc_params.rs index 61e6a9d..a18d6dc 100644 --- a/crates/freeze/src/types/rpc_params.rs +++ b/crates/freeze/src/types/rpc_params.rs @@ -1,7 +1,7 @@ use crate::{err, CollectError}; use alloy::{ primitives::{Address, BlockNumber, B256}, - rpc::types::{Filter, FilterBlockOption}, + rpc::types::{Filter, FilterBlockOption, Log}, }; /// Left-pad an address-shaped partition dim (`from_address` / `to_address`) into @@ -85,6 +85,16 @@ impl Params { self.call_data.clone().ok_or(err("call_data not specified")) } + /// from address + pub fn from_address(&self) -> Result, CollectError> { + self.from_address.clone().ok_or(err("from_address not specified")) + } + + /// to address + pub fn to_address(&self) -> Result, CollectError> { + self.to_address.clone().ok_or(err("to_address not specified")) + } + // // ethers versions // @@ -118,6 +128,24 @@ impl Params { fixed_from_slice(&self.contract()?, "contract") } + /// ethers from address + /// + /// # Errors + /// Errors if `from_address` is unset, or is not exactly 20 bytes. The CLI + /// hex-decodes `--from-address` without a length check, so the width is + /// validated here rather than panicking in `Address::from_slice`. + pub fn ethers_from_address(&self) -> Result { + fixed_from_slice(&self.from_address()?, "from_address") + } + + /// ethers to address + /// + /// # Errors + /// Errors if `to_address` is unset, or is not exactly 20 bytes. + pub fn ethers_to_address(&self) -> Result { + fixed_from_slice(&self.to_address()?, "to_address") + } + /// log filter /// /// # Errors @@ -150,6 +178,48 @@ impl Params { } } +/// True iff `log`'s topic at `idx` equals `want` (`None` ⇒ unconstrained). +/// +/// For the `--topic0..3` dims only, where the scalar path compares the user's +/// bytes to a topic verbatim. A non-32-byte value cannot equal any topic and so +/// matches nothing here — but it never reaches this point, because +/// `build_union_filter`'s `request.ethers_log_filter()?` already rejected it. +/// True iff `log` was emitted by `want` (`None` ⇒ unconstrained). +/// +/// The `--address` / `--contract` dim, which the by-block path pins at the RPC +/// through `Filter::address`. A value that is not exactly 20 bytes matches +/// nothing here; `ethers_log_filter` errors on the same input, so neither path +/// returns rows for it. +pub(crate) fn log_address_matches(log: &Log, want: &Option>) -> bool { + match want { + None => true, + Some(bytes) => bytes.len() == 20 && log.address() == Address::from_slice(bytes), + } +} + +pub(crate) fn topic_matches(log: &Log, idx: usize, want: &Option>) -> bool { + match want { + None => true, + Some(bytes) => log.topics().get(idx).is_some_and(|t| t.as_slice() == bytes.as_slice()), + } +} + +/// Like [`topic_matches`], but for the address-shaped dims (`--from-address`, +/// `--to-address`), which the CLI stores at whatever width the user typed. +/// +/// Compares against the left-padded 32-byte form, which is exactly what the +/// scalar extractors put into their RPC filter via [`address_dim_as_topic`]. A +/// value wider than 32 bytes matches nothing; the scalar path errors on the same +/// input, so neither returns rows. +pub(crate) fn address_topic_matches(log: &Log, idx: usize, want: &Option>) -> bool { + match want { + None => true, + Some(bytes) => { + address_dim_as_topic(bytes).is_some_and(|t| log.topics().get(idx) == Some(&t)) + } + } +} + /// Convert user-supplied bytes into a fixed-width alloy type, or error. /// /// alloy's `Address::from_slice` / `B256::from_slice` **panic** on a length diff --git a/crates/freeze/src/types/signatures.rs b/crates/freeze/src/types/signatures.rs index ee1b4ba..d063506 100644 --- a/crates/freeze/src/types/signatures.rs +++ b/crates/freeze/src/types/signatures.rs @@ -41,3 +41,138 @@ sol! { event Withdrawal(address indexed src, uint256 wad); } } + +sol! { + /// ERC-1155 multi-token. + /// + /// `TransferBatch` carries two parallel arrays rather than scalars, which + /// is why the transfers dataset explodes one row per token id instead of + /// storing the arrays whole. + /// + /// `ApprovalForAll` is byte-identical to ERC-721's, so a log carrying this + /// signature does not say which of the two standards the contract + /// implements. + contract ERC1155 { + event TransferSingle( + address indexed operator, + address indexed from, + address indexed to, + uint256 id, + uint256 value + ); + event TransferBatch( + address indexed operator, + address indexed from, + address indexed to, + uint256[] ids, + uint256[] values + ); + event ApprovalForAll(address indexed account, address indexed operator, bool approved); + event URI(string value, uint256 indexed id); + + function uri(uint256 id) external view returns (string memory); + function balanceOf(address account, uint256 id) external view returns (uint256); + } +} + +sol! { + /// ERC-165 standard interface detection. + contract ERC165 { + function supportsInterface(bytes4 interfaceId) external view returns (bool); + } +} + +sol! { + /// ERC-1967 proxy storage slots. + /// + /// The slots themselves are constants rather than functions; see the + /// `proxy_slots` dataset. These are the events a compliant proxy emits when + /// one of those slots changes. + contract ERC1967 { + event Upgraded(address indexed implementation); + event AdminChanged(address previousAdmin, address newAdmin); + event BeaconUpgraded(address indexed beacon); + } +} + +sol! { + /// ERC-4626 tokenised vault. + /// + /// `Deposit` here takes four arguments and is a different signature from + /// the two-argument WETH-style `Deposit` in [`ERC20Wrapper`], so the two + /// have different topic0 values and cannot be confused. + contract ERC4626 { + event Deposit( + address indexed sender, + address indexed owner, + uint256 assets, + uint256 shares + ); + event Withdraw( + address indexed sender, + address indexed receiver, + address indexed owner, + uint256 assets, + uint256 shares + ); + + function asset() external view returns (address); + function totalAssets() external view returns (uint256); + function totalSupply() external view returns (uint256); + function convertToAssets(uint256 shares) external view returns (uint256); + } +} + +sol! { + /// ERC-2612 permit: signature-based approvals. + /// + /// `permit` itself emits no event of its own — it emits the ordinary ERC-20 + /// `Approval` — so the only on-chain trace of a permit is the calldata + /// selector and the incremented nonce. + contract ERC2612 { + function nonces(address owner) external view returns (uint256); + function DOMAIN_SEPARATOR() external view returns (bytes32); + function permit( + address owner, + address spender, + uint256 value, + uint256 deadline, + uint8 v, + bytes32 r, + bytes32 s + ) external; + } +} + +sol! { + /// ERC-777 tokens with operators and data payloads. + /// + /// An ERC-777 token also emits a mirrored ERC-20 `Transfer` for every + /// `Sent`, so counting both without care double-counts the same movement. + contract ERC777 { + event Sent( + address indexed operator, + address indexed from, + address indexed to, + uint256 amount, + bytes data, + bytes operatorData + ); + event Minted( + address indexed operator, + address indexed to, + uint256 amount, + bytes data, + bytes operatorData + ); + event Burned( + address indexed operator, + address indexed from, + uint256 amount, + bytes data, + bytes operatorData + ); + event AuthorizedOperator(address indexed operator, address indexed tokenHolder); + event RevokedOperator(address indexed operator, address indexed tokenHolder); + } +} diff --git a/crates/freeze/src/types/sources.rs b/crates/freeze/src/types/sources.rs index cece99a..7ad0044 100644 --- a/crates/freeze/src/types/sources.rs +++ b/crates/freeze/src/types/sources.rs @@ -1,4 +1,4 @@ -use std::sync::Arc; +use std::{collections::HashMap, sync::Arc}; use alloy::{ eips::BlockNumberOrTag, @@ -9,6 +9,7 @@ use alloy::{ Provider, RootProvider, }, rpc::types::{ + state::StateOverride, trace::{ common::TraceResult, geth::{ @@ -36,7 +37,10 @@ use std::num::NonZeroU32; use tokio::sync::{AcquireError, Semaphore, SemaphorePermit}; use crate::{ - types::chains::{RpcBlock, RpcReceipt, RpcTransaction, TriodionProvider}, + types::{ + chains::{RpcBlock, RpcReceipt, RpcTransaction, TriodionProvider}, + state_override::{override_unavailable, OverrideSupport, StateReader}, + }, CollectError, }; @@ -80,6 +84,42 @@ pub struct Source { pub l1_chain_id: Option, /// rpc url passed via `--l1-rpc`. `None` if not configured. pub l1_rpc_url: Option, + /// Consecutive `eth_getStorageValues` misses, for the storage fast path. + /// + /// `eth_getStorageValues` reads many slots across many contracts natively — + /// no state override, no injected bytecode — but it landed in geth + /// v1.17.1 (March 2026) and nothing else ships it yet, so most endpoints + /// answer `-32601`. Worse, a load-balanced pool answers *inconsistently*: + /// the same URL served this method on one request and returned "Method not + /// found" on the next, because the pool holds a mix of node versions. + /// + /// So support cannot be probed once and cached either way. A positive is + /// not durable (the next request may hit an older backend) and a negative + /// is not either (the next may hit a newer one). Instead this counts + /// *consecutive* misses: the fast path is tried while the count is below + /// `STORAGE_VALUES_MISS_LIMIT`, a success resets it to zero, and an + /// endpoint that genuinely lacks the method stops being asked after a + /// handful of tries. A mixed pool keeps using it whenever it works. + pub storage_values_misses: Arc, + /// Whether this endpoint honours the `eth_call` state-override set. + /// + /// Shared by every clone of this `Source`, so the discovery cost is paid + /// once per run rather than once per chunk. See + /// [`OverrideSupport`](crate::types::state_override::OverrideSupport). + /// + /// A hard latch, unlike the consecutive-miss counter next door. The two + /// answer different questions. + /// [`storage_values_misses`](Self::storage_values_misses) tracks whether a + /// *method exists* on the backend a request happens to land on, which a + /// load-balanced pool of mixed geth versions genuinely answers both ways. + /// State-override support is not per-backend in that way: `eth_call` has + /// taken the third parameter since geth 1.9 (2019), so a refusal today is a + /// provider policy — a plan tier, a proxy that strips the argument — and is + /// uniform and permanent for the run. And the evidence is decisive rather + /// than circumstantial: a sentinel that did not read back proves the + /// override was dropped, where a single `-32601` proves only where the + /// request landed. + pub state_override_support: Arc, /// Optional consensus-layer access, for the beacon-chain datasets. /// /// Populated when `--beacon-rpc ` is passed. `None` for @@ -145,9 +185,9 @@ impl Source { /// Send one logical batch of identical calls, splitting it as needed. /// - /// Starts at [`DEFAULT_RPC_BATCH_SIZE`] calls per HTTP request. If the + /// Starts at `DEFAULT_RPC_BATCH_SIZE` calls per HTTP request. If the /// provider rejects a batch *because of its size* — and only then, see - /// [`batch_too_large`] — the size is halved and the remaining work retried, + /// `batch_too_large` — the size is halved and the remaining work retried, /// down to one call per request. The chosen size persists for the rest of /// this call, so a 200-receipt block against a ten-call provider pays the /// discovery cost once rather than per batch. @@ -159,7 +199,7 @@ impl Source { /// # Errors /// Any transport failure that is not a size complaint, and a size complaint /// that persists at one call per request. - async fn send_batch(&self, method: &'static str, params: &[P]) -> Result> + pub async fn send_batch(&self, method: &'static str, params: &[P]) -> Result> where P: alloy::rpc::json_rpc::RpcSend, T: alloy::rpc::json_rpc::RpcRecv, @@ -393,6 +433,8 @@ impl Source { l1_provider: None, l1_chain_id: None, l1_rpc_url: None, + storage_values_misses: Arc::new(std::sync::atomic::AtomicU32::new(0)), + state_override_support: Arc::new(OverrideSupport::default()), beacon: None, }; @@ -833,10 +875,35 @@ impl Source { address: Address, call_data: Vec, block_number: BlockNumber, + ) -> Result { + self.call_with_gas_limit(address, call_data, block_number, None).await + } + + /// Return output data of a contract call, refusing to spend more than + /// `gas_limit` on it. + /// + /// With `None` the node applies its own cap, which is generous — geth + /// defaults `eth_call` to 50,000,000 gas. That is the right default when + /// the callee is known, and the wrong one when it is not: probing an + /// arbitrary address runs whatever fallback function that address happens + /// to have, and a fallback that loops until it runs out of gas turns one + /// row into a node timeout that fails its whole chunk. A cap converts that + /// into an ordinary out-of-gas revert, which [`crate::contract_read`] + /// already folds into a null for the one row that caused it. + /// + /// The cap is a ceiling, not a reservation: a call that needs less spends + /// less, so a low limit costs a conforming contract nothing. + pub async fn call_with_gas_limit( + &self, + address: Address, + call_data: Vec, + block_number: BlockNumber, + gas_limit: Option, ) -> Result { let transaction = TransactionRequest { to: Some(address.into()), input: TransactionInput::new(call_data.into()), + gas: gas_limit, ..Default::default() }; let _permit = self.permit_request().await; @@ -845,6 +912,340 @@ impl Source { ) } + /// Return output data of a contract call, with a state-override set applied. + /// + /// The override is the third `eth_call` parameter: a per-address + /// `{balance, nonce, code, state, stateDiff}` map applied to a scratch copy + /// of the state before execution. [`crate::types::state_override`] uses it + /// to run extractor bytecode in a real contract's storage context, which is + /// how one request answers thousands of storage slots. + /// + /// # Errors + /// Propagates the provider error unchanged so callers can classify it — + /// an endpoint that rejects overrides outright, one that is missing archive + /// state, and one complaining about payload size all need different + /// responses, and stringifying here would erase the difference. + pub async fn call_with_overrides( + &self, + address: Address, + call_data: Vec, + overrides: StateOverride, + block_number: BlockNumber, + ) -> Result { + let transaction = TransactionRequest { + to: Some(address.into()), + input: TransactionInput::new(call_data.into()), + ..Default::default() + }; + let _permit = self.permit_request().await; + Self::map_err( + self.provider + .call(WithOtherFields::new(transaction)) + .overrides(overrides) + .block(block_number.into()) + .await, + ) + } + + /// Consecutive `eth_getStorageValues` misses after which the fast path is + /// abandoned for the rest of the process. + /// + /// Three, not one: a single `-32601` on a load-balanced pool means "this + /// request reached an older backend", not "this endpoint lacks the method". + /// Three in a row against a pool that actually supports it is unlikely, and + /// against one that does not it costs three cheap failures total. + const STORAGE_VALUES_MISS_LIMIT: u32 = 3; + + /// Whether the native storage fast path is still worth attempting. + pub fn storage_values_worth_trying(&self) -> bool { + self.storage_values_misses.load(std::sync::atomic::Ordering::Relaxed) < + Self::STORAGE_VALUES_MISS_LIMIT + } + + /// Read storage slots for one or more contracts with `eth_getStorageValues`. + /// + /// The method geth added in v1.17.1 does natively what the extractor does by + /// injection: many slots, across many contracts, in one round trip. Where it + /// exists it is strictly preferable — no `code` override, so none of the + /// override failure modes apply, and no dependence on the endpoint honouring + /// a parameter it may quietly drop. + /// + /// Values come back in the order their keys were sent, per address. The + /// server caps a request at 1024 slots summed across every address and + /// answers `-38026 "too many slots (max 1024)"` above it; callers must chunk. + /// + /// # Errors + /// Returns the provider error unchanged. `-32601` means this request reached + /// a backend without the method — see + /// [`storage_values_misses`](Self::storage_values_misses) for why that is + /// counted rather than cached. + pub async fn get_storage_values( + &self, + requests: &HashMap>, + block_number: BlockNumber, + ) -> Result>> { + let _permit = self.permit_request().await; + let params = (requests, BlockNumberOrTag::Number(block_number)); + let result: Result>> = + Self::map_err(self.provider.client().request("eth_getStorageValues", params).await); + match &result { + Ok(_) => self.storage_values_misses.store(0, std::sync::atomic::Ordering::Relaxed), + + // Throttling says nothing about whether the method exists, and + // giving up on the fast path here would be perverse: the per-row + // path it demotes to sends *more* requests to the endpoint that + // just asked for fewer. + Err(e) if is_retry_error(e) => {} + + // Everything else counts. `-32601` is the documented answer, but it + // is not the only one that means "stop asking": a gateway that + // rejects the method by prose, a `-38026` slot cap lower than ours, + // or a reply this client cannot deserialize all fail identically on + // every future request. Counting only `-32601` left those + // uncounted, so the fast path was re-attempted once per chunk for + // the whole run — measured at one wasted request per chunk against + // a node answering a shape we do not parse. + Err(_) => { + self.storage_values_misses.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + } + } + result + } + + /// Read many storage slots of one contract in a single request. + /// + /// Replaces `slots.len()` round trips of [`Self::get_storage_at`] with one + /// `eth_call`. Measured against a public mainnet endpoint: 500 slots went + /// from 500 requests / 1.17s to 1 request / 0.09s, and the per-call path + /// started returning HTTP 429 at ~2000 requests where the batched path + /// never did — against rate-limited providers the request-count reduction + /// matters more than the latency. + /// + /// Returns values in `slots` order, one per slot. + /// + /// # Errors + /// Returns [`CollectError::CollectError`] if the response does not prove + /// the override took effect — a wrong return length, or a sentinel slot + /// that did not read back as written. See [`StateReader::decode_response`]. + /// Provider errors propagate unchanged. + pub async fn get_storage_at_many( + &self, + address: Address, + slots: &[U256], + block_number: BlockNumber, + ) -> Result> { + let reader = StateReader::Storage; + if let Some(why) = reader.refuses_target(address) { + return Err(CollectError::CollectError(format!( + "cannot batch-read storage at {address}: {why}" + ))); + } + let (call_data, overrides, to) = reader.request(address, slots); + let output = + self.call_with_overrides(to, call_data.into(), overrides, block_number).await?; + reader.decode_response(&output, slots.len()) + } + + /// Read many storage slots of one contract, batched when the endpoint + /// allows it and one at a time when it does not. + /// + /// This is the entry point for a dataset whose *single row* needs several + /// slots — `proxy_slots` reads three ERC-1967 slots per address. Datasets + /// whose row is one slot go through + /// [`state_override_collect_by_block`](crate::state_override_collect_by_block) + /// instead, which can also batch across rows. + /// + /// Both paths consult the same [`Self::state_override_support`] memo, so an + /// endpoint that cannot do overrides is discovered once for the whole run + /// no matter which shape of caller discovers it. + /// + /// Values come back in `slots` order, one per slot, always. + /// + /// Chunks at [`DEFAULT_STATE_OVERRIDE_BATCH_SIZE`](crate::DEFAULT_STATE_OVERRIDE_BATCH_SIZE) + /// and not at `--state-override-batch-size`: a `Source` has no `Query`, and + /// threading one through for this would put a CLI type in the transport + /// layer. The knob still governs the runner, which is where large batches + /// actually arise — a caller here is reading a handful of slots for one row. + /// + /// # Errors + /// Only what the per-slot path itself returns. A batch failure is a routing + /// decision, not an outcome: the per-slot read is the ground truth and it + /// is about to run. A node that cannot serve the block still errors — from + /// [`Self::get_storage_at`], with that call's error, exactly as it did + /// before batching existed. + pub async fn read_storage_slots( + &self, + address: Address, + slots: &[U256], + block_number: BlockNumber, + ) -> Result> { + let batch_size = crate::DEFAULT_STATE_OVERRIDE_BATCH_SIZE as usize; + let mut out = Vec::with_capacity(slots.len()); + for chunk in slots.chunks(batch_size.max(1)) { + match self.try_batched(|| self.get_storage_at_many(address, chunk, block_number)).await + { + Some(values) => out.extend(values), + None => { + let reads = + chunk.iter().map(|slot| self.get_storage_at(address, *slot, block_number)); + for word in futures::future::try_join_all(reads).await? { + out.push(B256::from(word.to_be_bytes::<32>())); + } + } + } + } + Ok(out) + } + + /// Read many account balances, batched when the endpoint allows it and one + /// at a time when it does not. See [`Self::read_storage_slots`] for the + /// shape and the error rule. + /// + /// # Errors + /// As [`Self::read_storage_slots`], from [`Self::get_balance`]. + pub async fn read_balances( + &self, + addresses: &[Address], + block_number: BlockNumber, + ) -> Result> { + let batch_size = crate::DEFAULT_STATE_OVERRIDE_BATCH_SIZE as usize; + let mut out = Vec::with_capacity(addresses.len()); + for chunk in addresses.chunks(batch_size.max(1)) { + match self.try_batched(|| self.get_balances_many(chunk, block_number)).await { + Some(values) => out.extend(values), + None => { + let reads = chunk.iter().map(|a| self.get_balance(*a, block_number)); + out.extend(futures::future::try_join_all(reads).await?); + } + } + } + Ok(out) + } + + // There is deliberately no `read_code_sizes` / `read_code_hashes` here. + // Neither has an exact per-call equivalent to fall back to: + // `eth_getCode(...).len()` matches `EXTCODESIZE`, but `EXTCODEHASH` returns + // zero for an account that does not exist and `keccak256("")` for one that + // exists with no code, and `eth_getCode` answers `0x` for both. A fallback + // that silently collapsed those two would be exactly the kind of + // plausible-but-wrong value this module exists to prevent, so a caller who + // wants those readers must handle the demotion knowing what it costs. + + /// One batched attempt against the run-level verdict, or `None` when the + /// caller should read one at a time. + /// + /// Holds the whole policy for the many-words-per-row callers, so + /// [`Self::read_storage_slots`] and [`Self::read_balances`] do not each + /// re-derive it: skip when ruled out, serialise the run's first attempt, + /// record what came back, and swallow the batch error — because the caller + /// is about to get the same rows from the path that cannot be wrong. + async fn try_batched(&self, attempt: F) -> Option> + where + F: FnOnce() -> Fut, + Fut: std::future::Future>>, + { + let support = &self.state_override_support; + if !support.worth_attempting() { + return None + } + let gate = support.gate_first_attempt().await; + if !support.worth_attempting() { + return None + } + let result = attempt().await; + drop(gate); + match result { + Ok(values) => { + support.record_supported(); + Some(values) + } + Err(e) => { + if override_unavailable(&e) { + support.rule_out(); + } + None + } + } + } + + /// Read many account balances in a single request. + /// + /// Unlike [`Self::get_storage_at_many`] this touches no real contract: + /// `BALANCE` takes its account from the stack, so the extractor runs at + /// [`crate::types::state_override::SCRATCH_ADDRESS`] and the calldata names + /// the accounts. One request therefore covers arbitrarily many unrelated + /// addresses. + /// + /// Returns balances in `addresses` order. + /// + /// # Errors + /// Returns [`CollectError::CollectError`] if any address is one the reader + /// refuses (see [`StateReader::refuses_target`]) or if the return length + /// shows the override was ignored. + pub async fn get_balances_many( + &self, + addresses: &[Address], + block_number: BlockNumber, + ) -> Result> { + let words = self.read_accounts_many(StateReader::Balance, addresses, block_number).await?; + Ok(words.into_iter().map(|w| U256::from_be_bytes(w.0)).collect()) + } + + /// Read many account code sizes in a single request. See + /// [`Self::get_balances_many`] for the shape and the error conditions. + /// + /// # Errors + /// As [`Self::get_balances_many`]. + pub async fn get_code_sizes_many( + &self, + addresses: &[Address], + block_number: BlockNumber, + ) -> Result> { + let words = self.read_accounts_many(StateReader::CodeSize, addresses, block_number).await?; + // EXTCODESIZE is bounded by EIP-170's 24576-byte limit, so the word + // always fits a u64; saturate rather than wrap if a chain ever lifts it. + Ok(words.into_iter().map(|w| U256::from_be_bytes(w.0).saturating_to::()).collect()) + } + + /// Read many account code hashes in a single request. See + /// [`Self::get_balances_many`] for the shape and the error conditions. + /// + /// # Errors + /// As [`Self::get_balances_many`]. + pub async fn get_code_hashes_many( + &self, + addresses: &[Address], + block_number: BlockNumber, + ) -> Result> { + self.read_accounts_many(StateReader::CodeHash, addresses, block_number).await + } + + /// Shared body of the account-reading batch helpers. + async fn read_accounts_many( + &self, + reader: StateReader, + addresses: &[Address], + block_number: BlockNumber, + ) -> Result> { + // The extractor executes *at* the scratch address, so that one account + // reads back as our injected code rather than as the chain has it. The + // collision is absurdly unlikely and entirely silent, so it is refused + // here and the caller routes the row through the per-call path. + if let Some((clash, why)) = + addresses.iter().find_map(|a| reader.refuses_target(*a).map(|w| (a, w))) + { + return Err(CollectError::CollectError(format!( + "cannot batch-read {clash} with the {reader:?} extractor: {why}" + ))); + } + let words: Vec = + addresses.iter().map(|a| U256::from_be_slice(a.as_slice())).collect(); + let (call_data, overrides, to) = reader.request(Address::ZERO, &words); + let output = + self.call_with_overrides(to, call_data.into(), overrides, block_number).await?; + reader.decode_response(&output, addresses.len()) + } + /// Return output data of a contract call pub async fn trace_call2( &self, @@ -1361,6 +1762,17 @@ fn parse_geth_diff_object(map: serde_json::Map) -> Re Ok(DiffMode { pre, post }) } +/// Whether an error is the endpoint asking for less traffic rather than +/// refusing the request. +/// +/// A throttle is the one failure that must not be counted against a fast path: +/// every fast path demotes to something that sends *more* requests, so treating +/// a `429` as evidence of missing support makes the throttling worse. +fn is_retry_error(error: &CollectError) -> bool { + let CollectError::ProviderError(rpc_err) = error else { return false }; + rpc_err.as_error_resp().is_some_and(|payload| payload.is_retry_err()) +} + #[cfg(test)] mod batch_tests { use super::*; diff --git a/crates/freeze/src/types/state_override.rs b/crates/freeze/src/types/state_override.rs new file mode 100644 index 0000000..5a4d046 --- /dev/null +++ b/crates/freeze/src/types/state_override.rs @@ -0,0 +1,1891 @@ +//! Bulk state reads through the `eth_call` state-override set. +//! +//! [Multicall3](crate::types::multicall) batches `eth_call`s, but there is no +//! aggregator for `eth_getStorageAt`, `eth_getBalance` or `eth_getCode`: a +//! contract cannot read another contract's storage, and the account-reading +//! opcodes are not exposed through any deployed helper. So a dataset like +//! `slots` or `balances` issues one HTTP request per output row, and a scan of +//! a few thousand slots becomes a few thousand round trips — enough to be +//! rate-limited off most public endpoints before it finishes. +//! +//! Geth's `eth_call` takes an optional third parameter, the *state-override +//! set*: a per-address `{balance, nonce, code, state, stateDiff}` map applied +//! to a scratch copy of the state before the call executes. Overriding `code` +//! lets us run bytecode of our choosing *in the storage context of a real +//! contract*, at a real historical block, without a fork. This is the trick +//! [Dedaub](https://github.com/Dedaub/storage-extractor) published: replace the +//! target's code with a loop that treats calldata as a bare array of 32-byte +//! slot keys, `SLOAD`s each one, and returns the values contiguously. One +//! request then answers thousands of slots. +//! +//! The same loop shape reads accounts rather than storage if the opcode in the +//! middle changes: `BALANCE`, `EXTCODEHASH` and `EXTCODESIZE` all take an +//! address from the stack and push one word. Those three read *arbitrary* +//! accounts, so their extractor does not need to be injected at any particular +//! address — it runs from a scratch address and the calldata names the accounts. +//! See [`StateReader`]. +//! +//! # Why every extractor here avoids `PUSH0` +//! +//! Dedaub's published bytecode opens with `PUSH0` (`0x5f`), which saves two +//! bytes. That is correct for their use case — reading *current* storage — and +//! wrong for ours. `eth_call` at a historical block executes under *that +//! block's* fork rules, and `PUSH0` only became valid at Shanghai (EIP-3855, +//! mainnet block 17_034_870, April 2023). Below that height the call does not +//! return wrong data, it fails outright — measured against two independent +//! archive endpoints at block 15_000_000: +//! +//! ```text +//! geth: "invalid opcode: PUSH0" +//! reth: "EVM error: NotActivated" +//! ``` +//! +//! An archive-extraction tool spends most of its life below that height, and +//! many L2s enabled Shanghai far later than mainnet did. The `PUSH1 0x00` form +//! is two bytes longer and costs two more gas per *call* (not per element), and +//! it works at every block on every EVM chain. That is the trade this module +//! takes: the fork-independent form always, so there is no fork detection to +//! get wrong and no chain-specific table to keep current. +//! +//! # Verifying the override actually happened +//! +//! Some endpoints accept the third `eth_call` parameter and ignore it. That is +//! the one failure this technique must never wave through: with the override +//! dropped, the calldata is delivered to the *real* contract, which interprets +//! a list of slot keys as a function call. For most contracts that reverts or +//! falls through to a fallback returning empty — but "most" is not "all", and a +//! contract that happened to return 32 bytes would hand back a plausible, +//! entirely fabricated value. +//! +//! [`StateReader::decode_response`] therefore requires the return to be +//! *exactly* `32 * n` bytes, and — for the storage reader — requires the last +//! word to be [`SENTINEL_VALUE`](crate::types::state_override::SENTINEL_VALUE). The extractor +//! returns one word per calldata word by construction, so the length is a positive proof that our +//! code, not the contract's, produced the answer; the sentinel proves it a second time, +//! in band. Anything else is an error, never a partial result. +//! +//! # The policy: on by default, demoted once, never fatal +//! +//! The batch path is attempted automatically rather than hidden behind a flag, +//! because a flag nobody passes is a speedup nobody gets and the thing a flag +//! would guard against is already guarded: every failure lands on the per-row +//! path, which is the code that ran before this module existed. The only real +//! question is what the automatic attempt *costs* against an endpoint that +//! cannot serve it, and the answer has to be "one request for the whole run". +//! +//! [`OverrideSupport`] is what makes that true. It is a run-level memo, shared +//! by every clone of a [`Source`](crate::Source): +//! +//! * The run's first batch attempt is serialised ([`OverrideSupport::gate_first_attempt`]), so an +//! endpoint that cannot do overrides is discovered by one request rather than by every chunk that +//! happened to be in flight at startup. +//! * A failure that says *this endpoint does not apply state overrides* ([`override_unavailable`]) +//! latches [`OverrideSupport::rule_out`], and every later group skips the attempt entirely. +//! * Every other failure demotes only the batch in hand, and leaves the memo alone — a timeout says +//! nothing about whether the endpoint honours overrides. +//! +//! Without the memo the cost is not one wasted request but thousands: the +//! runner halves a failed batch, so a 1000-row group that fails for a reason +//! halving cannot fix spends 1999 doomed `eth_call`s before reaching the +//! per-row path — and then does it again for the next group, and the next. +//! Against a rate-limited endpoint that is not "slower", it is a run that dies +//! of 429s having produced nothing. +//! +//! The memo can only ever *suppress* batching. No state it can reach causes a +//! row to be written from a response [`StateReader::decode_response`] did not +//! verify. + +use crate::{CollectError, Params}; +use alloy::{ + primitives::{Address, Bytes, B256, U256}, + rpc::types::state::{AccountOverride, StateOverride}, +}; +use std::sync::atomic::{AtomicU8, Ordering}; +use tokio::sync::{Semaphore, SemaphorePermit}; + +/// Which piece of state one extractor call reads. +/// +/// All four share a single 25-byte loop that differs in exactly one opcode, at +/// offset `0x0b`. The loop walks calldata one 32-byte word at a time, applies +/// that opcode to the word, and writes the result to memory at the same offset +/// it read from — so the return is a parallel array: input word `i` produces +/// output word `i`. +/// +/// The distinction that matters operationally is *where the code must live*: +/// +/// * [`Storage`](Self::Storage) uses `SLOAD`, which only ever reads the storage of the contract +/// currently executing. Its bytecode must be injected **at the address whose storage is wanted**, +/// and one call therefore covers one address. Several addresses need either several calls or a +/// fan-out through Multicall3 / `eth_simulateV1`. +/// * [`Balance`](Self::Balance), [`CodeHash`](Self::CodeHash) and [`CodeSize`](Self::CodeSize) use +/// `BALANCE` / `EXTCODEHASH` / `EXTCODESIZE`, which take the account from the stack. Their +/// bytecode is injected **at a scratch address** ([`SCRATCH_ADDRESS`]) and the calldata names the +/// accounts, so one call covers arbitrarily many unrelated addresses. No real contract is touched +/// at all. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum StateReader { + /// `SLOAD` — storage slots of the injected-at address. Calldata words are + /// slot keys. + Storage, + /// `BALANCE` — account balances. Calldata words are right-aligned addresses. + Balance, + /// `EXTCODEHASH` — account code hashes. Calldata words are right-aligned + /// addresses. An account with no code hashes to `keccak256("")`; an account + /// that does not exist hashes to zero. + CodeHash, + /// `EXTCODESIZE` — account code lengths in bytes. Calldata words are + /// right-aligned addresses. + CodeSize, +} + +/// Address the account-reading extractors are injected at. +/// +/// `BALANCE` / `EXTCODEHASH` / `EXTCODESIZE` read whatever account the stack +/// names, so the code can run anywhere; it just must not clobber an account the +/// same call needs to read truthfully. This is a fixed, obviously-synthetic +/// address in the range reserved by no chain and occupied on none: overriding +/// it cannot mask a real contract's code, and it is not a precompile (those +/// live at `0x01..=0x11`), so no client's precompile dispatch is disturbed. +/// +/// It is deliberately *not* derived from the query. A constant keeps the +/// override set byte-identical between calls, which keeps request bodies +/// comparable in logs and caches. +pub const SCRATCH_ADDRESS: Address = Address::new([ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x03, 0x13, 0x37, +]); + +/// Storage slot the storage extractor reads as an in-band proof of the override. +/// +/// The exact-length check in [`StateReader::decode_response`] catches an +/// endpoint that ignored the override *whenever the real contract answers with +/// a different number of bytes*, which in practice is almost always — most +/// contracts revert or return empty for calldata that is a bare list of slot +/// keys. "Almost always" is not a guarantee, though, and the failure it misses +/// is the worst one available: plausible values, right shape, entirely +/// fabricated, written to a Parquet file that looks correct forever. +/// +/// So the storage path also proves the override positively. Alongside `code` +/// it sets `stateDiff { SENTINEL_SLOT: SENTINEL_VALUE }` — a single-slot +/// override, which leaves every other slot exactly as the chain has it — and +/// appends `SENTINEL_SLOT` as the final calldata word. Getting `SENTINEL_VALUE` +/// back in the final returned word means our bytecode ran, the storage override +/// was applied, and the words came back in the order they were sent. All three +/// are the properties the decoder depends on. +/// +/// The slot is a fixed nothing-up-my-sleeve constant rather than a per-process +/// random value. That is enough for the realistic threat — a provider that +/// silently drops the third `eth_call` parameter — which cannot conspire to +/// return this exact word. It is *not* proof against a contract deliberately +/// written to mimic the sentinel while overrides are being ignored; such a +/// contract would have to be one the user chose to query, on such an endpoint. +/// A per-process random sentinel would close even that, at the cost of a `rand` +/// dependency this crate does not otherwise carry. +/// +/// Chance of colliding with a slot a caller actually wants: 2^-256. Doing it on +/// purpose costs one copied constant, though, because this is public and +/// documented — so the collision is guarded rather than argued away. A row that +/// asks for this slot takes the per-call path, which reads it with no override +/// in sight. See [`StateReader::shadows_sentinel`]. +pub const SENTINEL_SLOT: B256 = B256::new([ + 0x7d, 0xed, 0xa1, 0x75, 0x0b, 0x1e, 0xe5, 0x10, 0x7e, 0x5c, 0xa1, 0x1e, 0xd0, 0x0b, 0xa5, 0xe5, + 0x10, 0xad, 0xed, 0xc0, 0xde, 0x5e, 0x77, 0x1e, 0x51, 0x0b, 0x5e, 0x77, 0x1e, 0x1a, 0xbe, 0x15, +]); + +/// Value [`SENTINEL_SLOT`] is overridden to, and must read back as. +pub const SENTINEL_VALUE: B256 = B256::new([ + 0x0e, 0xda, 0xc0, 0xde, 0x15, 0xfa, 0x11, 0xed, 0xba, 0xd0, 0x5e, 0x77, 0x1e, 0x0b, 0xad, 0xc0, + 0xde, 0xfa, 0xce, 0x0f, 0xf1, 0xce, 0x0d, 0xd5, 0x0d, 0xdb, 0xa1, 0x1a, 0xdd, 0x1e, 0x5a, 0xfe, +]); + +/// Addresses below this are treated as reserved and never used as an extractor +/// target. +/// +/// Overriding the `code` of a precompile is undefined across clients: some +/// dispatch the precompile before consulting the override and run the builtin +/// anyway, which would return a well-formed answer that is not our extractor's. +/// Ethereum's precompiles occupy `0x01..=0x11` and Arbitrum adds ArbOS builtins +/// around `0x64..=0x6f`, but the set grows with every fork and differs per +/// chain, so this reserves the whole low range instead of tracking the exact +/// membership. No real contract lives there; a row naming one falls back to the +/// per-call path, which reads it correctly. +const RESERVED_ADDRESS_CEILING: u64 = 0x1_0000; + +/// Offset of the swappable opcode inside [`TEMPLATE`]. +const OPCODE_OFFSET: usize = 0x0b; + +/// The extractor loop, with a placeholder at [`OPCODE_OFFSET`]. +/// +/// ```text +/// [00] PUSH1 0x00 cursor = 0 stack: [cursor] +/// [02] JUMPDEST <-- loop +/// [03] DUP1 stack: [cursor, cursor] +/// [04] CALLDATASIZE stack: [size, cursor, cursor] +/// [05] EQ cursor == size? stack: [done?, cursor] +/// [06] PUSH1 0x14 -> done +/// [08] JUMPI stack: [cursor] +/// [09] DUP1 stack: [cursor, cursor] +/// [0a] CALLDATALOAD word at cursor stack: [word, cursor] +/// [0b] SLOAD / BALANCE / ... stack: [value, cursor] +/// [0c] DUP2 stack: [cursor, value, cursor] +/// [0d] MSTORE mem[cursor] = value stack: [cursor] +/// [0e] PUSH1 0x20 +/// [10] ADD cursor += 32 stack: [cursor] +/// [11] PUSH1 0x02 -> loop +/// [13] JUMP +/// [14] JUMPDEST <-- done +/// [15] CALLDATASIZE return length == input length +/// [16] PUSH1 0x00 +/// [18] RETURN +/// ``` +/// +/// The cursor is reused as the memory write offset, which is what makes the +/// loop this short: the value for input word `i` lands at memory offset `32*i`, +/// so the return is already in input order with no bookkeeping. +/// +/// Termination is `EQ`, not `LT`, so calldata whose length is not a multiple of +/// 32 would step straight past the end and loop until it ran out of gas. +/// [`StateReader::calldata`] is the only constructor of this calldata and it +/// only ever emits whole words, so the condition cannot arise; the encoder is +/// the guard, not the bytecode. +const TEMPLATE: [u8; 25] = [ + 0x60, 0x00, // [00] PUSH1 0x00 + 0x5b, // [02] JUMPDEST <- loop + 0x80, // [03] DUP1 + 0x36, // [04] CALLDATASIZE + 0x14, // [05] EQ + 0x60, 0x14, // [06] PUSH1 0x14 -> done + 0x57, // [08] JUMPI + 0x80, // [09] DUP1 + 0x35, // [0a] CALLDATALOAD + 0x54, // [0b] , patched per StateReader + 0x81, // [0c] DUP2 + 0x52, // [0d] MSTORE + 0x60, 0x20, // [0e] PUSH1 0x20 + 0x01, // [10] ADD + 0x60, 0x02, // [11] PUSH1 0x02 -> loop + 0x56, // [13] JUMP + 0x5b, // [14] JUMPDEST <- done + 0x36, // [15] CALLDATASIZE + 0x60, 0x00, // [16] PUSH1 0x00 + 0xf3, // [18] RETURN +]; + +impl StateReader { + /// The opcode this reader substitutes at [`OPCODE_OFFSET`]. + const fn opcode(self) -> u8 { + match self { + Self::Storage => 0x54, // SLOAD + Self::Balance => 0x31, // BALANCE + Self::CodeHash => 0x3f, // EXTCODEHASH + Self::CodeSize => 0x3b, // EXTCODESIZE + } + } + + /// The extractor bytecode for this reader. + pub fn bytecode(self) -> Bytes { + let mut code = TEMPLATE; + code[OPCODE_OFFSET] = self.opcode(); + Bytes::from(code.to_vec()) + } + + /// Whether the bytecode must be injected at the address being read. + /// + /// True only for [`Storage`](Self::Storage): `SLOAD` has no address + /// operand. The account readers run from [`SCRATCH_ADDRESS`]. + pub const fn injects_at_target(self) -> bool { + matches!(self, Self::Storage) + } + + /// Encode a batch of inputs as the extractor's calldata. + /// + /// The layout is a bare contiguous array of 32-byte big-endian words: no + /// selector, no ABI header, no length prefix. The loop derives the element + /// count from `CALLDATASIZE`, so the encoding carries no redundancy — which + /// is why a slot costs 32 calldata bytes here and ~68 through any + /// ABI-encoded aggregator. + /// + /// For the account readers the caller passes addresses widened to `U256`, + /// which right-aligns them in the word. The address-taking opcodes mask + /// their operand to the low 160 bits, so the upper 12 bytes are ignored; + /// this encoder zeroes them regardless so request bodies stay comparable. + pub fn calldata(self, inputs: impl IntoIterator) -> Bytes { + let mut out: Vec = Vec::new(); + for word in inputs { + out.extend_from_slice(&word.to_be_bytes::<32>()); + } + Bytes::from(out) + } + + /// Whether this reader proves its override with an in-band sentinel. + /// + /// Only the storage reader can: the sentinel is a storage slot, and only + /// [`Storage`](Self::Storage) reads storage. The account readers do not + /// need one — they are sent to [`SCRATCH_ADDRESS`], which holds no code, so + /// an endpoint that dropped the override would be calling a codeless + /// account and would return empty data, which the length check rejects. + pub const fn uses_sentinel(self) -> bool { + self.injects_at_target() + } + + /// Build the state-override set that makes an extractor call work. + /// + /// `target` is the contract whose storage is wanted for + /// [`Storage`](Self::Storage), and is ignored by the account readers, which + /// always override [`SCRATCH_ADDRESS`]. + /// + /// `code` is always set. `state_diff` is set only for the storage reader, + /// and only for [`SENTINEL_SLOT`]. It is deliberately never `state`: `state` + /// replaces the account's *entire* storage, so a call made with it would + /// return a well-formed file of zeros — the exact silent corruption this + /// module exists to prevent. `state_diff` overrides the named slot and + /// leaves every other one as the chain has it. + pub fn overrides(self, target: Address) -> StateOverride { + let account = AccountOverride { + code: Some(self.bytecode()), + state_diff: self + .uses_sentinel() + .then(|| [(SENTINEL_SLOT, SENTINEL_VALUE)].into_iter().collect()), + ..Default::default() + }; + let mut over = StateOverride::default(); + over.insert(self.call_target(target), account); + over + } + + /// Build the complete request for one batch: calldata, overrides, and the + /// address to send it to. + /// + /// The sentinel word, when this reader uses one, is appended *last* so the + /// caller's inputs keep their indices and + /// [`Self::decode_response`] can check the tail without disturbing the + /// parallel-array property the decoder relies on. + pub fn request(self, target: Address, inputs: &[U256]) -> (Bytes, StateOverride, Address) { + let words = inputs + .iter() + .copied() + .chain(self.uses_sentinel().then(|| U256::from_be_bytes(SENTINEL_SLOT.0))); + (self.calldata(words), self.overrides(target), self.call_target(target)) + } + + /// Why this target cannot be used with this reader, if it cannot. + /// + /// Two cases, both of which must fall back to the per-call path rather than + /// producing a value: + /// + /// * A reserved low address (see `RESERVED_ADDRESS_CEILING`) — overriding a precompile's code + /// is undefined across clients. + /// * The scratch address, for the account readers (see [`Self::shadows_scratch`]). + pub fn refuses_target(self, target: Address) -> Option<&'static str> { + if self.shadows_scratch(target) { + return Some("address is where the extractor itself runs"); + } + if self.injects_at_target() && is_reserved_address(target) { + return Some("address is in the reserved precompile range"); + } + None + } + + /// The address an extractor call should be sent `to`. + /// + /// The same address the code is injected at: for + /// [`Storage`](Self::Storage) that is the contract under inspection, so + /// `SLOAD` resolves against its storage; for the account readers it is + /// [`SCRATCH_ADDRESS`]. + pub const fn call_target(self, target: Address) -> Address { + if self.injects_at_target() { + target + } else { + SCRATCH_ADDRESS + } + } + + /// Whether this reader would report on its own injected code rather than + /// on the chain, for `address`. + /// + /// The account readers run *from* [`SCRATCH_ADDRESS`], so that address is + /// the one account in the world whose code they cannot see truthfully: + /// `EXTCODESIZE(SCRATCH_ADDRESS)` returns 25 — the extractor's own length — + /// and `EXTCODEHASH` returns `keccak256(extractor)`. Measured against a live + /// node, which is how this was found: the batch answered correctly for two + /// real accounts and reported `0x19` for the scratch address. + /// + /// The collision is vanishingly unlikely and completely silent, which is + /// exactly the combination worth a guard. Callers must route a colliding + /// row through the per-call path instead of the batch. + /// + /// [`Storage`](Self::Storage) is never affected: its code is injected at + /// the target, so the address it reads and the address it runs at are the + /// same by construction, and `SLOAD` reads storage — which the `code` + /// override leaves untouched. + pub fn shadows_scratch(self, address: Address) -> bool { + !self.injects_at_target() && address == SCRATCH_ADDRESS + } + + /// Whether this reader would report its own sentinel rather than the chain, + /// for the row whose calldata word is `word`. + /// + /// The mirror of [`Self::shadows_scratch`] on the storage side, and it + /// closes the same kind of hole. The storage reader writes + /// `SENTINEL_VALUE` into [`SENTINEL_SLOT`] through `stateDiff`, so + /// [`SENTINEL_SLOT`] is the one slot in the world it cannot read + /// truthfully: a caller who asks for it is handed our own marker back. + /// + /// Nothing downstream can catch that. The return is exactly `32 * n` bytes, + /// the tail word is `SENTINEL_VALUE`, and the length check and the sentinel + /// check both pass — so the row would be written to Parquet as though it + /// were measured. This is precisely the silent corruption the sentinel + /// exists to prevent, arriving through the sentinel itself. + /// + /// A collision by chance is a `2^-256` event. A collision on purpose is one + /// copied constant away, because [`SENTINEL_SLOT`] is public and + /// documented. Callers must route a colliding row through the per-call + /// path, where `eth_getStorageAt` reads the slot with no override in sight. + pub fn shadows_sentinel(self, word: U256) -> bool { + self.uses_sentinel() && word == U256::from_be_bytes(SENTINEL_SLOT.0) + } + + /// Split an extractor return into one word per input, verifying the + /// override actually took effect. + /// + /// # Errors + /// Returns [`CollectError::CollectError`] unless the return is *exactly* + /// `32 * expected` bytes. + /// + /// The strictness is the safety property, not fussiness. A shorter return + /// is the signature of an endpoint that accepted the third `eth_call` + /// parameter and discarded it: the calldata reached the real contract, + /// which answered as best it could. Truncating or zero-filling to the + /// expected length would turn that into silently fabricated storage values + /// written to a Parquet file, indistinguishable from real ones forever + /// after. An exact-length check makes the override's effect observable in + /// the response itself, so the caller can demote to the per-call path + /// instead of trusting it. + #[allow(clippy::doc_markdown)] + pub fn decode_response(self, data: &[u8], n_inputs: usize) -> Result, CollectError> { + let expected = n_inputs + usize::from(self.uses_sentinel()); + let want = expected.saturating_mul(32); + if data.len() != want { + return Err(CollectError::CollectError(format!( + "state-override extractor returned {} bytes, expected {} ({} words): \ + {IGNORED_MARKER}", + data.len(), + want, + expected, + ))); + } + // The length is now known to be an exact multiple of 32, so the + // remainder `as_chunks` returns is empty by construction. + let mut words: Vec = + data.as_chunks::<32>().0.iter().copied().map(B256::new).collect(); + + if self.uses_sentinel() { + // Positive proof, not an inference from the length: our bytecode + // ran, the storage override was applied, and the words came back in + // the order they were sent. A contract answering by accident cannot + // produce this word. + let sentinel = words.pop().expect("length checked above includes the sentinel"); + if sentinel != SENTINEL_VALUE { + return Err(CollectError::CollectError(format!( + "{SENTINEL_MARKER}: read {sentinel} from the sentinel slot, expected \ + {SENTINEL_VALUE}. The endpoint accepted the eth_call state override and did \ + not apply it, so these values did not come from the extractor", + ))); + } + } + + Ok(words) + } +} + +// --------------------------------------------------------------------------- +// Knowing when to stop asking +// --------------------------------------------------------------------------- + +/// The tail of the message [`StateReader::decode_response`] produces when the +/// return length proves the extractor never ran. +/// +/// Matched by [`override_unavailable`] rather than restated there. This is the +/// one place in the crate where an error is classified by its text, and having +/// the producer and the classifier share a `const` is what keeps them from +/// drifting apart the next time somebody rewords the message. +const IGNORED_MARKER: &str = "the endpoint ignored the eth_call state override"; + +/// As [`IGNORED_MARKER`], for the sentinel check. +const SENTINEL_MARKER: &str = "state-override sentinel mismatch"; + +/// Whether this failure means the endpoint will not apply state overrides *at +/// all*, as opposed to failing this one batch. +/// +/// Two shapes count: +/// +/// * The endpoint took the third `eth_call` parameter and discarded it. The calldata then reached +/// the real contract, whose answer had the wrong length or the wrong sentinel — the two things +/// [`StateReader::decode_response`] refuses. This is the dangerous case, the one where the values +/// would have looked plausible, and it is why that decoder must never pad or truncate. +/// * The endpoint refused the parameter outright. Clients word that differently, so the list below +/// is a sample of what has been observed, not a specification. +/// +/// Misclassifying costs requests, never rows. An unrecognised failure still +/// demotes *this* batch to the per-row path (see the ladder in +/// `extractor_batch_with_fallback`); all it fails to do is stop the next +/// batch from trying. Correctness never rests on this function — only the +/// request count does. +pub fn override_unavailable(error: &CollectError) -> bool { + let message = match error { + CollectError::CollectError(message) => message.clone(), + CollectError::ProviderError(rpc_err) => match rpc_err.as_error_resp() { + Some(payload) => payload.message.to_string(), + // A transport failure — a dropped connection, a timeout — says + // nothing about whether the endpoint supports overrides. Ruling the + // feature out on a blip would cost the whole run its speedup. + None => return false, + }, + _ => return false, + }; + let message = message.to_ascii_lowercase(); + + [ + IGNORED_MARKER, + SENTINEL_MARKER, + // geth built without the override argument: "too many arguments, want at most 2". + "too many arguments", + // Hosted endpoints that gate the feature by tier, and clients that + // simply have not implemented it. Our own override set is one account + // with 25 bytes of code and one slot, so "state override" appearing in + // a size complaint is not a thing that happens. + "state override", + "stateoverride", + "state_override", + ] + .iter() + .any(|needle| message.contains(needle)) +} + +/// Run-level memory of whether this endpoint honours the `eth_call` +/// state-override set. +/// +/// One of these lives on [`Source`](crate::Source) and is shared by every clone +/// of it, so the discovery cost is paid once per run rather than once per +/// chunk. See the module docs for why that distinction is the difference +/// between "slower" and "dies of 429s". +/// +/// It is deliberately not a per-`(chain, url)` global: a `Source` is the +/// endpoint, and a process that opens a second one against a different +/// provider gets its own verdict rather than inheriting a stranger's. +#[derive(Debug)] +pub struct OverrideSupport { + /// `UNKNOWN` / `SUPPORTED` / `RULED_OUT`. + state: AtomicU8, + /// A single permit, held across the run's first attempt. + first_attempt: Semaphore, + /// Consecutive batches that failed without saying why, reset by a success. + /// + /// [`rule_out`](Self::rule_out) fires on a message that names the override + /// as the problem. An endpoint that refuses the third `eth_call` parameter + /// *below* the JSON-RPC layer never produces one — a stripping proxy + /// answering 400, an HTML error page, a closed connection all arrive as + /// bare transport errors, which `override_unavailable` does not claim and + /// `batch_may_shrink_to_fit` does claim. Those land on the halving rung, so + /// a 1000-row chunk spends 1999 doomed calls reaching singletons, nothing + /// latches, and the next chunk does it again. + /// + /// This counter is the backstop for that shape: enough silent failures in + /// a row is itself evidence, whatever the wire said. + consecutive_failures: AtomicU8, +} + +/// Nothing has been tried yet: attempt, and record what comes back. +const UNKNOWN: u8 = 0; +/// A batch came back verified. Attempt freely, with no gate. +const SUPPORTED: u8 = 1; +/// The endpoint does not apply overrides. Never attempt again this run. +const RULED_OUT: u8 = 2; + +impl Default for OverrideSupport { + fn default() -> Self { + Self { + state: AtomicU8::new(UNKNOWN), + first_attempt: Semaphore::new(1), + consecutive_failures: AtomicU8::new(0), + } + } +} + +impl OverrideSupport { + /// Whether a batch attempt is still worth a request. + /// + /// False only once [`Self::rule_out`] has fired. Callers re-check this + /// after [`Self::gate_first_attempt`] returns, because the answer can + /// change while they wait at the gate. + pub fn worth_attempting(&self) -> bool { + self.state.load(Ordering::Relaxed) != RULED_OUT + } + + /// Record that a batch came back verified. + /// + /// Cannot resurrect a ruled-out endpoint. A batch that was already in + /// flight when another one ruled the feature out must not undo that + /// ruling — it would put the run straight back into the halving cascade + /// the ruling exists to prevent. + pub fn record_supported(&self) { + // A verified batch means the last failures were not about the override. + self.consecutive_failures.store(0, Ordering::Relaxed); + let _ = + self.state.compare_exchange(UNKNOWN, SUPPORTED, Ordering::Relaxed, Ordering::Relaxed); + } + + /// Latch "this endpoint does not apply state overrides" for the rest of the + /// run. Irreversible by design; see [`Self::record_supported`]. + pub fn rule_out(&self) { + self.state.store(RULED_OUT, Ordering::Relaxed); + } + + /// Silent batch failures in a row before the fast path is abandoned. + /// + /// Three, matching `Source::STORAGE_VALUES_MISS_LIMIT` and for the same + /// reason: one failure against a load-balanced pool means "this request + /// landed badly", not "this endpoint cannot do it". Three in a row against + /// a pool that can is unlikely; against one that cannot it costs three + /// halving cascades total instead of one per chunk for the whole run. + const SILENT_FAILURE_LIMIT: u8 = 3; + + /// Record a batch that failed without naming the override as the cause, + /// and rule the endpoint out once enough have. + /// + /// Ruling out on this evidence can be wrong, and being wrong is cheap: the + /// rows are read by the per-row path either way, so the only cost is the + /// run losing a speedup. Not ruling out is the expensive mistake. + pub fn record_silent_failure(&self) { + let previous = self.consecutive_failures.fetch_add(1, Ordering::Relaxed); + if previous + 1 >= Self::SILENT_FAILURE_LIMIT { + self.rule_out() + } + } + + /// Serialise the run's first batch attempt. + /// + /// Returns `Some` only while the verdict is still `UNKNOWN` — the caller + /// holds the guard across its first request and drops it as soon as that + /// request returns, whatever it returned. Everyone else waits, then finds + /// the answer already recorded. + /// + /// This is worth one round-trip of startup latency: without it, every chunk + /// that starts before the first answer arrives pays the full doomed + /// cascade, and at the default concurrency that is a hundred of them. + pub async fn gate_first_attempt(&self) -> Option> { + if self.state.load(Ordering::Relaxed) != UNKNOWN { + return None + } + // `acquire` fails only on a closed semaphore, and this one is never + // closed. Treating the impossible error as "no gate" keeps the batch + // correct (just unserialised) rather than dropping the row. + let permit = self.first_attempt.acquire().await.ok()?; + // Somebody else may have settled the question while we waited. + (self.state.load(Ordering::Relaxed) == UNKNOWN).then_some(permit) + } +} + +/// Whether `address` is in the range reserved for precompiles and chain +/// builtins. See `RESERVED_ADDRESS_CEILING`. +fn is_reserved_address(address: Address) -> bool { + let bytes = address.into_array(); + // Everything above the low 8 bytes must be zero for the address to be small. + bytes[..12].iter().all(|b| *b == 0) && + u64::from_be_bytes(bytes[12..].try_into().expect("8 bytes")) < RESERVED_ADDRESS_CEILING +} + +// --------------------------------------------------------------------------- +// Batched collection +// --------------------------------------------------------------------------- + +/// Words per extractor call before the runner splits the batch. +/// +/// Chosen from measurement against a public mainnet endpoint rather than from +/// the gas ceiling. The gas ceiling is generous. Per slot: +/// +/// ```text +/// cold SLOAD 2100 +/// loop body, JUMPDEST..JUMP, SLOAD excluded 51 +/// calldata, 32 bytes, worst case all non-zero 512 +/// memory, amortised: MSTORE writes at 32*i, so 3 (+ n^2/512 across the call) +/// ---- +/// 2666 +/// ``` +/// +/// which puts a 50M-gas cap at ~18.5k slots — close to the ~20k measured +/// directly against a post-Berlin block, where not every calldata byte is +/// non-zero. +/// +/// Note the 51: it is the *whole* body, not the 22 gas that reaches the +/// loop-exit test. `JUMPI` is the halfway point of the iteration, not the end +/// of it, and everything after it — the `SLOAD`'s operand handling, the store, +/// the cursor advance and the jump back — costs another 29. +/// +/// The binding limits in practice arrive earlier and are about *payload*, not +/// gas: +/// +/// ```text +/// 100 slots -> 0.1s +/// 1_000 slots -> 0.2s +/// 5_000 slots -> 0.8s +/// 10_000 slots -> 1.5s (320 KB request, 320 KB response) +/// 20_000 slots -> HTTP 503 +/// ``` +/// +/// So the useful range tops out around 10k on that endpoint and the marginal +/// return past ~1000 is poor: ten parallel 1000-word calls finish sooner than +/// one 10k-word call, and they degrade gracefully when a provider caps request +/// bodies. 1000 words is a 32 KB request, comfortably inside every body limit +/// we know of, and still cuts the request count by three orders of magnitude. +/// Raise it with `--state-override-batch-size` when pointing at your own node. +pub const DEFAULT_STATE_OVERRIDE_BATCH_SIZE: u32 = 1000; + +/// Smallest group worth reading through an extractor. +/// +/// The whole win is requests-per-row, and at one row a group costs exactly one +/// request either way — so the override buys nothing and still pays for itself: +/// a 25-byte `code` override plus a sentinel slot in the request body, and an +/// EVM execution on the server where a plain state read would do. Two rows is +/// where one request starts replacing two. +/// +/// Rows below the threshold are not dropped; they take the per-call path, the +/// same one they would take on an endpoint without override support. +const MIN_BATCH_GROUP: usize = 2; + +/// Trait for `CollectByBlock` datasets that can be read through a state-override +/// extractor. +/// +/// The shape mirrors [`MulticallBatchable`](crate::types::multicall::MulticallBatchable): +/// a dataset says how one row maps to one calldata word and how one returned +/// word maps back to a row, and [`state_override_collect_by_block`] does the +/// grouping, batching, splitting and fallback. +/// +/// The difference from Multicall3 is what a batch may contain. Multicall3 sends +/// heterogeneous calls, so a batch is just "rows at the same block". An +/// extractor call runs one piece of bytecode with one override set, so a batch +/// is "rows at the same block *and* the same [`Self::target`]". For +/// [`StateReader::Storage`] the target is the contract being read, which is +/// what makes a slot scan group per contract; for the account readers every row +/// shares [`Address::ZERO`] as a target, so a whole block's addresses batch +/// together. +pub trait StateOverrideBatchable: crate::CollectByBlock { + /// Which extractor this dataset reads through. + fn reader() -> StateReader; + + /// The address whose code is overridden for this row. + /// + /// For [`StateReader::Storage`] this is the contract being read — rows only + /// batch together when it matches. The account readers ignore it and should + /// return [`Address::ZERO`] so every row in a block lands in one batch. + /// + /// # Errors + /// Returns `Err` when the row is missing the parameter, which routes it to + /// the per-call path rather than failing the partition. + fn target(params: &Params) -> crate::R
; + + /// The 32-byte word this row contributes to the extractor's calldata: a + /// storage slot key, or a right-aligned address. + /// + /// # Errors + /// As [`Self::target`]. + fn input_word(params: &Params) -> crate::R; + + /// Build this row's response from the word the extractor returned for it. + /// + /// # Errors + /// Returns `Err` only for unrecoverable encoding bugs — the extractor + /// cannot report a per-element failure, so there is no revert to map here. + fn decode_row(params: &Params, value: B256) -> crate::R; + + /// Words per call for this dataset. Override when a row's read is unusually + /// expensive; `Query::state_override_batch_size` takes precedence when set. + fn default_state_override_batch_size() -> u32 { + DEFAULT_STATE_OVERRIDE_BATCH_SIZE + } +} + +/// Whether a row would read back part of our own override instead of the chain. +/// +/// Each reader has exactly one such blind spot, and it is the same blind spot +/// in both cases: the one input for which the override we install *is* the +/// answer. Neither is detectable downstream — both produce a full-length +/// response with a correct sentinel — so both must be caught here, before the +/// row is ever put in a batch. +/// +/// * For the account readers the calldata word *is* the address, and the address they run at is +/// [`SCRATCH_ADDRESS`], which wears the extractor's own code (see +/// [`StateReader::shadows_scratch`]). +/// * For the storage reader the calldata word is a slot key, and the one slot it writes is +/// [`SENTINEL_SLOT`] (see [`StateReader::shadows_sentinel`]). +/// +/// A colliding row is not dropped. It takes the per-call path, which reads it +/// with no override in sight and so answers it correctly. +fn row_shadows_override(reader: StateReader, word: U256) -> bool { + if reader.injects_at_target() { + return reader.shadows_sentinel(word); + } + let bytes = word.to_be_bytes::<32>(); + reader.shadows_scratch(Address::from_slice(&bytes[12..])) +} + +/// Split `rows` into at most `batch_size`-sized chunks that are all within one +/// row of each other. +/// +/// `slice::chunks` puts the whole remainder in the final chunk, so a group of +/// `batch_size * k + 1` rows ends in a chunk of exactly one. A chunk of one +/// takes the full override path — a code override, a `stateDiff`, two calldata +/// words and a server-side EVM execution — to answer a single slot that +/// `eth_getStorageAt` answers in the same one request. That is the trade +/// [`MIN_BATCH_GROUP`] refuses at the group level, reappearing at the chunk +/// boundary. +/// +/// Spreading the remainder one row per chunk instead makes every chunk either +/// `len / n_chunks` or one more than that, so the smallest chunk is as large as +/// an even split allows. It cannot be avoided entirely: three rows at +/// `batch_size` two must still be split `2 + 1`. +fn even_chunks(rows: &[T], batch_size: usize) -> Vec<&[T]> { + let batch_size = batch_size.max(1); + let n_chunks = rows.len().div_ceil(batch_size).max(1); + let base = rows.len() / n_chunks; + let remainder = rows.len() % n_chunks; + + let mut out = Vec::with_capacity(n_chunks); + let mut start = 0; + for i in 0..n_chunks { + // The first `remainder` chunks take one extra row each, which is what + // keeps every chunk within one of every other. + let len = base + usize::from(i < remainder); + out.push(&rows[start..start + len]); + start += len; + } + out +} + +/// State-override-batched collection for `D: StateOverrideBatchable`. +/// +/// Groups the partition's rows by `(block, target)`, sends one `eth_call` per +/// [`DEFAULT_STATE_OVERRIDE_BATCH_SIZE`]-word chunk, and halves a chunk that +/// fails for a size-shaped reason — reusing the same classifier the Multicall3 +/// runner uses, for the same reason: a node that cannot serve the block at all +/// fails identically at every batch size, so splitting only multiplies the +/// traffic against a node that already said no. +/// +/// A chunk that has shrunk to one row falls through to +/// [`CollectByBlock::extract`](crate::CollectByBlock::extract), so the dataset's +/// own per-call path decides that row's outcome. That is what makes the whole +/// feature safe to leave on by default: an endpoint that rejects state +/// overrides, or silently ignores them (caught by [`StateReader::decode_response`]'s +/// length check), degrades to exactly today's behaviour instead of producing +/// wrong data or an empty file. +/// +/// # Errors +/// Returns `Err` only for unrecoverable conditions — mpsc send failure, a +/// dataset `transform` failure, or a per-call fallback that itself failed. +pub async fn state_override_collect_by_block( + partition: crate::Partition, + source: std::sync::Arc, + query: std::sync::Arc, + inner_request_size: Option, +) -> crate::R> +where + D: StateOverrideBatchable + Send + Sync + 'static, + D::Response: Send + 'static, +{ + use crate::{CollectByBlock, CollectError}; + use std::collections::HashMap; + + let (sender, receiver) = tokio::sync::mpsc::channel(1); + let chain_id = source.chain_id; + let reader = D::reader(); + + let batch_size = if query.state_override_batch_size > 0 { + query.state_override_batch_size + } else { + D::default_state_override_batch_size() + } + .max(1) as usize; + + // Group by (block, target). A row that cannot supply either — or that names + // a part of our own override rather than the chain, the scratch address for + // the account readers and the sentinel slot for storage — is ineligible and + // goes down the per-call path, so the partition's schema is preserved even + // when part of it cannot be batched. + let mut groups: HashMap<(u64, Address), Vec> = HashMap::new(); + let mut ineligible: Vec = Vec::new(); + for p in partition.param_sets(inner_request_size)? { + match (p.block_number, D::target(&p), D::input_word(&p)) { + (Some(block), Ok(target), Ok(word)) + if !row_shadows_override(reader, word) && + reader.refuses_target(target).is_none() => + { + groups.entry((block, target)).or_default().push(p) + } + _ => ineligible.push(p), + } + } + + // A group too small to pay for itself joins the per-call rows rather than + // sending an override that saves no requests. + groups.retain(|_, rows| { + if rows.len() < MIN_BATCH_GROUP { + ineligible.append(rows); + false + } else { + true + } + }); + + // Flatten to the unit of work — one `(block, target, rows)` chunk — so the + // native reader can be offered several contracts at once. The extractor + // cannot be: injected code runs at one address and reads only that + // address's storage, which is why the grouping above is per contract. + let mut work: Vec<(u64, Address, Vec)> = Vec::new(); + for ((block, target), rows) in groups { + for chunk in even_chunks(&rows, batch_size) { + work.push((block, target, chunk.to_vec())); + } + } + let merge = reader.injects_at_target() && source.storage_values_worth_trying(); + + let mut handles = Vec::new(); + for items in plan_native_chunks(work, merge) { + let sender = sender.clone(); + let source = source.clone(); + let query = query.clone(); + let handle = tokio::task::spawn(async move { + let responses = native_chunk_with_fallback::(items, &source, query).await?; + for resp in responses { + sender + .send(Ok(resp)) + .await + .map_err(|_| CollectError::CollectError("mpsc send failed".to_string()))?; + } + Ok::<(), CollectError>(()) + }); + handles.push(handle); + } + + for p in ineligible { + let sender = sender.clone(); + let source = source.clone(); + let query = query.clone(); + let handle = tokio::task::spawn(async move { + let resp = D::extract(p, source, query).await?; + sender + .send(Ok(resp)) + .await + .map_err(|_| CollectError::CollectError("mpsc send failed".to_string()))?; + Ok::<(), CollectError>(()) + }); + handles.push(handle); + } + + drop(sender); + + let columns = ::transform_channel(receiver, &query).await?; + crate::collect_generic::join_partition_handles(handles).await?; + columns.create_dfs(&query.schemas, chain_id) +} + +/// Get every row of one batch out, batched if the endpoint allows it and +/// row-by-row if it does not. +/// +/// The ladder, in order: +/// +/// 1. **Ruled out already.** [`OverrideSupport`] says this endpoint does not apply overrides, so +/// spend no request finding out again — straight to the per-row path. +/// 2. **Verified batch.** [`StateReader::decode_response`] accepted the length and the sentinel, so +/// the rows come from it and the endpoint is recorded as supporting overrides. +/// 3. **The endpoint does not do overrides** ([`override_unavailable`]). Latch that for the run and +/// read every outstanding row — this batch *and* whatever is still on the stack — one at a time. +/// No batch size fixes this, so halving would only multiply doomed requests. +/// 4. **Size-shaped failure, more than one row.** Halve it. This is the one failure a smaller +/// request can actually fix. +/// 5. **Anything else.** Read these rows one at a time. +/// +/// Rung 5 is why this runner cannot turn a working per-row query into a failed +/// one. It replaces an earlier `return Err(e)`, which meant that an endpoint +/// rejecting the override argument in words this module did not recognise +/// failed the whole partition — the one outcome a pure speed change must never +/// produce. A node that genuinely cannot serve the block still fails, but it +/// fails from the per-row call, with the per-row error, exactly as it did +/// before this module existed. +/// +/// # Errors +/// Only what the dataset's own [`extract`](crate::CollectByBlock::extract) +/// returns. A batch failure is a routing decision, never an outcome. +async fn extractor_batch_with_fallback( + block: u64, + target: Address, + batch: Vec, + source: &std::sync::Arc, + query: std::sync::Arc, +) -> crate::R> +where + D: StateOverrideBatchable, +{ + use crate::types::multicall::batch_may_shrink_to_fit; + + let support = &source.state_override_support; + if !support.worth_attempting() { + return per_row::(batch, source, &query).await + } + + // The run's first attempt is serialised so that discovering an endpoint + // cannot do this costs one request, not one per chunk in flight. + let mut gate = support.gate_first_attempt().await; + if !support.worth_attempting() { + // Another task ruled the endpoint out while this one waited. Release + // the permit before the fallback, not after: tasks parked in + // `first_attempt.acquire()` only wake on release, so holding it across + // a whole per-row batch would stall every one of them for the length + // of that batch. The gate exists so nothing waits on it longer than a + // single request. + drop(gate.take()); + return per_row::(batch, source, &query).await + } + + let mut stack: Vec> = vec![batch]; + let mut out: Vec = Vec::new(); + while let Some(current) = stack.pop() { + // The verdict can flip mid-cascade — by this task's own silent-failure + // count, or by another task's. Draining here is what bounds the cost: + // without it a 1000-row chunk keeps halving to singletons after the + // endpoint has already been ruled out, which is the 1999-doomed-call + // run this backstop exists to prevent. + if !support.worth_attempting() { + let mut remaining = current; + for pending in std::mem::take(&mut stack) { + remaining.extend(pending); + } + out.extend(per_row::(remaining, source, &query).await?); + return Ok(out) + } + + let result = extractor_batch::(block, target, ¤t, source).await; + // Whatever that said, the run now has evidence. Release the gate before + // interpreting it so the other chunks are not held behind a halving + // cascade that only concerns this one. + drop(gate.take()); + + match result { + Ok(responses) => { + support.record_supported(); + out.extend(responses) + } + + Err(ref e) if override_unavailable(e) => { + support.rule_out(); + let mut remaining = current; + for pending in std::mem::take(&mut stack) { + remaining.extend(pending); + } + out.extend(per_row::(remaining, source, &query).await?); + return Ok(out) + } + + Err(ref e) if current.len() > 1 && batch_may_shrink_to_fit(e) => { + support.record_silent_failure(); + let mid = current.len() / 2; + let mut left = current; + let right = left.split_off(mid); + // Push right first so left is popped next — keeps produced rows + // in roughly input order. The frame is sorted by schema at the + // end regardless, so this is legibility, not correctness. + stack.push(right); + stack.push(left); + } + + Err(_) => { + support.record_silent_failure(); + out.extend(per_row::(current, source, &query).await?) + } + } + } + Ok(out) +} + +/// Read rows the way the dataset always has: one RPC per row. +/// +/// Every row is put in flight at once rather than awaited in sequence, so +/// `Source`'s semaphore is what caps the concurrency — the same thing that caps +/// it when the per-row collector spawns a task per row. Demoting a batch +/// therefore costs requests, not serialisation: a run against an endpoint +/// without override support takes as long as it took before this module +/// existed, not `n` times longer. +/// +/// # Errors +/// The first per-row failure, as the per-row collector would have reported it. +async fn per_row( + rows: Vec, + source: &std::sync::Arc, + query: &std::sync::Arc, +) -> crate::R> +where + D: StateOverrideBatchable, +{ + futures::future::try_join_all( + rows.into_iter().map(|p| D::extract(p, source.clone(), query.clone())), + ) + .await +} + +/// One extractor call for one `(block, target)` batch. +async fn extractor_batch( + block: u64, + target: Address, + batch: &[Params], + source: &std::sync::Arc, +) -> crate::R> +where + D: StateOverrideBatchable, +{ + let reader = D::reader(); + let mut words: Vec = Vec::with_capacity(batch.len()); + for p in batch { + words.push(D::input_word(p)?); + } + + // The native reader is tried in `native_storage_batch`, before this + // function is reached, and across every contract at the block rather than + // one at a time. Retrying it here would spend a second request to be told + // the same thing. + let (call_data, overrides, to) = reader.request(target, &words); + let output = source.call_with_overrides(to, call_data.into(), overrides, block).await?; + let values = reader.decode_response(&output, batch.len())?; + + batch.iter().zip(values).map(|(p, v)| D::decode_row(p, v)).collect() +} + +/// Slots per `eth_getStorageValues` request. +/// +/// geth answers `-38026 "too many slots (max 1024)"` above this, and the limit +/// counts every slot in the request — across all the contracts named in it, not +/// per contract. It is one budget for the whole map. +const NATIVE_STORAGE_MAX_SLOTS: usize = 1024; + +/// Group the work items into requests the native reader can actually serve. +/// +/// `eth_getStorageValues` takes a `{contract: [slots]}` map, so slots for +/// *different* contracts at the same block travel in one request. That is the +/// one thing the extractor cannot do at any batch size: injected code runs at a +/// single address and reads only that address's storage. A partition of 500 +/// contracts × 2 slots is 500 extractor calls and one native call. +/// +/// When `merge` is false — the account readers, or an endpoint already known +/// not to have the method — every item becomes its own group and the runner +/// behaves exactly as it did before this function existed. +fn plan_native_chunks( + work: Vec<(u64, Address, Vec)>, + merge: bool, +) -> Vec)>> { + if !merge { + return work.into_iter().map(|item| vec![item]).collect() + } + + // Sorted by block so a partition's requests come out in a stable order; + // the frame is sorted by schema at the end, so this is legibility only. + let mut by_block: std::collections::BTreeMap)>> = + std::collections::BTreeMap::new(); + for item in work { + by_block.entry(item.0).or_default().push(item); + } + + let mut out = Vec::new(); + for (_, items) in by_block { + let mut current: Vec<(u64, Address, Vec)> = Vec::new(); + let mut budget = 0usize; + for item in items { + let n = item.2.len(); + // An item already over the ceiling on its own is left alone. + // Merging could only make the request larger, and its refusal would + // then be counted against the endpoint as a missing method. + if n > NATIVE_STORAGE_MAX_SLOTS { + out.push(vec![item]); + continue + } + if !current.is_empty() && budget + n > NATIVE_STORAGE_MAX_SLOTS { + out.push(std::mem::take(&mut current)); + budget = 0; + } + budget += n; + current.push(item); + } + if !current.is_empty() { + out.push(current); + } + } + out +} + +/// Answer a group of same-block work items: natively if the node can, and with +/// the per-contract extractor ladder if it cannot. +/// +/// # Errors +/// Only what the extractor ladder returns, which is only what the dataset's own +/// per-row path returns. A native miss is a routing decision, never an outcome. +async fn native_chunk_with_fallback( + items: Vec<(u64, Address, Vec)>, + source: &std::sync::Arc, + query: std::sync::Arc, +) -> crate::R> +where + D: StateOverrideBatchable, +{ + if let Some(responses) = native_storage_batch::(&items, source).await { + return Ok(responses) + } + + // Each item keeps its own ladder and they run concurrently, so a native + // miss costs the one request it spent — never the parallelism. + let per_item = items + .into_iter() + .map(|(block, target, rows)| { + extractor_batch_with_fallback::(block, target, rows, source, query.clone()) + }) + .collect::>(); + Ok(futures::future::try_join_all(per_item).await?.into_iter().flatten().collect()) +} + +/// One `eth_getStorageValues` request covering every item in the group. +/// +/// This is the rung above the extractor, and it is strictly better where it +/// exists: no `code` override, so none of the override failure modes apply and +/// there is no sentinel to check — the node is answering the question that was +/// asked, not running our bytecode. It landed in geth v1.17.1 and nothing else +/// ships it, so most endpoints fall straight through. +/// +/// Returns `None`, never an error, whenever the answer is not complete and +/// in order. "This node does not have the method" is a routing fact rather than +/// a failure, and so is a short or unexpected answer — the caller falls through +/// to the extractor either way, which is what it would have done before this +/// function existed. +async fn native_storage_batch( + items: &[(u64, Address, Vec)], + source: &std::sync::Arc, +) -> Option> +where + D: StateOverrideBatchable, +{ + if !D::reader().injects_at_target() || !source.storage_values_worth_trying() { + return None + } + // `plan_native_chunks` keeps a group under the ceiling, except for a single + // item that was already over it on its own — which happens when + // `--state-override-batch-size` is set above 1024. Sending it would buy a + // guaranteed `-38026` (geth, erigon) or `-32602` (reth). Those do now count + // as misses, so the run would stop asking after three — but three wasted + // requests for a limit we already know is a worse answer than not asking. + if items.iter().map(|(_, _, rows)| rows.len()).sum::() > NATIVE_STORAGE_MAX_SLOTS { + return None + } + let block = items.first()?.0; + + // One contract can appear in more than one item, when its rows were split + // by batch size. So the map is *extended* per contract, never inserted + // into: an insert would drop the earlier item's slots on the floor and the + // length check below would then reject the whole group. + let mut request: std::collections::HashMap> = + std::collections::HashMap::new(); + for (_, target, rows) in items { + let entry = request.entry(*target).or_default(); + for p in rows { + entry.push(B256::from(D::input_word(p).ok()?.to_be_bytes::<32>())); + } + } + + let answer = source.get_storage_values(&request, block).await.ok()?; + + // Take the values back out in the order they went in, walking a per-contract + // cursor so a contract split across items gets each item's own slice. + let mut cursors: std::collections::HashMap = std::collections::HashMap::new(); + let mut out = Vec::with_capacity(items.iter().map(|(_, _, rows)| rows.len()).sum()); + for (_, target, rows) in items { + let values = answer.get(target)?; + let start = *cursors.get(target).unwrap_or(&0); + let end = start.checked_add(rows.len())?; + if end > values.len() { + return None + } + for (p, v) in rows.iter().zip(&values[start..end]) { + out.push(D::decode_row(p, *v).ok()?); + } + cursors.insert(*target, end); + } + + // A node that answered with more slots than were asked for, or named a + // contract that was not in the request, is not answering this question. + // Refuse the lot rather than pick the plausible parts out of it. + if answer.len() != request.len() { + return None + } + for (target, values) in &answer { + if cursors.get(target).copied().unwrap_or(0) != values.len() { + return None + } + } + + Some(out) +} + +#[cfg(test)] +mod tests { + + #[test] + fn chunking_never_leaves_a_lone_row_behind() { + // `chunks(batch_size)` puts the whole remainder last, so `batch * k + 1` + // rows ended in a chunk of one — the single-row override call that + // `MIN_BATCH_GROUP` exists to refuse. + for batch in [2usize, 3, 10, 50, 100, 250, 1000, 1024] { + for len in 2..3000usize { + let rows: Vec = (0..len).collect(); + let chunks = even_chunks(&rows, batch); + + let sizes: Vec = chunks.iter().map(|c| c.len()).collect(); + assert_eq!(sizes.iter().sum::(), len, "len={len} batch={batch}"); + assert!(sizes.iter().all(|n| *n <= batch), "len={len} batch={batch} {sizes:?}"); + + // Every chunk within one row of every other is the strongest + // statement available: three rows at batch two must still be + // 2 + 1. + let (min, max) = ( + *sizes.iter().min().expect("at least one chunk"), + *sizes.iter().max().expect("at least one chunk"), + ); + assert!(max - min <= 1, "len={len} batch={batch} {sizes:?}"); + + // The rows come back in order, and all of them come back. + let flat: Vec = chunks.concat(); + assert_eq!(flat, rows, "len={len} batch={batch}"); + } + } + } + + #[test] + fn a_group_that_fits_in_one_batch_stays_one_chunk() { + let rows: Vec = (0..1000).collect(); + assert_eq!(even_chunks(&rows, 1000).len(), 1); + assert_eq!(even_chunks(&rows, 1001).len(), 1); + // One row over is two near-equal chunks, not 1000 + 1. + let rows: Vec = (0..1001).collect(); + let sizes: Vec = even_chunks(&rows, 1000).iter().map(|c| c.len()).collect(); + assert_eq!(sizes, vec![501, 500]); + } + use super::*; + + /// Every extractor is the same 25 bytes with one opcode swapped, and the + /// jump destinations must land on the two `JUMPDEST`s regardless. + #[test] + fn bytecode_is_the_template_with_one_opcode_swapped() { + for (reader, opcode) in [ + (StateReader::Storage, 0x54), + (StateReader::Balance, 0x31), + (StateReader::CodeHash, 0x3f), + (StateReader::CodeSize, 0x3b), + ] { + let code = reader.bytecode(); + assert_eq!(code.len(), 25, "{reader:?}"); + assert_eq!(code[OPCODE_OFFSET], opcode, "{reader:?}"); + // Everything except the swapped byte is identical across readers. + for (i, (got, want)) in code.iter().zip(TEMPLATE.iter()).enumerate() { + if i != OPCODE_OFFSET { + assert_eq!(got, want, "{reader:?} byte {i:#04x}"); + } + } + // The two forward jumps must target real JUMPDESTs, or every call + // reverts with "invalid jump destination". + assert_eq!(code[0x02], 0x5b, "loop JUMPDEST"); + assert_eq!(code[0x14], 0x5b, "exit JUMPDEST"); + assert_eq!(code[0x07], 0x14, "JUMPI target is the exit JUMPDEST"); + assert_eq!(code[0x12], 0x02, "JUMP target is the loop JUMPDEST"); + } + } + + /// The published Dedaub bytecode is the `PUSH0` form of the same loop. We + /// deliberately do not emit it (see the module docs), so this pins the + /// difference to exactly the two `PUSH0`s and the shifted jump targets — + /// if someone "optimises" the template back to 23 bytes, this fails. + #[test] + fn we_do_not_emit_the_push0_form() { + let code = StateReader::Storage.bytecode(); + let dedaub = alloy::hex::decode("5f5b80361460135780355481526020016001565b365ff3").unwrap(); + assert_eq!(dedaub.len(), 23); + assert_ne!(code.as_ref(), dedaub.as_slice()); + assert!(!code.contains(&0x5f), "PUSH0 is invalid below Shanghai"); + } + + #[test] + fn calldata_is_bare_contiguous_words() { + let data = StateReader::Storage.calldata([U256::from(0), U256::from(1), U256::from(2)]); + assert_eq!(data.len(), 96, "no selector, no ABI header, no length prefix"); + assert_eq!(data[31], 0); + assert_eq!(data[63], 1); + assert_eq!(data[95], 2); + // Empty input is empty calldata: the loop exits on the first iteration. + assert!(StateReader::Storage.calldata([]).is_empty()); + } + + #[test] + fn addresses_are_right_aligned_in_their_word() { + let addr = Address::from_slice(&[0xab; 20]); + let data = StateReader::Balance.calldata([U256::from_be_slice(addr.as_slice())]); + assert_eq!(data.len(), 32); + assert_eq!(&data[..12], &[0u8; 12], "upper 12 bytes zeroed"); + assert_eq!(&data[12..], addr.as_slice()); + } + + #[test] + fn storage_overrides_the_target_and_account_readers_do_not() { + let target = Address::from_slice(&[0x11; 20]); + + let storage = StateReader::Storage.overrides(target); + assert!(storage.contains_key(&target)); + assert_eq!(StateReader::Storage.call_target(target), target); + + for reader in [StateReader::Balance, StateReader::CodeHash, StateReader::CodeSize] { + let over = reader.overrides(target); + assert!(!over.contains_key(&target), "{reader:?} must not touch a real contract"); + assert!(over.contains_key(&SCRATCH_ADDRESS), "{reader:?}"); + assert_eq!(reader.call_target(target), SCRATCH_ADDRESS); + } + } + + /// `state` would replace the account's whole storage — the values this call + /// exists to read. Only `code` and a single-slot `state_diff` are ever set. + #[test] + fn overrides_never_replace_whole_storage() { + let target = Address::from_slice(&[0x22; 20]); + + let over = StateReader::Storage.overrides(target); + let account = over.get(&target).expect("target override present"); + assert_eq!(account.code, Some(StateReader::Storage.bytecode())); + assert!(account.balance.is_none()); + assert!(account.nonce.is_none()); + assert!(account.state.is_none(), "`state` would mask the real storage"); + // The sentinel, and only the sentinel. + let diff = account.state_diff.as_ref().expect("storage reader sets a sentinel"); + assert_eq!(diff.len(), 1); + assert_eq!(diff.get(&SENTINEL_SLOT), Some(&SENTINEL_VALUE)); + + // The account readers read no storage, so they set no diff at all. + for reader in [StateReader::Balance, StateReader::CodeHash, StateReader::CodeSize] { + let over = reader.overrides(target); + let account = over.get(&SCRATCH_ADDRESS).expect("scratch override present"); + assert!(account.state.is_none(), "{reader:?}"); + assert!(account.state_diff.is_none(), "{reader:?}"); + } + } + + /// The sentinel rides along as the final calldata word so the caller's + /// inputs keep their indices. + #[test] + fn storage_requests_append_the_sentinel_last() { + let target = Address::from_slice(&[0x44; 20]); + let inputs = [U256::from(7), U256::from(8)]; + + let (call_data, _over, to) = StateReader::Storage.request(target, &inputs); + assert_eq!(to, target); + assert_eq!(call_data.len(), 32 * 3, "two inputs plus the sentinel"); + assert_eq!(&call_data[64..96], SENTINEL_SLOT.as_slice()); + + // Account readers carry no sentinel: an ignored override would call a + // codeless address and return empty, which the length check rejects. + let (call_data, _over, to) = StateReader::Balance.request(target, &inputs); + assert_eq!(to, SCRATCH_ADDRESS); + assert_eq!(call_data.len(), 32 * 2); + } + + #[test] + fn decode_splits_the_return_into_words() { + // Storage: two values then the sentinel. + let mut data = Vec::new(); + data.extend_from_slice(&[0xaa; 32]); + data.extend_from_slice(&[0xbb; 32]); + data.extend_from_slice(SENTINEL_VALUE.as_slice()); + let words = StateReader::Storage.decode_response(&data, 2).unwrap(); + assert_eq!(words, vec![B256::repeat_byte(0xaa), B256::repeat_byte(0xbb)]); + + // Account readers: no sentinel to strip. + let mut data = Vec::new(); + data.extend_from_slice(&[0xcc; 32]); + let words = StateReader::Balance.decode_response(&data, 1).unwrap(); + assert_eq!(words, vec![B256::repeat_byte(0xcc)]); + assert_eq!(StateReader::Balance.decode_response(&[], 0).unwrap(), Vec::::new()); + } + + /// The load-bearing safety test. An endpoint that accepts the third + /// `eth_call` parameter and ignores it delivers our calldata to the real + /// contract, whose answer must never be mistaken for storage. + #[test] + fn decode_rejects_a_return_the_override_did_not_produce() { + // Ignored override, contract fell through to an empty fallback. + let err = StateReader::Storage.decode_response(&[], 3).unwrap_err(); + assert!(format!("{err:?}").contains("ignored"), "error must name the cause: {err:?}"); + + // Wrong length in either direction — never truncate or pad to fit. + assert!(StateReader::Storage.decode_response(&[0u8; 64], 3).is_err()); + assert!(StateReader::Storage.decode_response(&[0u8; 192], 3).is_err()); + // Not a whole number of words. + assert!(StateReader::Storage.decode_response(&[0u8; 31], 1).is_err()); + } + + /// The case the length check alone cannot catch: a contract that answers + /// with exactly the right number of bytes. Only the sentinel separates a + /// real read from a coincidence. + #[test] + fn a_right_sized_return_without_the_sentinel_is_still_rejected() { + let mut forged = Vec::new(); + forged.extend_from_slice(&[0x11; 32]); // plausible "slot 0" + forged.extend_from_slice(&[0x22; 32]); // plausible "slot 1" + forged.extend_from_slice(&[0x00; 32]); // where the sentinel should be + assert_eq!(forged.len(), 96, "exactly the length the decoder expects"); + + let err = StateReader::Storage.decode_response(&forged, 2).unwrap_err(); + let msg = format!("{err:?}"); + assert!(msg.contains("sentinel"), "must name the sentinel: {msg}"); + assert!(msg.contains("did not apply it"), "must say what went wrong: {msg}"); + } + + /// Overriding a precompile's code is undefined across clients, so those + /// addresses are never used as extractor targets. + #[test] + fn reserved_low_addresses_are_refused_as_storage_targets() { + for byte in [0x01u8, 0x02, 0x09, 0x11, 0x64, 0x6f, 0xff] { + let mut raw = [0u8; 20]; + raw[19] = byte; + let addr = Address::from(raw); + assert!(is_reserved_address(addr), "{addr} is in the reserved range"); + assert!(StateReader::Storage.refuses_target(addr).is_some(), "{addr}"); + } + // Just past the ceiling, and an ordinary contract, are both fine. + let mut raw = [0u8; 20]; + raw[17] = 0x01; // 0x010000 + assert!(!is_reserved_address(Address::from(raw))); + assert!(StateReader::Storage.refuses_target(Address::from(raw)).is_none()); + assert!(StateReader::Storage.refuses_target(Address::from_slice(&[0xab; 20])).is_none()); + // A high address whose low bytes look small must NOT be mistaken for one. + let mut high = [0u8; 20]; + high[0] = 0x01; + high[19] = 0x01; + assert!(!is_reserved_address(Address::from(high))); + } + + /// Found by running the emitted bytecode against a live node: the account + /// readers execute *at* the scratch address, so that one account reports + /// the extractor's own code (size 25, hash `keccak256(extractor)`) instead + /// of the chain's. Storage is immune — its code lives at the target and + /// `SLOAD` reads storage, which the `code` override does not touch. + #[test] + fn account_readers_cannot_see_the_scratch_address_truthfully() { + for reader in [StateReader::Balance, StateReader::CodeHash, StateReader::CodeSize] { + assert!(reader.shadows_scratch(SCRATCH_ADDRESS), "{reader:?}"); + assert!(!reader.shadows_scratch(Address::from_slice(&[0x33; 20])), "{reader:?}"); + } + assert!( + !StateReader::Storage.shadows_scratch(SCRATCH_ADDRESS), + "storage runs at the target, not the scratch address", + ); + // The value a shadowed EXTCODESIZE would report, pinned so the guard's + // reason stays legible: the extractor's own length. + assert_eq!(StateReader::CodeSize.bytecode().len(), 25); + } + + /// The storage-side mirror of the test above, and the more dangerous of the + /// two: `SENTINEL_SLOT` is the one slot the storage reader writes, so a row + /// asking for it would be answered with `SENTINEL_VALUE` — our own marker, + /// returned as though it were measured. + /// + /// Nothing downstream can catch it. The response is exactly `32 * n` bytes + /// and its tail word is `SENTINEL_VALUE`, so the length check and the + /// sentinel check both pass and the row is written out. The guard has to be + /// at the runner's filter, which is what this pins. + #[test] + fn the_storage_reader_cannot_see_the_sentinel_slot_truthfully() { + let sentinel = U256::from_be_bytes(SENTINEL_SLOT.0); + assert!(StateReader::Storage.shadows_sentinel(sentinel)); + assert!(row_shadows_override(StateReader::Storage, sentinel)); + + // Any other slot, including the neighbouring keys, batches as usual. + for other in [U256::ZERO, U256::from(1), sentinel - U256::from(1), sentinel + U256::from(1)] + { + assert!(!row_shadows_override(StateReader::Storage, other), "{other}"); + } + + // The account readers write no storage, so the sentinel is not their + // blind spot — the scratch address is, and that guard still stands. + for reader in [StateReader::Balance, StateReader::CodeHash, StateReader::CodeSize] { + assert!(!reader.shadows_sentinel(sentinel), "{reader:?}"); + assert!(!row_shadows_override(reader, sentinel), "{reader:?}"); + let scratch = U256::from_be_slice(SCRATCH_ADDRESS.as_slice()); + assert!(row_shadows_override(reader, scratch), "{reader:?}"); + } + } + + /// Pin the per-slot gas budget that `DEFAULT_STATE_OVERRIDE_BATCH_SIZE`'s + /// derivation rests on, so an edit to `TEMPLATE` cannot silently invalidate + /// it. + /// + /// The 51 is the whole loop body. An earlier version of that doc said 22, + /// which is the prefix that reaches `JUMPI` — the loop-exit test is the + /// middle of the iteration, not the end of it. + #[test] + fn the_extractor_loop_costs_fifty_one_gas_per_slot() { + // (mnemonic, gas) for offsets 0x02..=0x13 of TEMPLATE: one iteration, + // with the SLOAD at 0x0b excluded because it is the swapped opcode and + // its cost is fork-dependent. + let body = [ + ("JUMPDEST", 1), + ("DUP1", 3), + ("CALLDATASIZE", 2), + ("EQ", 3), + ("PUSH1", 3), + ("JUMPI", 10), + ("DUP1", 3), + ("CALLDATALOAD", 3), + ("DUP2", 3), + ("MSTORE", 3), + ("PUSH1", 3), + ("ADD", 3), + ("PUSH1", 3), + ("JUMP", 8), + ]; + let per_iteration: u32 = body.iter().map(|(_, g)| g).sum(); + assert_eq!(per_iteration, 51, "loop body cost changed; fix the batch-size derivation"); + + // The prefix that stops at the loop-exit test, kept so the off-by-29 is + // named rather than merely absent. + let through_jumpi: u32 = body.iter().take(6).map(|(_, g)| g).sum(); + assert_eq!(through_jumpi, 22); + + // 2100 cold SLOAD + 51 loop + 512 worst-case calldata + 3 memory. + let per_slot = 2100 + per_iteration + 512 + 3; + assert_eq!(per_slot, 2666); + // Which leaves the 50M cap comfortably above the 1000-word default. + assert!(u64::from(DEFAULT_STATE_OVERRIDE_BATCH_SIZE) * u64::from(per_slot) < 50_000_000); + } + + // ----------------------------------------------------------------------- + // The demotion policy + // ----------------------------------------------------------------------- + + /// The classifier must recognise what the decoder actually produces, not a + /// paraphrase of it. Both errors are built by calling `decode_response`, so + /// rewording either message without updating the marker fails here rather + /// than silently costing every run its speedup. + #[test] + fn the_classifier_recognises_what_the_decoder_produces() { + // An endpoint that dropped the override: the real contract fell through + // to an empty fallback, so the return is the wrong length. + let ignored = StateReader::Storage.decode_response(&[], 3).unwrap_err(); + assert!(override_unavailable(&ignored), "{ignored:?}"); + + // An endpoint that dropped the override and whose contract happened to + // answer with the right *number* of bytes. Only the sentinel catches + // this one, and it must classify the same way. + let mut plausible = vec![0u8; 32 * 3]; + plausible.extend_from_slice(&[0u8; 32]); // the sentinel slot read back as zero + let mismatch = StateReader::Storage.decode_response(&plausible, 3).unwrap_err(); + assert!(override_unavailable(&mismatch), "{mismatch:?}"); + } + + /// A failure that is about size, or about the node, must NOT rule the + /// feature out — halving fixes the first and nothing here fixes the second, + /// but neither says the endpoint refuses overrides. Ruling out on these + /// would cost every run its speedup after one unlucky request. + #[test] + fn the_classifier_ignores_failures_that_are_not_about_overrides() { + for message in [ + "request entity too large", + "execution reverted", + "missing trie node: state is not available", + "your app has exceeded its compute units per second capacity", + ] { + let err = CollectError::CollectError(message.to_string()); + assert!(!override_unavailable(&err), "{message}"); + } + } + + /// An opcode the block's fork does not have must land on the per-call path, + /// and on nothing else. + /// + /// `EXTCODEHASH` arrived at Constantinople (block 7280000) and `PUSH0` at + /// Shanghai, so an old enough block rejects the extractor outright. The + /// right response is to read those rows one at a time. The two wrong ones + /// are both reachable by accident: + /// + /// * `override_unavailable` would latch `rule_out()` for the whole process, so one historical + /// chunk would cost every later dataset its batching. + /// * `batch_may_shrink_to_fit` would halve the batch down to singletons, each one failing the + /// same way — the doomed cascade the demotion policy exists to bound. + /// + /// Neither message contains any of the needles today. This pins that, so a + /// needle added later for some other endpoint cannot quietly poison the + /// fork path. + #[test] + fn an_unsupported_opcode_is_neither_a_missing_feature_nor_a_size_complaint() { + use crate::types::multicall::batch_may_shrink_to_fit; + for message in [ + "invalid opcode: opcode 0x3f not defined", + "invalid opcode: PUSH0", + "EVM error: NotActivated", + ] { + let err = CollectError::CollectError(message.to_string()); + assert!(!override_unavailable(&err), "would latch for the run: {message}"); + // `batch_may_shrink_to_fit` returns true for a non-provider error by + // design — one smaller attempt is cheap. What matters is that it is + // not a *provider* error saying "too large", so check the shape the + // node actually sends. + let rpc = CollectError::ProviderError(alloy::transports::RpcError::ErrorResp( + alloy::rpc::json_rpc::ErrorPayload { + code: -32000, + message: std::borrow::Cow::Owned(message.to_string()), + data: None, + }, + )); + assert!(!override_unavailable(&rpc), "would latch for the run: {message}"); + assert!(!batch_may_shrink_to_fit(&rpc), "would halve pointlessly: {message}"); + } + } + + /// The memo starts optimistic, latches once, and cannot be talked back out + /// of it by a batch that was already in flight when the verdict landed. + #[test] + fn the_memo_latches_one_way() { + let support = OverrideSupport::default(); + assert!(support.worth_attempting(), "an untried endpoint is worth one request"); + + support.record_supported(); + assert!(support.worth_attempting()); + + support.rule_out(); + assert!(!support.worth_attempting()); + + // The in-flight batch that started before the ruling comes back OK. + // Honouring it would put the run straight back into the cascade the + // ruling exists to end. + support.record_supported(); + assert!(!support.worth_attempting(), "a ruled-out endpoint must stay ruled out"); + } + + /// The gate opens for the first caller and stays shut for everyone else + /// once a verdict exists — that is the whole "one wasted request per run, + /// not one per chunk" property. + #[tokio::test] + async fn the_gate_admits_the_first_attempt_and_then_stands_aside() { + let support = OverrideSupport::default(); + + let first = support.gate_first_attempt().await; + assert!(first.is_some(), "the first attempt of a run is gated"); + // The first attempt reports back and releases the gate. + support.record_supported(); + drop(first); + + assert!( + support.gate_first_attempt().await.is_none(), + "with a verdict recorded there is nothing left to serialise", + ); + + let ruled_out = OverrideSupport::default(); + ruled_out.rule_out(); + assert!(ruled_out.gate_first_attempt().await.is_none()); + } + + // --------------------------------------------------------------- + // Cross-contract planning for the native reader + // --------------------------------------------------------------- + + /// `n` rows at one block against one contract. The rows' contents do not + /// matter to the planner — only how many there are. + fn item(block: u64, target: u8, n: usize) -> (u64, Address, Vec) { + let mut address = [0u8; 20]; + address[19] = target; + let rows = (0..n) + .map(|i| Params { + block_number: Some(block), + address: Some(address.to_vec()), + slot: Some(U256::from(i).to_be_bytes::<32>().to_vec()), + ..Default::default() + }) + .collect(); + (block, Address::new(address), rows) + } + + fn shape(plan: &[Vec<(u64, Address, Vec)>]) -> Vec> { + plan.iter().map(|g| g.iter().map(|(_, _, rows)| rows.len()).collect()).collect() + } + + /// With merging off — an account reader, or an endpoint already known not + /// to have the method — the plan must be exactly what the runner did before + /// any of this existed: one request per contract per block. + #[test] + fn the_planner_is_a_no_op_when_merging_is_off() { + let work = vec![item(10, 1, 3), item(10, 2, 4), item(11, 1, 5)]; + assert_eq!(shape(&plan_native_chunks(work, false)), vec![vec![3], vec![4], vec![5]]); + } + + /// The whole point: different contracts at the same block share a request. + #[test] + fn the_planner_merges_contracts_that_share_a_block() { + let work = vec![item(10, 1, 3), item(10, 2, 4), item(10, 3, 5)]; + let plan = plan_native_chunks(work, true); + assert_eq!(plan.len(), 1, "one block, one request"); + assert_eq!(shape(&plan), vec![vec![3, 4, 5]]); + } + + /// A request names one block, so two blocks can never share one however + /// small they are. Merging across them would read the wrong state. + #[test] + fn the_planner_never_merges_across_blocks() { + let work = vec![item(10, 1, 1), item(11, 1, 1), item(12, 1, 1)]; + let plan = plan_native_chunks(work, true); + assert_eq!(plan.len(), 3); + for group in &plan { + let blocks: std::collections::BTreeSet = + group.iter().map(|(b, _, _)| *b).collect(); + assert_eq!(blocks.len(), 1); + } + } + + /// The ceiling counts every slot in the request, across contracts. Three + /// 400-slot contracts are 1200 slots, which geth refuses, so the group must + /// break before it gets there. + #[test] + fn the_planner_respects_the_shared_slot_ceiling() { + let work = vec![item(10, 1, 400), item(10, 2, 400), item(10, 3, 400)]; + let plan = plan_native_chunks(work, true); + assert_eq!(shape(&plan), vec![vec![400, 400], vec![400]]); + for group in &plan { + let slots: usize = group.iter().map(|(_, _, rows)| rows.len()).sum(); + assert!(slots <= NATIVE_STORAGE_MAX_SLOTS, "{slots} slots in one request"); + } + } + + /// An item already over the ceiling on its own — the user raised + /// `--state-override-batch-size` past 1024 — must not drag neighbours into + /// a request that was going to be refused anyway. Its refusal would be + /// counted against the endpoint as a missing method. + #[test] + fn the_planner_isolates_an_item_that_is_already_too_large() { + let work = vec![item(10, 1, NATIVE_STORAGE_MAX_SLOTS + 1), item(10, 2, 2)]; + assert_eq!(shape(&plan_native_chunks(work, true)), vec![vec![1025], vec![2]]); + } + + /// Every row must survive planning exactly once. A planner that dropped or + /// duplicated an item would produce a short or double-counted frame, which + /// no downstream length check would catch. + #[test] + fn the_planner_conserves_every_row() { + let work = vec![ + item(10, 1, 700), + item(10, 2, 700), + item(11, 1, 2), + item(10, 3, 1), + item(11, 2, NATIVE_STORAGE_MAX_SLOTS + 5), + ]; + let before: usize = work.iter().map(|(_, _, rows)| rows.len()).sum(); + let plan = plan_native_chunks(work, true); + let after: usize = plan.iter().flatten().map(|(_, _, rows)| rows.len()).sum(); + assert_eq!(before, after); + } +} diff --git a/crates/python/rust/collect_adapter.rs b/crates/python/rust/collect_adapter.rs index 5813abf..718438e 100644 --- a/crates/python/rust/collect_adapter.rs +++ b/crates/python/rust/collect_adapter.rs @@ -205,6 +205,15 @@ pub fn _collect( multicall, multicall_batch_size, multicall_require_success, + // Request-batching switches, not yet exposed as Python keyword + // arguments. They are set to the same defaults the CLI uses so a + // programmatic run batches exactly like a `triodion` run; none of + // them can change results, only how many requests those results + // cost, because every batch path falls back to one request per row + // on failure. `0` means "use the dataset's own default size". + batch_state_reads: true, + batch_rpc_calls: true, + state_override_batch_size: 0, // Backwards-compat CLI alias for the legacy `--multicall` flag — accepted as // a no-op so the field is wired but unread. The Python entrypoint sets it to // false; the canonical `multicall` field above carries the real value. diff --git a/crates/python/rust/freeze_adapter.rs b/crates/python/rust/freeze_adapter.rs index 27297b8..cf45907 100644 --- a/crates/python/rust/freeze_adapter.rs +++ b/crates/python/rust/freeze_adapter.rs @@ -199,6 +199,15 @@ pub fn _freeze( multicall, multicall_batch_size, multicall_require_success, + // Request-batching switches, not yet exposed as Python keyword + // arguments. They are set to the same defaults the CLI uses so a + // programmatic run batches exactly like a `triodion` run; none of + // them can change results, only how many requests those results + // cost, because every batch path falls back to one request per row + // on failure. `0` means "use the dataset's own default size". + batch_state_reads: true, + batch_rpc_calls: true, + state_override_batch_size: 0, // Backwards-compat CLI alias for the legacy `--multicall` flag — accepted as // a no-op so the field is wired but unread. The Python entrypoint sets it to // false; the canonical `multicall` field above carries the real value. diff --git a/examples/all.sh b/examples/all.sh index b6dae03..3413975 100755 --- a/examples/all.sh +++ b/examples/all.sh @@ -39,6 +39,11 @@ $EXECUTABLE access_lists -b $BLOCKS -o $OUTPUT_DIR/access_lists $EXECUTABLE address_appearances -b $BLOCKS -o $OUTPUT_DIR/address_appearances +# ApprovalForAll logs. ERC-721 and ERC-1155 declare this event identically, so +# one topic0 covers both and a log cannot say which standard the contract +# implements. Join contract_address against contract_interfaces to find out. +$EXECUTABLE approvals_for_all -b $BLOCKS -o $OUTPUT_DIR/approvals_for_all + # EIP-7702 authorization tuples. Only type-0x04 transactions carry them, so a # pre-Prague range yields no rows. $EXECUTABLE authorizations -b $BLOCKS -o $OUTPUT_DIR/authorizations @@ -59,8 +64,28 @@ $EXECUTABLE codes -b $BLOCKS -o $OUTPUT_DIR/codes --address 0xc02aaa39b223fe8d0a $EXECUTABLE contracts -b $BLOCKS -o $OUTPUT_DIR/contracts +# What a contract answers to ERC-165 supportsInterface, for a fixed table of +# interface ids. Read answers_true_to_everything first: a contract that says yes +# to the reserved id 0xffffffff says yes to everything, so its row is noise. +# Alias: erc165 +$EXECUTABLE contract_interfaces --address 0xed5af388653567af2f388e6224dc7c4b3241c544 & + +# ERC-1155 URI logs. The event is optional and most contracts never emit it, so +# zero rows is the normal result. Alias: erc1155_uris +$EXECUTABLE erc1155_metadata -b $BLOCKS -o $OUTPUT_DIR/erc1155_metadata + +# ERC-1155 transfers, one row per token id moved. A TransferBatch of five ids +# becomes five rows -- an aggregate cannot be taken apart again. +$EXECUTABLE erc1155_transfers -b $BLOCKS -o $OUTPUT_DIR/erc1155_transfers + $EXECUTABLE erc20_approvals -b $BLOCKS -o $OUTPUT_DIR/erc20_approvals +# ERC-20 allowance in force at a block, per (token, owner, spender). This is the +# state; erc20_approvals is the history. transferFrom decrements an allowance +# without emitting Approval, so replaying events overstates what is spendable. +# Alias: allowances +$EXECUTABLE erc20_allowances --contract 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 --from-address 0xd8da6bf26964af9d7eed9e03e53415d37aa96045 --to-address 0x68b3465833fb72a70ecdf485e0e4c7bd8665fc45 & + $EXECUTABLE erc20_balances -b $BLOCKS -o $OUTPUT_DIR/erc20_balances --contract 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 --address 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 & $EXECUTABLE erc20_metadata -b $BLOCKS -o $OUTPUT_DIR/erc20_metadata --address 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 & @@ -72,10 +97,29 @@ $EXECUTABLE erc20_transfers -b $BLOCKS -o $OUTPUT_DIR/erc20_transfers # Deposit / Withdrawal events of a wrapped native token. Aliases: wrapper_events, weth_events $EXECUTABLE erc20_wrapper_events -b $BLOCKS -o $OUTPUT_DIR/erc20_wrapper_events --address 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 & +# ERC-2612 permit nonce, per token per owner. Needs both dimensions to name a +# row. A null nonce means the token has no permit support; 0 means it has permit +# support and this owner has not used it. +$EXECUTABLE erc2612_nonces --contract 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 --address 0xd8da6bf26964af9d7eed9e03e53415d37aa96045 & + +# ERC-4626 vault asset, total assets and share supply. There is deliberately no +# share-price column: divide the two yourself, with your own convention. +$EXECUTABLE erc4626_metadata --address 0x83f20f44975d03b1b09e64809b757c47f942beea & + +# ERC-4626 vault Deposit and Withdraw events. Not the WETH-shape Deposit -- +# different signature, different topic0, disjoint from erc20_wrapper_events. +# Alias: erc4626_events +$EXECUTABLE erc4626_vault_events -b $BLOCKS -o $OUTPUT_DIR/erc4626_vault_events + $EXECUTABLE erc721_metadata -b $BLOCKS -o $OUTPUT_DIR/erc721_metadata --contract 0xed5af388653567af2f388e6224dc7c4b3241c544 & $EXECUTABLE erc721_transfers -b $BLOCKS -o $OUTPUT_DIR/erc721_transfers --contract 0xed5af388653567af2f388e6224dc7c4b3241c544 & +# ERC-777 Sent / Minted / Burned. DO NOT UNION WITH erc20_transfers: a compliant +# ERC-777 token emits an ERC-20 Transfer for the same movement, so both tables +# describe it and a union double-counts every ERC-777 movement on the chain. +$EXECUTABLE erc777_transfers -b $BLOCKS -o $OUTPUT_DIR/erc777_transfers + $EXECUTABLE eth_calls -b $BLOCKS -o $OUTPUT_DIR/eth_calls --call-data 0x18160ddd --contract 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 & $EXECUTABLE logs -b $BLOCKS -o $OUTPUT_DIR/logs @@ -88,6 +132,15 @@ $EXECUTABLE nonce_reads -b $BLOCKS -o $OUTPUT_DIR/nonce_reads $EXECUTABLE nonces -b $BLOCKS -o $OUTPUT_DIR/nonces --address 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 & +# ERC-1967 proxy slots, read from storage. Null in all three columns is the +# normal answer for an ordinary contract. Alias: erc1967_slots +$EXECUTABLE proxy_slots --address 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 & + +# ERC-1967 proxy events: Upgraded, BeaconUpgraded, AdminChanged. The event view +# of a proxy; proxy_slots is the state view. Neither replaces the other -- a +# proxy can write the slot and emit nothing. Alias: erc1967_events +$EXECUTABLE proxy_upgrades -b $BLOCKS -o $OUTPUT_DIR/proxy_upgrades + # storage slot values. `slots` is the dataset name, `storages` is an alias for it $EXECUTABLE slots -b $BLOCKS -o $OUTPUT_DIR/slots --address 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2 --slot 0x0000000000000000000000000000000000000000000000000000000000000000 &