Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,7 @@ triodion datasets
─────────────────
- access_lists
- address_appearances
- approvals_for_all (aliases = erc721_approvals_for_all, erc1155_approvals)
- authorizations
- balance_diffs
- balance_reads
Expand All @@ -473,16 +474,24 @@ triodion datasets
- code_reads
- codes
- consolidation_requests
- contract_interfaces (alias = erc165)
- contracts
- deposit_requests
- erc1155_metadata (alias = erc1155_uris)
- erc1155_transfers
- erc20_allowances (alias = allowances)
- erc20_balances
- erc20_metadata
- erc20_supplies
- erc20_transfers
- erc20_approvals
- erc20_wrapper_events (aliases = wrapper_events, weth_events)
- erc2612_nonces
- erc4626_metadata
- erc4626_vault_events (alias = erc4626_events)
- erc721_metadata
- erc721_transfers
- erc777_transfers
- eth_calls
- four_byte_counts (alias = 4byte_counts)
- geth_calls
Expand All @@ -497,6 +506,8 @@ triodion datasets
- nonce_diffs
- nonce_reads
- nonces
- proxy_slots (alias = erc1967_slots)
- proxy_upgrades (alias = erc1967_events)
- slots (alias = storages)
- storage_diffs (alias = slot_diffs)
- storage_reads (alias = slot_reads)
Expand Down
12 changes: 12 additions & 0 deletions book/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- [Data Acquisition](./basic_usage/data_acquisition.md)
- [Data Formatting](./basic_usage/data_formatting.md)
- [Data Querying](./basic_usage/data_querying.md)
- [Request Batching](./basic_usage/request_batching.md)
- [Python Bindings](./triodion_python/overview.md)
- [Installation](./triodion_python/installation.md)
- [Example Usage](./triodion_python/example_usage.md)
Expand All @@ -30,6 +31,7 @@
- [Overview of each dataset](./datasets/dataset_reference.md)
- [access_lists](./datasets/access_lists.md)
- [address_appearances](./datasets/address_appearances.md)
- [approvals_for_all](./datasets/approvals_for_all.md)
- [authorizations](./datasets/authorizations.md)
- [balance_diffs](./datasets/balance_diffs.md)
- [balance_reads](./datasets/balance_reads.md)
Expand All @@ -40,16 +42,24 @@
- [code_reads](./datasets/code_reads.md)
- [codes](./datasets/codes.md)
- [consolidation_requests](./datasets/consolidation_requests.md)
- [contract_interfaces](./datasets/contract_interfaces.md)
- [contracts](./datasets/contracts.md)
- [deposit_requests](./datasets/deposit_requests.md)
- [erc1155_metadata](./datasets/erc1155_metadata.md)
- [erc1155_transfers](./datasets/erc1155_transfers.md)
- [erc20_allowances](./datasets/erc20_allowances.md)
- [erc20_balances](./datasets/erc20_balances.md)
- [erc20_metadata](./datasets/erc20_metadata.md)
- [erc20_supplies](./datasets/erc20_supplies.md)
- [erc20_transfers](./datasets/erc20_transfers.md)
- [erc20_approvals](./datasets/erc20_approvals.md)
- [erc20_wrapper_events](./datasets/erc20_wrapper_events.md)
- [erc2612_nonces](./datasets/erc2612_nonces.md)
- [erc4626_metadata](./datasets/erc4626_metadata.md)
- [erc4626_vault_events](./datasets/erc4626_vault_events.md)
- [erc721_metadata](./datasets/erc721_metadata.md)
- [erc721_transfers](./datasets/erc721_transfers.md)
- [erc777_transfers](./datasets/erc777_transfers.md)
- [eth_calls](./datasets/eth_calls.md)
- [four_byte_counts](./datasets/four_byte_counts.md)
- [geth_calls](./datasets/geth_calls.md)
Expand All @@ -64,6 +74,8 @@
- [nonce_diffs](./datasets/nonce_diffs.md)
- [nonce_reads](./datasets/nonce_reads.md)
- [nonces](./datasets/nonces.md)
- [proxy_slots](./datasets/proxy_slots.md)
- [proxy_upgrades](./datasets/proxy_upgrades.md)
- [slots](./datasets/slots.md)
- [storage_diffs](./datasets/storage_diffs.md)
- [storage_reads](./datasets/storage_reads.md)
Expand Down
145 changes: 145 additions & 0 deletions book/basic_usage/request_batching.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# Request Batching

Most datasets read one thing per row: one storage slot, one balance, one header.
The obvious way to collect a partition is one RPC per row, and that is what
triodion did. It is also what makes a large query slow, and what gets a public
endpoint to answer `429`.

Triodion now takes the best route each dataset and endpoint allow. Every route
is **on by default**, and every route falls back to the per-row path when it
fails — so batching changes how many requests your results cost, never the
results themselves.

## The switches

| Flag | Turns off | Applies to |
|---|---|---|
| `--no-batch-rpc-calls` | JSON-RPC request batching | `blocks`, `nonces`, `codes` |
| `--no-batch-state-reads` | `eth_call` state overrides | `slots`, `balances`, `proxy_slots` |
| `--state-override-batch-size N` | (sets size, 0 = default 1000) | `slots`, `balances`, `proxy_slots` |
| `--no-multicall` | Multicall3 aggregation | `eth_calls`, the ERC-20 and vault read datasets |

Multicall3 is on by default like the other two, so the flag that changes
anything is `--no-multicall`. A bare `--multicall` is still accepted, as a
no-op, so older scripts keep working.

They are separate switches because they ask different things of the node. A
JSON-RPC batch needs nothing beyond JSON-RPC itself. A state override needs an
endpoint that honours the third `eth_call` parameter, which not every endpoint
does. An operator who distrusts an endpoint's override support has no reason to
give up plain batching as well.

## JSON-RPC batching

JSON-RPC has always allowed an array of requests in one HTTP body. `blocks`,
`nonces` and `codes` use it, because none of them has a better route available.

Measured on `blocks -b 21000000:+500` through a counting proxy in front of a
public mainnet endpoint:

| | HTTP requests | embedded calls | wall clock |
|---|---|---|---|
| `--no-batch-rpc-calls` | 501 | 501 | 0.96s |
| default | **6** | 501 | **0.46s** |

The two parquet files are byte-identical.

The request count is the reliable number: 83× fewer, and it follows from the
batch size rather than from the endpoint. The wall-clock figure does not
generalise — this endpoint was fast and did not throttle, so batching only won
2×. The further an endpoint is, and the harder it rate-limits, the more of the
501 round trips batching removes and the wider the gap gets. That is also the
case where the per-request path stops finishing at all.

The batch size negotiates itself downward when a provider refuses one — OP
Mainnet answers `413` above ten calls, Base returns
`-32014 "maximum 10 calls in 1 batch"` — and the discovered size is kept for the
rest of the call rather than rediscovered per batch.

`codes` uses a smaller batch (50) because whole contract bytecode is far larger
than a 32-byte word, and a hundred EIP-170-sized contracts is a 2.4MB response.

`nonces` is the one account field that can *only* batch this way. See
[nonces](../datasets/nonces.md).

## State overrides

`eth_call` takes a third parameter, a state-override set, that applies changes to
a scratch copy of state before execution. Overriding a contract's **code** with a
small extractor loop turns "read one slot" into "read every slot I name".

There is no Multicall3 equivalent for storage, because a deployed contract
cannot read another contract's slots. Bytecode *injected into* the target can
read all of them.

The same 25-byte loop with one opcode swapped serves four readers: `SLOAD` for
storage, and `BALANCE` / `EXTCODEHASH` / `EXTCODESIZE` for accounts. The account
readers take their address from the stack, so they run at a scratch address and
batch across unrelated addresses without overriding any real contract.

### Historical blocks

`eth_call` at a historical block executes under *that block's* fork rules.
Triodion's extractor is deliberately `PUSH0`-free, because `PUSH0` only became
valid at Shanghai (mainnet block 17034870, April 2023). Below that a `PUSH0`
extractor is rejected outright — geth reports `invalid opcode: PUSH0`, reth
reports `EVM error: NotActivated`.

The published 23-byte extractor from Dedaub's write-up of this technique is the
`PUSH0` form. Triodion emits a 25-byte variant instead, verified byte-identical
to `eth_getStorageAt` at latest, block 15000000 and block 12000000. There is no
fork detection to get wrong.

## The ladder

For `slots`, triodion tries three things in order. Measured on 4 contracts ×
6 slots at one block, through a proxy that denies methods on demand:

| rung | condition | requests |
|---|---|---|
| 1 | `eth_getStorageValues` available | **1** |
| 2 | native denied, overrides honoured | 4 `eth_call` + 1 probe |
| 3 | both denied | 24 `eth_getStorageAt` |

All three produce identical output.

**Rung 1** is `eth_getStorageValues`, added in geth v1.17.1. It does natively
what the extractor does by injection — many slots, across many contracts, one
round trip — with no override and none of the override failure modes. Because it
takes a `{contract: [slots]}` map, slots for *different* contracts at the same
block travel in one request. That is the one thing the extractor cannot do at
any batch size, since injected code reads only its own address's storage.

Support for it cannot be cached. A load-balanced pool holds a mix of node
versions: the same URL served this method on one request and returned
`Method not found` on the next. So triodion counts *consecutive* misses rather
than latching a verdict, and a mixed pool keeps using it whenever it works.

**Rung 2** is the extractor, one `eth_call` per contract.

**Rung 3** is the per-row path, exactly as it was before any of this existed.

## Verification

An endpoint that *accepts* the override parameter and silently ignores it is the
dangerous case. Our slot-key calldata would reach the real contract, and its
fallback output would be decoded as storage — plausible values, right shape,
entirely fabricated, written to a Parquet file that looks correct forever.

Two things prevent that:

1. The return must be **exactly** `32 × n` bytes. The extractor returns one word per calldata word by construction, so the length is positive proof that our code produced the answer.
2. The storage reader also sets a sentinel slot via `stateDiff` and appends that slot as the final calldata word. Getting the sentinel value back in the final returned word proves our bytecode ran, that the override was applied, and that words came back in order.

Anything else is an error, never a partial result — and an error only routes
those rows to the per-row path.

Note that `stateDiff` is used, never `state`. `state` replaces an account's
entire storage, which would return a perfectly well-formed file of zeros.

The sentinel slot is itself the one slot this path cannot read, because it is
the one slot this path writes. A row that asks for it would come back holding
the sentinel value, with the length check and the sentinel check both passing.
So a row naming that slot skips the batch and is read by `eth_getStorageAt`,
with no override in sight. The account readers have the same blind spot at the
scratch address they run at, and it is guarded the same way.
114 changes: 114 additions & 0 deletions book/datasets/approvals_for_all.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# approvals_for_all

One row per `ApprovalForAll(address,address,bool)` log.

```
approvals_for_all
─────────────────
- can collect by block or by transaction
- required parameters: [none]
- optional parameters: address, topic1, topic2, from_address, to_address
- dataset aliases: erc721_approvals_for_all, erc1155_approvals
- parameter aliases:
- contracts -> addresses


schema for approvals_for_all
────────────────────────────
- block_number: uint32
- transaction_index: uint32
- log_index: uint32
- transaction_hash: binary
- contract_address: binary
- owner: binary
- operator: binary
- approved: bool
- chain_id: uint64

sorting approvals_for_all by: block_number, log_index

other available columns: block_hash
```

Run `triodion help approvals_for_all` to print this from the binary.

Aliases: `erc721_approvals_for_all`, `erc1155_approvals`.

## Why the name has no standard in it

ERC-721 and ERC-1155 declare this event with the same argument types:

```solidity
// ERC-721
ApprovalForAll(address indexed owner, address indexed operator, bool approved)
// ERC-1155
ApprovalForAll(address indexed account, address indexed operator, bool approved)
```

The argument names differ; the types do not. Both therefore hash to one topic0,
and a log that carries it does not say which of the two standards the emitting
contract implements. No decoding recovers that — the bytes are identical.

Calling this dataset `erc1155_approvals` would put a false claim on roughly half
of its rows. Both names are kept as aliases, because both arrive at the same
rows, but neither names the dataset, so no output file claims a standard the
logs cannot prove.

To classify a contract, join `contract_address` against
[contract_interfaces](./contract_interfaces.md), which asks the contract itself
through the ERC-165 `supportsInterface` call.

## What the grant covers

`owner` is the token holder whose permission changes. `operator` is the address
that gains or loses permission over **every** token the owner holds in that
contract.

Unlike the ERC-721 `Approval` event, this grant names no token id, so it also
covers tokens the owner acquires after this block. A blanket grant made once is
still in force against a token bought a year later.

## These are events, not state

`approved == false` is a revocation. It is a measurement of what happened, not
the absence of one, so it is stored as `false` and never as null.

That makes `WHERE approved` wrong on its own. An operator approved in one block
and revoked in the next leaves both rows behind, and the filter keeps the stale
one. Take the latest row per grant first, then filter:

```sql
SELECT * FROM (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY contract_address, owner, operator
ORDER BY block_number DESC, log_index DESC) AS rn
FROM approvals_for_all
) WHERE rn = 1 AND approved
```

## Filtering

The event carries exactly one signature, so topic1 is the owner and topic2 is
the operator on every row, and address-shaped filtering is unambiguous here.
Both are reachable two ways: as a raw 32-byte topic, or as an ordinary 20-byte
address.

`--from-address` maps to topic1 and `--to-address` to topic2, matching what
[erc20_approvals](./erc20_approvals.md) calls its owner and spender. The names
are loose — an approval has no direction — but demanding a hand-padded 32-byte
word for a value read off a block explorer is worse.

Give one form per slot, not both. `--topic1` together with `--from-address` is
an error rather than a precedence rule: both are partition dimensions, so
resolving the conflict silently would still multiply the run by the discarded
one and write the same rows to several files.

```bash
# every blanket approval a wallet has granted
triodion approvals_for_all -b 18M:+1000 \
--from-address 0xd8da6bf26964af9d7eed9e03e53415d37aa96045
```

This dataset is not a member of the `log_events` group. Requesting it
alongside `logs` or `erc20_transfers` therefore costs a separate
`eth_getLogs` per block range rather than sharing one with them.
Loading
Loading