feat: in-process ethrex integration (embedded EL, --execution-mode inprocess)#530
Draft
pablodeymo wants to merge 43 commits into
Draft
feat: in-process ethrex integration (embedded EL, --execution-mode inprocess)#530pablodeymo wants to merge 43 commits into
pablodeymo wants to merge 43 commits into
Conversation
Add ethlambda-ethrex-client crate speaking JWT HS256-authenticated JSON-RPC to the EL auth endpoint, with typed V3 wrappers for the four engine_* methods we use (exchangeCapabilities, forkchoiceUpdatedV3, newPayloadV3, getPayloadV3) and field-for-field schema match against the canonical execution-apis spec. The blockchain actor takes an optional EngineClient and fires engine_forkchoiceUpdatedV3 at interval 0 of every slot, fire-and-forget; errors are logged but never block consensus. Integration is opt-in via --execution-endpoint + --execution-jwt-secret flags (clap enforces both-or-neither). Verified end-to-end against real ethrex: capability handshake returns the 18 advertised methods, FCUs round-trip in sub-ms with SYNCING (expected -- Lean blocks do not carry an executionPayload yet; that schema change is deferred to an upstream leanSpec proposal, see docs/plans/engine-api-integration.md). Tests: 12 unit + 2 wire smoke tests covering JWT signing, V3 type serde, RPC error surfacing, and full request/response against a hand-rolled mock HTTP server.
…d FCU - types.rs: PayloadStatusKind now uses SCREAMING_SNAKE_CASE so `InvalidBlockHash` round-trips as `INVALID_BLOCK_HASH` (was `INVALIDBLOCKHASH`, which would silently fail to deserialize from any spec-compliant EL). - types.rs: PayloadId serializes/deserializes as a hex DATA string (`"0x..."`) instead of `[serde(transparent)]` over `[u8; 8]` (which emitted a JSON integer array). - types.rs: Added `hex_address` serde helper and applied it to `PayloadAttributesV3.suggested_fee_recipient`, `Withdrawal.address`, and `ExecutionPayloadV3.fee_recipient` — previously these `[u8; 20]` fields were emitted as integer arrays rather than the spec-required hex DATA strings. - types.rs: `hex_u256::deserialize` now returns a serde error on >32-byte input rather than panicking via `copy_from_slice`. - client.rs: HTTP responses now run through `.error_for_status()` before body parsing so 401/403/5xx surface as `EngineClientError::Transport` with a readable message instead of `DeserializeResponse`. - blockchain/lib.rs: `notify_execution_layer` now sends `H256::ZERO` for head/safe/finalized instead of beacon roots. Beacon roots are not EL block hashes; passing them confuses the EL into syncing to garbage. Zero is the spec-friendly "unknown head" sentinel until Lean blocks carry an executionPayload. - bin/ethlambda/main.rs: Fixed misleading warn log — the capability handshake is one-shot at startup, not retried on each tick. - docs/plans/engine-api-integration.md: Replaced absolute local filesystem paths with GitHub URLs. Added unit tests for each bug fix (6 new tests, 16 total in ethrex-client lib). All targeted tests pass, `cargo fmt --all -- --check` clean, `cargo clippy --workspace --all-targets -- -D warnings` clean.
Phase 1a of the M6 plan (docs/plans/engine-api-integration.md, also updated in this commit). The canonical block-component types — ExecutionPayloadV3, Withdrawal, HexBytes, and the hex_* serde helpers — move from the engine-client crate into the foundational types crate, where the Lean BlockBody can later embed them directly. ethlambda-ethrex-client's public API stays stable through re-exports. No SSZ derives yet; those land in Phase 2 alongside the BlockBody embed.
Phase 2a of the M6 plan. Adds SszEncode/SszDecode/HashTreeRoot derives
so the canonical execution-payload types can later be embedded in
BlockBody and State (Phase 2c).
Variable-length list fields move to bounded SSZ types — the spec
requires limits at compile time for merkle tree layout:
extra_data: Vec<u8> → ByteList<MAX_EXTRA_DATA_BYTES>
transactions: Vec<HexBytes> → SszList<ByteList<MAX_BYTES_PER_TX>, MAX_TXS>
withdrawals: Vec<Withdrawal> → SszList<Withdrawal, 16>
Fixed-size byte fields move to plain arrays:
logs_bloom: Vec<u8> → [u8; 256]
JSON wire format is preserved byte-for-byte through new helper modules
(byte_list_hex, hex_bytes_fixed, transactions_serde, withdrawals_serde).
HexBytes is removed; its role is subsumed by ByteList<N> plus the new
transactions serde wrapper.
Manual Default impl on ExecutionPayloadV3: stdlib only auto-derives
Default for arrays up to length 32, and logs_bloom is 256 bytes.
Verified: 32 ethlambda-types tests pass (new SSZ + JSON roundtrips
check hash_tree_root consistency across both encodings); 12
ethrex-client lib tests pass; fmt clean; clippy clean.
Phase 2b of the M6 plan. Adds the cached projection that the consensus
state will carry between blocks (Capella+Deneb spec): every fixed-size
field copies from the payload verbatim, and the two variable-length
lists (transactions, withdrawals) collapse to their SSZ hash-tree roots
so the header itself stays bounded.
ExecutionPayloadV3::to_header() — explicit method
From<&ExecutionPayloadV3> for ExecutionPayloadHeader — sugar
Default — manual (same [u8; 256] reason as ExecutionPayloadV3)
Genesis convention: ExecutionPayloadHeader::default() is all-zeros.
The first Lean block carrying a real payload will assert its
parent_hash matches state.latest_execution_payload_header.block_hash —
which is H256::ZERO at genesis. Subsequent blocks chain forward
normally.
35 ethlambda-types tests pass (3 new: header default, header
SSZ+JSON roundtrip, to_header projects transactions/withdrawals to
their hash tree roots and copies every other field verbatim).
…reak)
Phase 2c of the M6 plan. Adds the canonical execution-payload fields
to the two SSZ containers that block import and STF revolve around:
BlockBody { attestations,
execution_payload: ExecutionPayloadV3 }
State { ...,
latest_execution_payload_header: ExecutionPayloadHeader }
State::from_genesis seeds the header all-zero so the first non-genesis
blocks payload must have parent_hash = H256::ZERO to be accepted —
clean genesis convention without pinning a real EL block hash.
This is the schema-breaking commit in the M6 sequence. Hash tree roots
for BlockBody, Block, and State all change. Consequences:
* Pinned genesis state_root + block_root unit test in
crates/common/types/src/genesis.rs updated to the new values.
* Every fixture-driven spec test that exercises these containers is
gated behind a FIXTURES_AWAIT_M6_REGEN: bool = true flag at the
top of its fn run(). To re-enable a group, flip the flag and
make leanSpec/fixtures after upstream lands the schema. Groups
gated wholesale: forkchoice_spectests (84 cases), stf_spectests
(49 cases), signature_spectests (11 cases). The ssz_spectests
dispatch skips only the BlockBody / Block / State / SignedBlock
arms (127 unrelated cases keep running).
* All other workspace tests pass; ethlambda-types lib tests still
cover ExecutionPayloadV3 / Header SSZ + JSON roundtrips.
Trade-off taken (vs. cargo feature flag, see plan doc Phase 7): the
fixture skips are explicit and tracked in code, but a real cargo
feature would have inflated every BlockBody / State construction with
cfg pollution. The feature-flag alternative was rejected in favor of
the localized skip.
Phase 2d will wire process_execution_payload into the STF —
parent_hash and timestamp assertions per the Capella spec.
Phase 2d of the M6 plan, closing out Phase 2. The STF now enforces the Capella-style payload assertions Pablo flagged on PR #367: 1. payload.parent_hash == state.latest_execution_payload_header.block_hash 2. payload.timestamp == compute_time_at_slot(slot) Both checks run inside process_block between header processing and attestation processing — same ordering as the spec. On success the new payloads header is cached onto state so the next block can chain forward. New error variants InvalidPayloadParentHash and InvalidPayloadTimestamp. Omitted vs. the spec, by design: * verify_and_notify_new_payload (engine_newPayloadV3 roundtrip): lives in the blockchain actor (Phase 3). The STF runs in-process, fork-choice testing, and spec-test harness contexts — none want a network call. * prev_randao: Lean state has no randao mix and leanSpec hasnt defined one. Re-add when upstream lands the field. * SECONDS_PER_SLOT is a duplicate const that must track ethlambda_blockchain::MILLISECONDS_PER_SLOT (currently 4000). state_transition cant depend on blockchain (wrong direction) and the millisecond resolution is wasted in STF. Documented in the consts doc comment. To keep non-EL proposers minting valid blocks until Phase 4 wires engine_getPayloadV3, build_block now calls a synthetic_payload helper that fills in (parent_hash, timestamp) deterministically from state. Phase 4 will swap this for the real EL response when an endpoint is configured. The 20 existing blockchain lib tests (including the two build_block tests) continue to pass. 4 new state_transition unit tests cover the happy path, parent-hash mismatch, timestamp mismatch, and a two-block chain-forward case.
…oadV3
Phase 3 of the M6 plan. Every block arriving over the network — gossip
or BlocksByRoot req-resp — now passes through the EL before fork-choice
insertion. The Handler\<NewBlock\> in BlockChainServer awaits
engine_newPayloadV3(payload, \[\], H256::ZERO) and drops the block on
explicit INVALID / INVALID_BLOCK_HASH verdicts; VALID, SYNCING, and
ACCEPTED all proceed to the existing on_block sync path. Transport
failures are permissive — same policy as notify_execution_layer:
warn-and-accept so EL flakes cant gridlock consensus.
Design notes:
* The EL call lives in the actors async handler, not in
store::on_block. Keeping the store layer sync preserves the
on_block_without_verification seam that fork-choice spec tests
rely on, and avoids fanning async fn across the whole import
pipeline. The validate_payload_with_el helper is private to the
actor.
* Own-built blocks (proposer path at line 453) bypass the pre-check
intentionally — they were either built from a real
engine_getPayloadV3 response (Phase 4, future) or via the
synthetic_payload helper, neither of which the EL needs to
re-validate.
* Pending children inherit validation: every block enters via
Handler<NewBlock>, so anything in pending_blocks already
passed the EL once. The cascade processing at line 610 stays
sync.
* The V3 calls last two params — expected_blob_versioned_hashes
and parent_beacon_block_root — are stubbed to vec![] and
H256::ZERO. Lean blocks dont carry blob transactions or a
beacon-root analogue yet; refine when those land.
Phase 4 next will replace synthetic_payload in build_block with the
real engine_getPayloadV3 response when an EL endpoint is configured.
Phase 4 of the M6 plan. Closes the proposer loop end-to-end when an EL
endpoint is configured:
interval 4 of slot N-1:
request_payload_id_for_next_slot(N-1) — if any of our validators
will propose at slot N, fire engine_forkchoiceUpdatedV3 with
PayloadAttributesV3 (correct slot-timestamp). Stash the returned
payload_id.
interval 0 of slot N:
take_prepared_payload(N) — pop the stashed id, call
engine_getPayloadV3, parse executionPayload, hand to
produce_block_with_signatures. build_block embeds it directly into
BlockBody.execution_payload; STFs process_execution_payload from
Phase 2d then enforces parent_hash + timestamp.
Fallback paths (any of which trigger the Phase 2d synthetic_payload):
* no EL configured
* we didnt queue a build (interval-4 path skipped)
* EL was syncing at interval 4 (payload_id = None on the FCU response)
* stashed slot doesnt match the proposal slot (we skipped a tick)
* engine_getPayloadV3 transport / parse failure
API touches:
* EngineClient::get_payload_v3 now returns ExecutionPayloadV3 directly
(extracts executionPayload from the envelope; drops blobsBundle
and blockValue for now).
* produce_block_with_signatures and the private build_block take an
Option<ExecutionPayloadV3>. None → synthesize.
* propose_block is now async (was sync) and accepts the optional
payload. The two on_tick call sites adjust accordingly.
* New BlockChainServer field pending_payload_id: Option<(u64, PayloadId)>.
What still wont work end-to-end until M6 is fully complete:
* notify_execution_layer still sends H256::ZERO for head/safe/finalized
(Phase 5). Until that goes, the EL has no idea what we consider the
canonical head and may stay in SYNCING.
* suggested_fee_recipient and prev_randao are hardcoded zero. A real
devnet needs CLI flags / RANDAO accumulation.
* Lean blocks still dont propagate blob transactions or a meaningful
parent_beacon_block_root.
These are the next phase-5/6/7 items in docs/plans/engine-api-integration.md.
All workspace tests pass; wire_smoke is sandbox-only.
… draft
Phase 7 of the M6 plan, closing out the in-repo work:
* build_block_embeds_provided_execution_payload — unit test in
crates/blockchain/src/store.rs. Confirms that when produce_block_with_signatures
is handed an engine_getPayloadV3-style ExecutionPayloadV3, build_block
embeds it verbatim (block_hash + full hash_tree_root match) instead of
falling back to synthetic_payload. The other M6-related units are
already covered: process_execution_payloads parent/timestamp guards in
state_transition (Phase 2d) and the ExecutionPayloadV3/Header SSZ + JSON
roundtrips in ethlambda-types (Phases 2a/2b).
* docs/plans/lean-execution-payload-schema.md — draft of the leanSpec
issue proposing the schema upstream. Frames the missing-EL-payload
problem, the canonical-V3 mirror choice, the genesis convention, and
enumerates ethlambdas reference commits as proof of feasibility.
File this verbatim on leanSpec when ready.
* docs/plans/engine-api-integration.md — Phase 7 section now reflects
what landed vs whats deferred. The two EL-mocked tests
(on_block_rejects_when_el_says_invalid /
notify_execution_layer_sends_real_hashes_after_first_block) sit behind
an EngineClient-trait-abstraction refactor that isnt worth blocking
on for this PR; tracked as follow-up in the same section.
21 blockchain lib tests pass (was 20). fmt + clippy clean.
…datedV3 Phase 5 of the M6 plan, retiring the H256::ZERO placeholder that MegaRedHand flagged on PR #367 (\"This is wrong\") and that started this whole expansion. notify_execution_layer now resolves each of the head, safe, and finalized Lean roots to its blocks body.execution_payload.block_hash via a small el_hash_at helper. At genesis the helper naturally rolls back to H256::ZERO (because BlockBody::default() carries ExecutionPayloadV3::default()), so the existing "first FCU is all zeros" startup behavior is preserved — no extra slot-0 special case needed. From the first non-genesis block onward the EL receives the hash it actually minted via engine_getPayloadV3 (Phase 4), so it can chain its own fork choice forward off blocks its already seen via engine_newPayloadV3 (Phase 3) instead of chasing zeros indefinitely. el_hash_at defensively falls back to H256::ZERO when a root is missing from storage. That shouldnt fire for head/safe/finalized (which are always present), but a torn write or pruning bug shouldnt crash the EL notifier; warn-on-failure semantics are preserved further down by the spawned forkchoice_updated_v3 call. Doc comment on notify_execution_layer updated; the in-body comment before the call site no longer claims "we send all-zero hashes". All workspace tests pass; wire_smoke is sandbox-only.
`crates/common/types/src/genesis.rs::GenesisValidatorEntry` was already on the dual-pubkey schema (`attestation_pubkey` / `proposal_pubkey`), but the validators-file parser in `bin/ethlambda/src/main.rs` was still on the older single-pubkey, role-by-filename layout. A node booting from a current `lean-quickstart` bundle would crash on `missing field \`pubkey_hex\`` before reaching the consensus stack — this commit aligns the two. New `AnnotatedValidator` carries both pubkeys and both privkey filenames on the same entry, matching the on-disk format the genesis generator emits today. `read_validator_keys` loses the role-by-filename indirection (classify_role + RoleSlots) since each entry is now self-describing — one entry per validator, both files explicit. No behavior change for the success path on the new format. The single- pubkey/by-filename path is dropped entirely; no client is producing that shape anymore. Net: +22 / -71.
…ml schema" This reverts commit dc25b97.
…lock-hash New CLI flag `--execution-genesis-block-hash` takes the ELs genesis block hash (32-byte hex, `0x`-prefixed or bare) and stores it in `state.latest_execution_payload_header.block_hash` at genesis-state construction. Without this seed the first `engine_forkchoiceUpdatedV3` carries an all-zero head, ethrex replies `SYNCING`, the build-mode FCU at interval 4 returns `payload_id = None`, and the chain never bootstraps a real EL payload. With the seed, the very first FCU references the ELs actual genesis block, ethrex accepts, and the get-payload / new-payload loop starts producing real execution payloads. The flag requires `--execution-endpoint` (clap enforces) and is parsed through the new `parse_h256_hex` helper which rejects wrong-length input. `State::from_genesis` is untouched — the seed happens in `fetch_initial_state` right after construction, so the dozen other test/internal call sites of `from_genesis` dont change. Operators find the value in the EL boot log (`Genesis Block Hash: 0xb923...` for ethrex).
After `propose_block` successfully runs STF on a freshly-built block, fire `engine_newPayloadV3` to the EL so it imports the payload as a real block in its chain. Without this call the EL knows the payload only as a candidate from `engine_getPayloadV3`, so subsequent FCU `head_block_hash` lookups against that hash bounce as SYNCING and the chain doesnt advance on the EL side. For received blocks the same call already happens in `Handler<NewBlock>`s EL pre-check (Phase 3 of M6); this commit closes the parallel gap for own-built blocks, which never re-enter the gossip-handler path. Fire-and-forget via `tokio::spawn` (~ms roundtrip, next FCU is 4s away). INVALID/error verdicts are logged but dont reverse the local `process_block` — by design, mirroring `notify_execution_layer`s "consensus must keep running regardless of EL state" stance.
Three intertwined changes that together unblock real `engine_newPayload`
acceptance on ethrex and end the "FCU always carries ZERO head" loop:
1. **V4 newPayload.** Add `engine_newPayloadV4` to ethrex-client and
switch both call sites (Phase 3 receive-side import check; Phase 5
follow-up own-built notify) from V3 to V4. V4 takes the same
`ExecutionPayloadV3` shape plus an `executionRequests` parameter
(EIP-7685 system contracts — empty for Lean blocks). ethrex rejects
V3 with `-38005 Unsupported fork: Prague` once the payload timestamp
crosses `pragueTime`, which our devnet genesis sets at 0. The
capability advertisement is updated to include V4.
2. **Genesis BLOCK body seed.** The previous commit seeded
`state.latest_execution_payload_header.block_hash` (which drives
STFs parent_hash check) but `el_hash_at` — Phase 5s FCU
`head_block_hash` source — reads
`block.body.execution_payload.block_hash`, and the genesis blocks
body was synthesized as `BlockBody::default()` (all zero) regardless
of any state seeding. `fetch_initial_state` now also builds an
explicit genesis BlockBody whose `execution_payload.block_hash`
equals the seed, updates the headers `body_root`, and uses
`Store::get_forkchoice_store` (which persists the body) instead of
`from_anchor_state` (which assumed the empty body). With both seeds
in place, the very first FCU at interval 0 of slot 0 carries the
real EL genesis hash.
3. **Build-mode FCU uses real hashes.**
`request_payload_id_for_next_slot` (Phase 4 — interval-4 FCU+attrs
that asks the EL to start building) was hardcoding the
`ForkChoiceState` triplet to ZERO, so even with everything else
correct the EL would never recognize our head and return
`payload_id = None`. Factored both that path and
`notify_execution_layer` onto a shared
`current_el_forkchoice_state()` helper.
After these three: at slot 0 interval 4 ethlambda fires
FCU+attrs(head=EL genesis hash); ethrex accepts, returns a payload_id;
at slot 1 interval 0 ethlambda fetches via `engine_getPayloadV3`,
Three intertwined fixes that finally close the loop: ethrex now
receives real, non-zero execution payloads from ethlambda every slot
and reports the new head back via FCU, slot-by-slot.
1. **Anchor consistency at genesis seed.** `Store::get_forkchoice_store`s
`anchor_pair_is_consistent` requires `block.state_root` to equal
`state.hash_tree_root()` (with header.state_root zeroed) exactly —
`ZERO` is not accepted. `fetch_initial_state` now zero-passes
`latest_block_header.state_root`, computes the canonical state root,
stamps it on both the state header and the genesis block before
calling get_forkchoice_store.
2. **Build-mode FCU carries real EL hashes.** `request_payload_id_for_next_slot`
(interval-4 FCU+attrs that asks the EL to start building) was
hardcoding head/safe/finalized to ZERO. Refactored both this path
and `notify_execution_layer` onto a shared
`current_el_forkchoice_state()` helper that reads via `el_hash_at`,
so the EL sees the same head whether the call is heartbeat or
build-mode.
3. **V4 + V5 newPayload / getPayload.** Added typed wrappers for both
on `EngineClient`; advertised in `ETHLAMBDA_ENGINE_CAPABILITIES`.
The actor calls V5 because ethrex on main implements V5 for
Amsterdam-era (EIP-7928 BAL) and rejects V4 with
`-38005 Unsupported fork` once `timestamp >= amsterdamTime`.
The genesis JSON must activate Amsterdam at 0 for V5 to apply.
Confirmed end-to-end against ethrex main: per-slot FCU carries
0xb923…c7af (ethrexs genesis), interval-4 FCU+attrs returns a real
`payload_id`, getPayloadV5 returns a payload ethrex minted, newPayloadV5
on the proposed block lands cleanly, next slots FCU advances to the
new ethrex-minted block_hash. Real EL chain advancing in lockstep.
…oadV5 d0c5b72 switched the build path (engine_getPayload + the FCU-then-build chain) to V5 but missed the import-validation path (`validate_payload_with_el`) and the post-propose self-notification path. Both kept calling new_payload_v4 while every surrounding log line, comment, and the commit message itself claimed V5. This worked against the demos ethrex genesis only because that genesis activates forks through Osaka @0 with no `amsterdamTime` set — without an Amsterdam timestamp the EL doesnt gate V4 yet. The moment a paired EL activates Amsterdam, ethrex would have returned `-38005 Unsupported fork: Osaka/Amsterdam` on those two paths and our import-validation would silently flip to the permissive "accepting block" branch. - validate_payload_with_el now calls new_payload_v5; surrounding doc comment rewritten to describe Amsterdam-era V5 (BAL on the payload, same JSON-RPC param shape as V4). - The own-built fire-and-forget notification after propose_block now calls new_payload_v5; surrounding comment updated to reference engine_getPayloadV5 (the version that minted the candidate) and engine_newPayloadV5 (the version that promotes it). - Module docstring and ETHLAMBDA_ENGINE_CAPABILITIES doc-comment in ethrex-client/src/lib.rs both now reflect V3/V4/V5 across newPayload and getPayload, with the version-selection rule (timestamp against the EL fork schedule) made explicit. - cargo fmt picks up three .await line breaks in get_payload_v3/v4/v5 that d0c5b72 introduced but didnt reformat. Caught by an automated three-agent code review; two of the agents flagged the V4 calls independently as a functional drift between the code and the commit message.
# Conflicts: # Cargo.lock # crates/blockchain/src/lib.rs # crates/blockchain/src/store.rs # crates/blockchain/state_transition/src/lib.rs
…&State so the blockchain actor can reuse the STF slot-timestamp formula when preparing PayloadAttributes; update the three call sites. Drop engine_exchangeCapabilities from the advertised ETHLAMBDA_ENGINE_CAPABILITIES list per the execution-apis spec (the method must not list itself), and refresh the V3 doc comments to V5 to match the getPayloadV5/newPayloadV5 calls the actor makes.
…icts in block_builder.rs (keep EL payload + select-timing), blockchain/lib.rs (keep both attestation-coverage and EL-client actor fields/params), and bin/ethlambda/main.rs (combine multi-URL checkpoint sync with execution-genesis-hash seeding, keep build_execution_client, adopt main's Result-returning read_hex_file_bytes). Add latest_execution_payload_header to the new finalization test's State literal.
…Option-C payload pipeline (per-slot FCU, interval-4 build request, interval-0 getPayloadV5/embed/newPayloadV5, newPayloadV5 revalidation on import, FCU on real block hashes) instead of the obsolete scaffolding note that claimed Lean blocks carry no payload and the EL only sees SYNCING against zeros.
…e issue: target the lstar fork containers, reflect the landed Option-C pipeline (getPayloadV5/newPayloadV5, build-mode FCU, import revalidation) instead of the old scaffold framing, add an Engine-API version note separating the V3 container shape from the V5 method version, add a 'Decision requested' section (accept schema and pin field order, genesis EL-hash convention, regenerate fixtures), and refresh the constants and reference-implementation tables to the as-shipped files.
…e blockchain actor can be tested against a mock execution layer. The actor now holds Option<Arc<dyn ExecutionEngine>> instead of a concrete EngineClient; EngineClient implements the trait by forwarding to its inherent methods, and build_execution_client returns the boxed trait object. Adds async-trait as a workspace dependency. Unblocks the two EL-dependent tests previously deferred for lack of a mockable engine: validate_payload_with_el now has coverage for the INVALID/INVALID_BLOCK_HASH drop verdicts, the VALID/SYNCING/ACCEPTED accept verdicts, permissive behavior on an EL roundtrip failure, and the no-EL-configured case; el_hash_at is covered for resolving a block's real execution_payload.block_hash after import with ZERO fallback for zero/unknown roots.
…recipient now comes from an optional key in validator-config.yaml's config block (zero default burns rewards; EL-paired nodes warn at startup). parent_beacon_block_root follows the lean-parent-root convention: the proposer's build-mode FCU commits the payload to its head root, and validators pass block.parent_root to newPayloadV5 — deterministic on both paths, mirroring EIP-4788, so the EL block hash commits to the Lean chain. The pending payload stash now carries the build-head root and take_prepared_payload discards the id if the head moved before proposal, since the prepared payload's parent_hash and embedded beacon root would be stale. prev_randao stays zero until Lean defines a RANDAO mix. Adds four tests (beacon-root passthrough, head-change discard, happy path, slot mismatch) and resolves the fee-recipient and beacon-root open questions in the leanSpec proposal doc.
…hrex makes engine_newPayloadV5 require the EIP-7928 block_access_list (Amsterdam), which is off by default and which getPayloadV5 doesn't even return, so V5 can't round-trip against a default EL. V4 is the pre-Amsterdam, no-BAL path: the ExecutionEngine trait and the actor's propose/import hooks now use get_payload_v4/new_payload_v4. The inherent V5 client methods stay for a future fork-aware version selection. Verified live against ethrex v15 with a Prague-level genesis: payloads build, import, and finalize on both sides with the configured feeRecipient present.
…demo/run.sh starts a single ethlambda validator paired with one ethrex node over the Engine API (build, start both, print what to show; 'run.sh stop' to tear down), reading the EL genesis hash from ethrex's log so nothing is hardcoded. Ships a Prague-level EL genesis (chainId 9, pre-Amsterdam, system contracts only) and a README; adds a 'make run-el-demo' target. The dual-key lean genesis bundle is a documented prerequisite (generated via lean-quickstart), not committed.
…ngine trait version-agnostic (get_payload/new_payload) so the V4/Prague pin and the lean no-blobs/no-requests policy live solely in the EngineClient impl; pass payloads by reference to drop a full clone per imported block, and take the executionPayload subtree out of the getPayload envelope instead of cloning it. Move the genesis EL-hash anchor-pair seeding from main.rs into State::from_genesis_with_el_hash where its invariants are documented once. Make engine-method references in doc comments version-neutral, consolidate the two hex CLI parsers into a generic parse_fixed_hex, remove the redundant From<&ExecutionPayloadV3> header impl, and make the list serde modules private.
…oving the EL integration into new modules: the actor's Engine API hooks to blockchain/src/el_integration.rs, their mock-EL tests to src/execution_engine_tests.rs, the STF's process_execution_payload plus slot-timestamp helpers to state_transition/src/execution_payload.rs (re-exported, public API unchanged), and the EL genesis anchor seeding to types/src/el_genesis.rs. Also apply the simplification review: version-agnostic ExecutionEngine trait methods (get_payload/new_payload) so the V4/Prague pin and the lean no-blobs/no-requests policy live only in the EngineClient impl, payloads passed by reference (drops a clone per imported block), Value::take instead of clone when extracting getPayload responses, version-neutral engine-method doc comments, a shared parse_fixed_hex helper for the CLI hex parsers, and removal of the redundant From<&ExecutionPayloadV3> impl and pub on internal serde modules.
…ion.rs, execution_engine_tests.rs, execution_payload.rs, and el_genesis.rs were left untracked by it.
…e the unused new_payload_v3/v5, get_payload_v3/v5, and get_client_version_v1 wrappers (and inline the helpers that only served them into the V4 pair), advertise only forkchoiceUpdatedV3/newPayloadV4/getPayloadV4 in engine_exchangeCapabilities, and drop the smoke example, which the engine-api-demo script and the wire_smoke tests supersede. Other method versions come back alongside fork-aware version selection when a second fork window is needed.
…-1559 transfers from the well-known hardhat dev account (now prefunded in genesis-el.json) using an ephemeral uv-run eth-account, and submits them to ethrex's HTTP-RPC; they land in the next slot's payload and appear inside the Lean block. README gains a 'With transactions' section showing the receipt on the EL side and the raw transactions inside the Lean block's executio payload.
merged-proof signature model.
Main replaced the per-block signature model (SignedBlock.signature:
BlockSignatures { attestation_signatures, proposer_signature }, with
per-attestation AggregatedSignatureProof) with a single merged
SignedBlock.proof: MultiMessageAggregate built from TypeOneMultiSignature
components (lean-multisig devnet5 / verify_type_2). The EL integration
predated this, so the merge is a port rather than a marker cleanup.
Conflict resolutions (take main's signature model, keep the branch's EL
additions, combine where both are additive):
- block.rs: drop the now-removed XmssSignature import, keep the
execution_payload import; BlockBody keeps its execution_payload field.
- state_transition/lib.rs: keep both error-variant sets (EL payload
parent_hash/timestamp checks + main's aggregation-bits bounds errors).
- block_builder.rs / store.rs::produce_block_with_signatures: keep the
branch's execution_payload param alongside main's TypeOneMultiSignature
return types.
- lib.rs: keep both actor fields (execution_client / suggested_fee_recipient
/ pending_payload_id and main's sync_status); thread execution_payload
through produce_block_with_signatures under main's type_one_proofs
naming; keep both #[cfg(test)] modules (execution_engine_tests + the
inline sync-status tests).
- signature_spectests.rs: keep both FIXTURES_AWAIT_M6_REGEN and SKIP_TESTS;
run() uses both.
- ssz_spectests.rs: take main's side, skipping SignedBlock / BlockSignatures
/ AggregatedSignatureProof / SignedAggregatedAttestation (the branch's
referenced BlockSignatures / AggregatedSignatureProof types no longer
exist).
Silent breakage fixed beyond the markers:
- execution_engine_tests.rs built SignedBlock { signature: BlockSignatures
{..} } with types removed on main. Ported to proof:
MultiMessageAggregate::default(), dropped the dead AttestationSignatures /
blank_xmss_signature imports, and added the new sync_status field to its
test_server literal.
Dropped the obsolete verify_signatures_rejects_participants_mismatch test:
it asserted StoreError::ParticipantsMismatch, which main removed along with
the old BlockSignatures model.
Verified: cargo check/clippy (-D warnings)/fmt clean; cargo test --workspace
--release green (the two ethrex-client wire_smoke tests fail only under the
command sandbox, which blocks TcpListener::bind, and pass outside it).
- Remove the `hex_address` serde module: it was byte-for-byte identical to `hex_bytes_fixed` specialized to 20 bytes. Point the three `[u8; 20]` fields (Withdrawal.address, ExecutionPayloadV3/Header.fee_recipient) and ethrex-client's PayloadAttributesV3.suggested_fee_recipient at `hex_bytes_fixed`; the JSON wire output is unchanged. - Drop EngineClient::with_http_client: dead code, never constructed anywhere. - Drop EngineClient::endpoint(): it existed only for a unit-test assertion; the build-success test now relies on the constructor's Result. Net -42 lines, no behavior change. fmt/clippy clean; types and ethrex-client unit tests pass.
… it. The body of Handler<NewBlock> (EL pre-check then on_block) moves to BlockChainServer::import_gossiped_block in el_integration.rs, so the INVALID-verdict drop path can be exercised against the mock ExecutionEngine without standing up a real actor Context or a live TCP server — the coverage gap Phase 7 deferred. Adds import_gossiped_block_drops_block_on_invalid_verdict: an INVALID EL verdict leaves the block out of the store and the EL is consulted with the block's parent_root as the beacon root. Factors the shared SignedBlock construction into signed_block_with_parent (reused by the existing insert_block_with_payload_hash helper).
…mation" to the as-shipped state: Option C / M1–M7 implemented and verified live against ethrex v15, pinned to the V4 (Prague) methods, with the remaining upstream-gated items (leanSpec schema filing, fork_digest bump, fixture regen) called out.
First step toward integrating ethrex in-process as a library crate (counterpart to #367, which integrates ethrex out-of-process over the Engine API). This is a de-risk spike proving ethrex embeds as an unmodified pinned git dependency and compiles cleanly inside the workspace before any functional code is written. - New crate crates/net/ethrex-engine (ethlambda-ethrex-engine) linking the three core ethrex library crates (ethrex-common, ethrex-storage, ethrex-blockchain), pinned to rev de9b249b. No functionality yet; ExecutionEngine impl follows. - ethrex ships its own EVM (levm) and devp2p, so there is no revm/libp2p/ ethereum_ssz clash; lockfile resolves with zero unification conflicts. The heavyweight ethrex-rpc crate is intentionally not a dependency. - docs/plans/ethrex-inprocess-poc.md documents the design and phased plan.
Implement EthrexEngine, the #367-independent core of the in-process ethrex integration. It bootstraps an in-memory ethrex store from an EL genesis and drives the execution layer directly via ethrex library calls — no Engine API or JSON-RPC hop. - from_genesis: in-memory Store + Blockchain from a parsed Genesis - build_block: create_payload + build_payload on top of the canonical head (mirrors forkchoiceUpdated(build)+getPayload synchronously) - import_block: add_block (execute + persist) - set_forkchoice: apply_fork_choice to advance the canonical head - head_hash / head_number accessors - EngineError wraps the underlying ethrex Store/Chain/ForkChoice errors Speaks ethrex-native types (Block, H256); the ethlambda ExecutionPayloadV3 ⇄ Block conversion and the ExecutionEngine trait impl are layered on when this crate stacks on the Engine-API branch (#367). Proven by the roundtrip test: genesis → build → execute → fork-choice → head advances to block 1.
pablodeymo
changed the title
Absorbs the out-of-process Engine-API integration (#367) so the full EL pipeline — ExecutionEngine trait, ExecutionPayloadV3 embedded in BlockBody, process_execution_payload STF, per-slot forkchoiceUpdated, newPayload on import and on own-built blocks — lands on this branch alongside the in-process ethrex-engine crate. Reconciled against 55 commits of main evolution since #367 branched: - Block-production timing (Option A): main builds+publishes the next slot's block one interval early (interval 4). The EL payload is now built inline and synchronously at interval 4 via build_execution_payload (build-mode forkchoiceUpdated + getPayload), replacing #367's request-at-4 / fetch-at-0 stash. Dropped pending_payload_id, request_payload_id_for_next_slot, take_prepared_payload. - BlockChainConfig gained execution_client + suggested_fee_recipient (main had folded actor params into the config struct); CliOptions moved to cli.rs and gained the --execution-* flags there. - Signature type rename TypeOneMultiSignature -> SingleMessageAggregate applied to the merged block-builder / store paths; build_block and produce_block_with_signatures now thread both proposer_config and the optional execution_payload. - StateDiff carries latest_execution_payload_header so reconstructed states keep the EL block-hash chain (added the field + Eq/PartialEq on ExecutionPayloadHeader). Test-only State literals updated for the new field. Workspace builds, clippy -D warnings is clean, fmt clean; blockchain (58), state-transition (108), storage (67) unit tests and the ethrex-engine roundtrip pass. The in-process ExecutionEngine impl + --execution-mode selector are the next step.
…rocess
Make the in-process EthrexEngine a drop-in ExecutionEngine so the blockchain
actor drives it through the same call sites as the out-of-process EngineClient.
- conversion.rs: ExecutionPayloadV3 <-> ethrex Block mapping reimplemented
against ethrex-common (no ethrex-rpc dependency), mirroring ethrex's
into_block/from_block: transactions via RLP decode_canonical/encode_canonical,
transactions/withdrawals roots via ethrex helpers, base fee as big-endian
u256<->u64, Cancun/V3 header shape (Prague+ fields round-trip as None).
- impl ExecutionEngine for EthrexEngine:
* forkchoice_updated_v3: apply_fork_choice best-effort; in build mode
create_payload+build_payload, cache the converted payload keyed by the
derived payload id (BuildPayloadArgs::id), return the id.
* get_payload: drain the cached payload by id.
* new_payload: convert + add_block; Ok->VALID, Err->INVALID (never errors
the call, matching the permissive consensus posture).
- CLI: --execution-mode {external,inprocess} (default external) + --el-genesis;
build_inprocess_engine bootstraps EthrexEngine::from_genesis_path.
- Tests: added a trait-path roundtrip (forkchoiceUpdated build -> getPayload ->
newPayload) that executes the built payload back through the EL. Switched the
test genesis to Cancun-only: a Prague genesis requires a requests_hash the
Cancun-shaped ExecutionPayloadV3 cannot carry, so newPayload rejected the
block — the V3 PoC pairs with a Cancun EL genesis.
cargo build --workspace, clippy -D warnings, and fmt are clean;
ethrex-engine tests (inherent + trait roundtrip) pass.
pablodeymo
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description / Motivation
Integrates ethrex in-process as a library crate — the execution layer runs embedded, driven by direct library calls, with no separate EL binary and no Engine API / JSON-RPC hop. Selectable at runtime alongside the existing out-of-process path via
--execution-mode {external,inprocess}.This PR absorbs and supersedes #367 (the out-of-process Engine-API integration): #367's
ExecutionEnginetrait,ExecutionPayloadV3-in-BlockBody,process_execution_payloadSTF, and slot-loop wiring are the substrate; the in-processEthrexEngineis a second implementation of the same trait.What Changed
Dependency foundation (phase 0):
ethrex-common/ethrex-storage/ethrex-blockchainas pinned git deps (rev de9b249b). ethrex ships its own EVM (levm) and devp2p, so there is norevm/libp2p/ethereum_sszclash; the heavyweightethrex-rpccrate is deliberately not a dependency.crates/net/ethrex-engine—EthrexEngine: bootstraps an in-memory ethrexStore+Blockchainfrom an EL genesis and implementsExecutionEngine:forkchoice_updated_v3:apply_fork_choice; in build modecreate_payload+build_payload, caching the payload under its derived id.get_payload: returns the cached payload.new_payload: converts +add_block(VALID/INVALID).conversion.rs:ExecutionPayloadV3 ⇄ ethrex Blockreimplemented againstethrex-common(RLP tx, roots, base-fee, Cancun/V3 header shape).Merge reconciliation vs current main (main moved 55 commits since #367): block production now happens one interval early (interval 4) — Option A: the EL payload is built inline and synchronously there (
build_execution_payload), replacing #367's request-at-4/fetch-at-0 stash.StateDiffcarrieslatest_execution_payload_headerso reconstructed states keep the EL block-hash chain. Adapted to main'sResult-returning store API,BlockChainConfig, andSingleMessageAggregaterename.CLI:
--execution-mode {external,inprocess}(defaultexternal) +--el-genesis <path>.How to Test
builds_executes_and_advances_head— inherent API: genesis → build → execute → fork-choice → head advances.engine_trait_build_get_new_payload_roundtrip— fullExecutionEnginetrait path through the conversion:forkchoiceUpdated(build)→getPayload→newPayloadaccepts the built payload.Notes
requests_hashthe Cancun-shapedExecutionPayloadV3cannot carry, sonewPayloadrejects the block (caught by the trait-path test).--execution-mode;externalbehavior is unchanged.