Typed peer-to-peer RPC over iroh. You define a service; callers get async methods. Everything underneath — NAT traversal, framing, schema versioning, authorization — is invisible from the call site.
keylink::service! {
service Inventory;
rpc GetItem(sku: String) -> Option<Item>;
rpc Reserve(sku: String, qty: u32) -> ReserveOutcome;
tell Note(line: String) -> (); // fire-and-forget, no reply
recv Fetch(sku: String) -> Datasheet; // a file streams down beside the reply
send Upload(name: String) -> Stored; // and up beside the request
blob Manual(sku: String) -> Sheet; // an address; iroh-blobs moves the bytes
}let node = keylink::Node::bind().await?;
let client = inventory::Client::connect(&node, peer_id).await?;
let item = client.get_item("ADU-014".into()).await?; // this is the whole APINo IP address, no port, no framing, no serialization call, no ALPN string, no version number appears in your code.
Status: early, and nothing here has ever run in production. The API will move. Read Status before you build on it.
Not on crates.io yet, so a git dependency is the only way in. keylink needs Rust 1.91 or later, which is the floor its own dependency graph imposes.
[dependencies]
keylink = { git = "https://github.com/pedro-pscunha/keylink" }default = ["blocking", "fs", "env", "persist-file"]. Build with
--no-default-features for a browser: all four need an OS facility a tab does
not have. Three say so with keylink's own compile_error! rather than failing
deep inside a dependency, and persist-file stops earlier still, inside tokio.
| feature | default | what it adds |
|---|---|---|
blocking |
yes | The synchronous client, and Node::enter. Needs OS threads. |
fs |
yes | Body::from_file. Needs a filesystem. |
env |
yes | Node::bind_from_env and Allowlist::from_env. Needs a process environment. |
persist-file |
yes | persist, plus keylink-persist's local-file adapter. |
persist |
no | The Persist observer over the port alone. No adapter, and it compiles for a browser. |
web |
no | Body::from_web_file and Body::from_web_stream. Meaningful on wasm32-unknown-unknown alone. |
blobs-fs |
no | Blobs::persistent, which pulls redb and reflink-copy. Blobs::memory needs neither, and is the one to reach for first. |
| crate | what it is |
|---|---|
keylink |
The library. service!, Node, Server, the generated clients, actors and the registry. |
keylink-persist |
A port for actor state that outlives its process, and its local-file adapter. |
keylink-http |
The HTTP/JSON gateway, for callers that speak neither Rust nor the Keylink protocol. |
keylink-openapi |
OpenAPI 3.1 and JSON Schema for a gateway surface. Opt-in, and its schema dependency reaches neither crate above. |
demo |
The two-machine walkthrough under Try it. Not published. |
fixtures/imported-ledger |
A service defined in one crate and consumed by another, so the generated seam is tested across a crate boundary. Not published. |
keylink-http/README.md— the gateway's route table, its error codes and its limitskeylink-openapi/README.md— generating an OpenAPI 3.1 document from a mounted surfaceCONTRIBUTING.md— the gate, the wire-contract pin, the public API rules, and when--no-verifyis legitimatemise run docsbuilds the rustdoc, which is the complete API reference- Examples — every one of them is run by
mise run gate, so each is the shape that works rather than the shape a document remembers. The authoritative list is[tasks.examples]inmise.toml; commands are under Try it.
| example | what it is about |
|---|---|
reliable_call |
effectively-once, composed from a key, a Peer and attaching_for |
payment_approval |
#[defer], Allowlist::only, and both durability ports of one store |
replay_guard |
the retry a memory ledger cannot see, and Applied inside the state |
durable_ledger |
Persist as an observer, storing inside every turn |
firmware_depot |
blob, send and tell in one service |
cold_store |
a blob store on a disk, and the flush that commits it |
closure_share |
one grant for a whole tree, have/want, and taking the grant back; one process or two machines |
match_room |
a keyed registry, two callers on one parked reply, and a feed clock |
substation |
supervision, and what a caller parked across a restart is told |
trading_desk |
one scenario, run locally and over QUIC, against two backpressure policies |
foundry |
the compose patterns: pool, factory, keyed topic, names, group, schedule |
gatehouse |
admission, per-method rules from TOML, and revoking a live connection |
toll_booth |
middleware, and the two budgets that answer alike and mean opposite things |
control_room |
events, an inspector and a console reading a node over QUIC |
noticeboard |
pushing to peers, which is the direction a request cannot travel |
bridge |
driving a service from a wire keylink did not design, over one hand-written codec |
toolroom |
the synchronous API, from a main with no runtime in sight |
waypoint |
a hand-written Dial, and a handle that finds a peer that moved |
wire_contract |
what moves the ALPN hash, what does not, and what a move costs at dial time |
fault_lines |
eight error types, one condition, and the vocabulary above them |
inventory |
the HTTP gateway on one singleton surface |
gateway |
every served surface at once, plus the OpenAPI documents |
The RFCs carry the design reasoning, including the decisions that were reversed.
- RFC-01 — iroh-blobs in keylink
- RFC-02 — runtime mount and unmount of protocols
- RFC-03 — keylink in the browser
- RFC-04 — an actor model for keylink
- RFC-05 — idempotency: what settles a key, and what does not
- RFC-06 — observers, and durable state as an adapter, plus an implementation brief that shares its number
- RFC-07 — the turn model: one dispatch path, a typed mailbox, two inputs
- RFC-08 — serving-side idempotency: which methods accept a key, and where its reply lives
Peers are dialed by Ed25519 public key. iroh hole-punches directly when it can and relays when it cannot; neither side names a location.
Every call takes its own QUIC stream, so calls run in parallel and a slow handler blocks nobody. There is no client mutex, no request-id correlation and no cancellation bookkeeping — dropping the stream cancels the call.
A tell travels on a unidirectional stream and returns a proof of admission —
Queued locally, Transmitted remotely — never the declared type: there is no
reply stream to decode one from. The Ok means the
bytes reached the transport and nothing more — not delivery, not handling, and
no order against a call. It does not accept an idempotency key either, and
the compiler is what says so: with_idempotency hands back an
IdempotentClient, and a tell is not among its methods. The ledger answers a
repeat with the recorded reply and a tell has no reply to record, so a key
attached to one could only be discarded. There is no reliable tell.
Anything that has to arrive is an rpc.
A body is not framed: its extent is the rest of the stream and its flow control is QUIC's. A 4 GiB file costs zero framing bytes and one chunk of peak memory. A body that ends early is an error, never a short file.
A blob method returns a Grant; iroh-blobs moves the bytes, with verified
resume and deduplication. keylink owns the mount, because the mount iroh-blobs'
own documentation teaches serves every hash to every peer (and, up to iroh-blobs
0.103.0, accepted writes too).
Returning a grant to a peer records it for that exact (peer, root) pair,
and a blobs request is admitted only against a grant — so every entry was
created by an RPC that already passed its service's allowlist and layers.
A Grant is one blob (id.into()) or a closure (seq.into()). A closure is a
hash sequence built with Blobs::add_seq, and it is one grant that covers
the sequence and every blob it lists, however many that is. The peer reaches a
member through the root, by position; a member asked for by its own hash is
refused unless it was granted on its own. A closure is one level deep, so a
tree with sub-sequences is flattened into one sequence that lists every blob.
A closure authorizes every hash it lists: list only what the recipient may
read, and never hashes a peer supplied.
handle.fetch_into(&store) fetches a closure on one dial: the root, then only
what the store lacks of the members, 4096 members to a window. It answers with a
Fetched { read, kept }: the payload bytes read from the holder, and the verified
bytes the store already held and did not ask for. Resume goes down into a member:
the store's bitfield says which 16 KiB chunk groups it holds, and only the others
are requested, so after a cut no verified group is sent twice. A 4 MiB member cut
after exactly 100 groups resumed reading 2,556,004 of its 4,194,404 bytes and
keeping 1,638,400. A request carries at most 64 KiB of ranges; a blob, a closure's
root, or a window of members with more fragments than that is cut across several
requests and never refetched whole. kept counts each distinct blob once for the
whole call and never counts bytes the call itself read (a 32-byte key and table overhead
for each distinct member, O(distinct members), where planning memory is not).
Blobs::missing (and
handle.missing(&store)) answer what is lacking, from the fetcher's own store.
Blobs::reader(id) reads a blob the node holds, as the same BlobReader a
holder's blob opens as: pieces of at most a chunk group, a 256 KiB payload window
at a time (a little over twice that in memory while a window is verified), each
window verified against the address before any of it is handed out.
A blob the store does not hold in full, absent or part way, is refused with
Error::BlobNotHeld before a byte is read, and one whose stored data or hashes
no longer verify fails with Error::BlobCorrupt at the window where they differ.
A store that cannot be read at all is Error::Io, not corruption. (The file
store's own export already validates down from the address and refuses data that
no longer matches; the reader's decode is a second, independent verification.)
Blobs::read_to_bytes(id, limit) is the same for a small blob into memory.
A closure has no limit on how many members it lists. Planning memory and
request size are bounded, the work is not linear: iroh-blobs 0.104 walks offsets
one at a time on both ends, so a window starting at member k costs k steps and
a whole closure about N^2 / 4096 of them. A 300,000-member closure with
alternating gaps takes 74 s from a cold start on an idle machine and 216 to 283 s on a busy one (mise run closure-300k, not in
the gate), 12,289 members take 8 s (in it). A blob listed twice in one window is
requested once. The fetcher plans each
window of 4096 members from that slice of the root alone, so what it holds in
memory is a window's, and a window's request has at most 4096 entries, far under
iroh-blobs' message limit (a compile-time check). The root is checked for a whole
number of addresses before it is downloaded, from the size the holder proves with
the last chunk (BlobFault::NotASequence). The holder refuses a request whose
furthest run starts past the members the root lists, read from the root's size
(one metadata read), because a request can name an offset near u64::MAX in a
few bytes and iroh-blobs walks offsets one at a time. A run that starts at a
member and goes on is fine, so an open-ended request for a whole closure passes.
Missing has no serde, so no rpc carries a list of what a store lacks to a peer
unless a handler converts it by hand.
A grant lasts until it is revoked. Blobs::revoke(peer, root) and
Blobs::revoke_everywhere(root) refuse the next request; a request already
admitted finishes. Blobs::revoke_peer(peer) also closes the peer's live
blobs connections, so a transfer in flight stops, and it refuses the grant of a
call whose handler was already running when it landed
(RemoteError::GrantsWithdrawn). Bytes already delivered stay with the
fetcher. Remove a peer from admission first, then call revoke_peer:
removing it from an allowlist does not touch its blob grants. Both DynamicAdmission and Trust close every RPC connection of a removed peer. For DynamicAdmission, apply the admission change before calling revoke_peer. For delegated authority, bind Trust::with_blobs(&blobs) before serving calls, using the same Blobs handle as the node's services; Trust then performs both steps.
The ledger holds at most GRANT_LIMIT grants (172,032) across every peer, and
Blobs::set_grant_limit changes it. A grant past the limit is answered
RemoteError::GrantsExhausted { limit }, and nothing is evicted to make room. A
peer has an entry only while it holds a grant, so one count bounds both tables,
and a table is rebuilt when its length falls under half the most it has held, so
what a revoke leaves allocated stays bounded. The cost is measured with a
counting allocator, on aarch64 macOS: the dense shape (a few peers, many grants
each, grant_allocation) is 64.2 bytes per grant filling to 100,000 and at worst
128.5 under trade-one-for-one churn; the sparse shape (one grant per peer) is
386.3 at 1,000 peers and 320.7 at 100,000. The default limit's size, about 64 MiB,
uses the worst of these, the sparse 386.3. Grants die with the process.
The closure_share example runs it end to
end, in one process or between two machines.
postcard is not self-describing: swapping two same-shaped enum variants silently changes their meaning. A service's ALPN is <crate>/<Service>/<version label>/<shape hash>. The label names a version; the derived hash still decides whether two declarations agree. A field added or renamed anywhere in the contract moves the hash. Renaming a type or moving its module does not.
With no common contract, the dial returns Error::SchemaMismatch(Box<Mismatch>). An admitted peer answers a per-service probe in the same handshake, so the error names both windows. A probe that does not finish within five seconds is closed and reported as unknown. If both sides claim one label with different shapes, it names that too. A peer that serves neither a matching version nor the probe is reported as unknown, not as a guessed version.
Declare version 3; immediately after service Inventory;; leaving it out means version 1. Add previous 2 { hash frozen::inventory::VERSION.hash(); keyed_hash frozen::inventory::KEYED_VERSION.hash(); ... } after every current declaration. Repeat the original wire declarations in their original order, including attributes: serde metadata can change the schema hash, and configuration attributes gate the declarations and conversion arms together. A shipped application pins both literal hashes read from that release's VERSION and KEYED_VERSION; examples in this repository name the frozen module's VERSION and KEYED_VERSION so changes to keylink's own wire remain free.
Mark each changed or removed method changed. Implement the generated version_2::Convert for version_2::Step: all four request/reply upgrade/downgrade functions are required. Unchanged methods are carried automatically. More blocks extend the window through adjacent conversion steps, up to 64 versions including current. An oversized generated window fails to compile; hidden manual construction also refuses the 65th entry. Blocks must descend by label, and a frozen singleton or keyed declaration that no longer matches its pin fails to compile. Use crate:: paths inside blocks rather than super::, since the frozen types live one generated module deeper. See demo/src/main.rs for the complete implementation.
One handler serves every version. Authorization, layers and the idempotency ledger see only current types. Blob grants follow the address carried by the negotiated reply. The server selects its newest common version in one handshake. client.version() reports it; inventory::window() reports the build's supported versions. A newer client talking to an older server converts both ways. Request conversion failure is Error::UnsupportedRequest: no stream opened and nothing was sent. TellError::Unsupported has the same pre-send guarantee. Error::UnsupportedResponse means a reply arrived but could not be converted: the call was sent and may have executed. Server request conversion fails before policy, key claim and dispatch as RemoteError::UnsupportedRequest; a server reply conversion failure is RemoteError::UnsupportedResponse and does not promise that nothing ran. Use idempotency before retrying a reply-conversion failure.
Conversions own the policy: adding a field needs an explicit upgrade default, and downgrading may lose it. Bodies and blob content are application bytes and are not versioned. Blob metadata may convert, but its complete Content identity (blob or closure kind and root) must not change: a live conversion that changes it fails as UnsupportedResponse, and such a stored reply becomes Forgotten during recovery. Durable replies carry their full contract version (label and hash): recovery upgrades a known older reply to the current shape, and a malformed, unknown or unconvertible reply becomes Forgotten, never a newly executable call. Older connections run the downgrade again on replay; use deterministic conversions when byte-for-byte replies matter.
An idempotency key is bound to the received message bytes in the version the caller spoke, not to a re-encoding of the upgraded request. The binding also includes the negotiated version label and authorizing principal/holder. A retry over another negotiated label returns IdempotencyRefusal::RequestMismatch, even when its request bytes happen to be identical. Peer can therefore return this refusal after an upgrade negotiates a newer label; it does not rerun the operation. Reconcile the original operation rather than blindly issuing a new key. Same-version retries still share the service's ledger across connections and restarts.
Keylink's own envelope and errors are in every hash. Until its wire is stable, two binaries must use the same keylink wire revision even when their application versions differ. A window does not promise compatibility across keylink wire changes.
iroh proves who a peer is; it never decides whether they are welcome.
Allowlist does, and fails closed: it has no Default, admitting everyone
must be typed out, and from_env errors at startup rather than bouncing every
peer at runtime.
trust::Trust admits peers from signed delegation chains rooted in explicitly trusted keys. Every child narrows resources, operations, validity, and delegation depth. Push chains with insert, or let peers offer their own through trust::present before dialing. Revocable::apply records signed revocations or replaces roots before closing connections that no longer qualify.
Bind trust.with_blobs(&blobs)? before issuing grants or serving calls, with the same store used by the node's services. Every clone shares that binding; binding a different store is refused. Losing any established authority proof withdraws all of that holder's old grants, even if another chain still admits it or it has no live RPC connection. This is deliberately conservative: content grants do not record which chain authorized them. Changes that return an exhaustion error and pending revocations established by insert, verify, open, admission, or preflight have the same effect. Lost RPC admission closes first, then blob grants and in-flight grant permits are revoked. Apply trust::Change::Expire on a timer to remove expired authority and its grants; a blob fetch does not check certificate time.
trust::Signed<T> carries a forwardable value and its delegation chain. A verifier supplies a Purpose, trusted roots, and a time; open verifies signatures and authority before decoding the value. Signatures are domain-separated from certificates and revocations. Postcard bytes must round-trip exactly, so nondeterministically ordered values may be refused. Use Trust::open when revocations must count.
trust.guard::<Request, _>(WithinCaveats).resource(extract).install(...) requires admission and checks principal, service, method, and resource before idempotency, every layer, and actor lookup. A keyed resource starts with the actor id; an extractor refines it to actor/name. Singleton services use the extracted name. An application's trust::Access adds policy after caveats. A policy outage is AccessUnavailable, not a forbidden caller. The runnable recipe is keylink/examples/trust_chain.rs.
A call that fails mid-flight leaves the caller unable to say whether it ran.
client.with_idempotency(key) removes the choice between a double booking and
no booking: the serving side records the reply under (peer, key) and answers
a repeat from that record without entering the handler again. Each record also binds a blake3 fingerprint of the received request-message bytes, excluding transport and retry controls, including method, actor and arguments, plus the negotiated contract label, selected principal and holder. Reusing the key for different request bytes, contract label or authority returns IdempotencyRefusal::RequestMismatch, even if the new request is otherwise authorized. Durable records retain the same binding.
The word that does the work is ran. A layer that refuses a call before it
reaches the handler leaves the key claimable, because nothing happened; one that
refuses after leaves Interrupted, because what happened is genuinely unknown.
A record that expires or is evicted leaves a tombstone, so a forgotten key is
refused rather than silently run a second time — for as long as the tombstone
pool holds it, which is limit 4 below.
use std::time::Duration;
use keylink::prelude::*;
use keylink::Idempotency;
// wait for the call already running under this key instead of being refused
client.with_idempotency(Idempotency::attaching_for(key, Duration::from_secs(5)))
.reserve("ADU-014".into(), 2).await?;
client.key_state(key).await? // Fresh means the ledger holds nothing for this keyThe record is in memory by default and dies with the process. Records::Durable
puts it on a disk instead, at two fsyncs per keyed call. Composed with a
reconnecting handle it is effectively-once for one rpc, which is the recipe
below; RFC-05 has the reasoning.
Every other failure here arrives as the answer to a question you asked. A dead
peer does not — you find out on your next call, however long after the fact
that is. client.watch() is a future that resolves when the connection ends,
and it distinguishes the three cases that matter:
use keylink::prelude::*;
use keylink::Departure;
match client.watch().await {
Departure::Closed => info!("peer said goodbye"),
Departure::Local => {} // we shut down; nothing happened
Departure::Lost(e) => reconnect(e).await, // it broke
}Dial-per-message is wasteful and holding one client forever is fragile.
Peer<C> connects on first use, reuses the connection, and reconnects
exactly once when a call reports Transport or Closed. Remote, local
I/O, and per-stream errors return without replay:
let peer: Peer<inventory::Client> = Peer::new(&node, addr);
let item = peer.call(|c| {
let sku = sku.clone(); // `call` takes an FnMut
async move { c.get_item(sku).await }
}).await?;Once, not until it works. Retrying forever turns a peer that is genuinely gone into a caller that never returns.
Many services share one node, and a keylink service can sit beside a foreign
iroh protocol via Server::router(). Registering two services on one ALPN is a
configuration error, not a last-one-wins race.
A Middleware sees the request typed, so a per-method rule is a match on
the real enum. Make the impl generic over Req: Rpc and one layer covers every
service you ever define. QueueTimeout ships with the crate.
The same service! block also generates an Actor trait taking &mut self,
so state lives on a task instead of behind a Mutex:
use keylink::prelude::*;
let warehouse = inventory::actor(Warehouse::seeded()); // in this process
warehouse.reserve("ADU-014".into(), 2).await?; // Err = Unanswered
let server = warehouse.serve(&node, allow)?; // and to other machinesServing it changes nothing on the wire — peers use the same inventory::Client,
on the same ALPN. What changes is that calls no longer overlap, which is the
whole trade: a maintainer can add an .await inside a handler that mutates
state. Under the concurrent model that same edit fails the build, because
clippy::await_holding_lock is deny-level here and a std::sync guard is what
&self state needs.
Two costs, both structural. A turn is never cancelled: dropping the call does not
stop the handler, and a deadline answers the caller without ending the turn. Two
remedies, both below: a deadline bounds the caller's wait and sheds work that has
not started, and Deferred::away takes a slow step out of the turn so it does not
hold the actor. And an actor must not call itself: keylink detects a re-entrant
local call by actor identity and refuses it instead of relying on mailbox depth.
The simple actor, setup(..).observe(..).actor(),
setup(..).feeds(..).actor(), and supervised doors park at depth 1 by default.
The depth is configurable and is not pinned at 1. Use the generated
setup(..).backpressure(..).actor() or
restarting(..).backpressure(..).actor() door—or LocalActor::spawn_deep
directly—and choose what a full mailbox does:
use std::num::NonZeroUsize;
use keylink::Backpressure;
let depth = NonZeroUsize::new(32).expect("non-zero depth");
let warehouse = inventory::setup(Warehouse::seeded())
.backpressure(Backpressure::Park(depth))
.actor();Backpressure::Park(depth) waits for a slot; Backpressure::Shed(depth) refuses
immediately with Full. A deeper mailbox lets callers queue further ahead. It
does not make self-calls safe, bound the memory held by parked senders, or remove
the need to budget enough time for shutdown to drain queued turns. Keyed actors use RegistryBuilder::mailbox_depth(NonZeroUsize) as their default.
A generated .backpressure(..).mountable() setting overrides that default for
one actor. Park waits, and Shed returns MailboxFull.
A send method cannot be served this way at all, and serve stops compiling for
a service declaring one. Its handler reads the upload inside the turn, so a peer
that declares a large body and then stops writing holds the actor until the
connection idle timeout, answering nothing else meanwhile. Bulk transfer goes on
a second service, registered beside the actor on the same node.
Middleware intercepts a message on the wire. An Observer watches a turn
against the state — after the method, before the reply. It sees the state and
how the turn went, and never a request or a reply payload; a layer sees the
opposite. Neither grows toward the other.
use keylink::prelude::*;
use keylink::Ended;
let ledger = ledger::setup(state).observe(saver).actor();
ledger.post(1).await?; // the observer ran before this returned
match ledger.ended().await {
Ended::Stopped => {}
Ended::Observer { name, .. } => error!("{name} ended it; the turn was not committed"),
_ => {}
}Returning Err ends the actor. The refused caller is told
Unanswered::Turn(Withheld::Refused { name }), so it learns which observer
without a second lookup, and ended() adds that observer's own error —
Withheld::Gone means "gone" and must not be stretched to mean "gave up on
durability". A refused turn reaches a peer as
RemoteError::Withheld(Withheld::Refused { name }) rather than NoSuchActor,
which would say "there is nothing here, stop asking" about a turn that ran. The
peer decodes the same Withheld the local caller matches on, rather than reading
a prefix off the front of a string.
The value is withheld either way — a veto says "do not commit", a crash says
nobody read back whether it committed — but both are told
Withheld::ObserverPanicked { name }, which is a variant of its own and not a
shade of the one above it. Reporting a crash as a refusal named a decision
nobody made.
The actor owns its observers, so nothing is shared and no Sync bound appears
anywhere. after_turn returns a boxed future rather than being an async fn,
which is what lets an implementation finish with the state before it awaits — an
implementation that holds it across an await needs A: Sync, and one that does
not, does not.
A mailbox is one way into an actor. A feed is the other. Declare feed Restocked(item: Restock); and the generated Actor trait grows async fn on_restocked(&mut self, item: Restock) — no Caller, no return value, because
nobody called and there is nothing to reply to — and a Stream is wired into
the same loop at spawn, one argument per declared feed:
let warehouse = inventory::setup(Warehouse::seeded())
.feeds(inventory::Feeds::new(deliveries)) // one argument per declared feed
.actor();Feed items and mailbox messages are interleaved fairly: with no off-turn work
waiting, the mailbox takes half the turns and each of n feeds takes 1/2n; with
finished off-turn work ready too (next section), each of the three inputs takes a
third. A feed is local only and never on the wire, so declaring one does not
move the ALPN. A feed that ends, and a feed handler that panics, both end the
actor — a fault with no caller to resume it into would otherwise be dropped onto a
state the handler may have left half applied — and Ended names which happened.
A turn holds its actor until it returns, so a handler that spends an hour on a
check holds the actor for an hour. #[defer] lets the handler return without
answering: reply.later() parks the caller. Deferred::away finishes the job.
It sends the slow work out of the turn, and the work's output comes back as a new
turn on the same actor, with the caller still waiting:
#[defer]
rpc Land(change: Change) -> Landed;
async fn land(&mut self, _cx: Caller<'_>, reply: Reply<Landed>, change: Change)
-> Replied<Landed>
{
let plan = self.prepare(&change); // in the turn, fast
let (deferred, replied) = reply.later();
deferred.away(self, run_checks(plan)); // off the turn, slow
replied
}
impl Back<Checked, Landed> for Repo {
async fn back(&mut self, checked: Checked, reply: Deferred<Landed>) {
match self.compare_and_swap(checked) { // in a turn again
Ok(landed) => reply.send(landed), // gated by this turn's commit
Err(stale) => reply.away(self, recheck(stale)),
}
}
}The work belongs to the actor. It starts only after the launching turn commits, so
an observer that refuses that turn stops it from ever starting, and a durable
actor's work starts only after the turn that chose it is on disk. It is cancelled
when the actor ends. stop, a refusal and a restart wait for the cancelled work
to be gone, so the parked caller is told Gone or Restarting by its own
incarnation and never by the next one. abort does not wait: it destroys the
actor now, and the caller is still told Gone, but the actor's end can be
published while the cancelled work is being destroyed. In a browser the wait is
weaker for every end: the cancelled future is dropped on the executor's next
poll, which can be after the end is published. Only the future passed to away
belongs to the actor: a task it spawns, a spawn_blocking closure it awaits and
anything it starts outside the process are not cancelled with it, and need their
own cancellation. The back turn is an ordinary turn: the panic guard, the
observers and the commit all apply, and other turns ran while the work was away,
so back re-checks what the launching turn assumed. The work itself holds no
reference to the state. It is not durable: a crash loses it, and the actor
reconciles what was pending on restart. stop does not wait for it to finish. Calling away outside a turn, or with another state
type than the running turn's, drops the work and tells the caller Dropped on the
first call.
ServiceBuilder::deadline(budget) answers RemoteError::DeadlinePassed to any
call with no answer after budget, and a caller can set a shorter one of its own
with with_deadline, which travels in the request. What becomes of the work
depends on the call. A running actor turn finishes, and its reply is discarded.
A keyed call is awaited to the end for as long as its connection survives, and
its key settles as what the call really returned, so a retry with the same key
attaches to it or replays its answer and the work runs once. An unkeyed call to a
concurrent Handler is cancelled: its future is dropped where it stands.
let reserved = client
.with_idempotency(Idempotency::attaching_for(key, Duration::from_secs(30)))
.with_deadline(Duration::from_secs(2))
.reserve("ADU-014".into(), 2)
.await;
// Err(Error::DeadlinePassed { .. }) here is the caller's own timer. Retry with the same key.The budget is also stamped on the call as a queueing bound, as QueueTimeout
does, so a message that is still queued when the caller stops waiting never
starts and its key stays claimable. The caller's own error is
Error::DeadlinePassed, and it knows nothing about the far side: the call may not
have arrived, may be running, or may have finished with its reply lost. A deadline
does not bound the body of a recv reply, and the blocking clients do not take
one. A service-only deadline also cannot end a call whose own upload source
stalls: the client writes the whole request body before it reads any reply, so a
send call whose Body stops producing never reads the DeadlinePassed already
written. The caller's with_deadline wraps the whole exchange, upload included,
and does end it.
In a browser a deadline is clamped to i32::MAX milliseconds (about 24.8 days),
the longest wait its timer can express: a longer one would wrap to an immediate
expiry, so it waits that long instead.
Keyed actors, addressed by an id inside the request, live in
inventory::Registry. Build one, call registry.mount(id, inventory::spawn(state)), and serve it with registry.serve(&node, allow).
Peers reach it through inventory::RegistryClient. That path once cost
per-method layers, per-method authorization, blob methods and observers,
because its payload was opaque to keylink. It no longer does: a keyed request
is Addressed<Request> and its reply is the service's own typed Response,
and a keyed mount runs the same turn loop a local Handle does — so layers,
per-method authorization, blob grants, observers, the panic guard and
Observer::finish all reach it. The idempotency ledger was never on that list;
what was missing was a client that could send a key, and
RegistryClient::with_idempotency is it.
What the keyed path still costs: one QUIC connection per actor service — not one
for a node's whole fleet, each with its own handshake, admission decision,
idempotency ledger, keep-alive and congestion window. registry.ended(&id) now
returns a non-inserting future for a mounted instance, so a listener can await
the entry's final loop-or-teardown Ended; registry.stop(&id, budget) remains the separate
question of whether that shutdown drained before its budget. The budget is not
optional: every caller names how long it is prepared to wait before the actor is
killed instead.
inventory::at_local(®istry, id) addresses a mounted actor in this process
and returns inventory::LocalRef. Its generated rpc, tell, and blob
methods use the registry route and the actor's normal turn loop, returning
transport-free LocalError; send and recv are absent because a local
reference has no QUIC body stream. Missing ids, capacity and closed mailboxes
remain distinct from a turn that ran, and the same lifecycle, factory and
eviction machinery is used as for peer delivery.
A function that runs against a local actor in a test and a remote one in
production could not be written: a local call answers Unanswered, a served
one answers Error, and no Result type is both. service! also generates
inventory::Calls, a trait carrying the service's rpc methods with one
error type — Failure, which puts a Withheld in the same arm whichever side
withheld the value. It is implemented for Handle, ActorRef, Client and
IdempotentClient, so one async fn stock<C: inventory::Calls>(surface: &C) -> Result<u64, Failure> runs against all four. The concrete surfaces are
untouched — their own methods still answer Unanswered and Error, and
Failure is reachable only through the bound, which is the act of saying I
have chosen not to know which surface I hold. impl Calls + Governed narrows
it to the surfaces a Dispatch stands in front of, which refuses a Handle:
it means the layers and the allowlist ran, and it does not mean
at-most-once, for which the honest instrument is IdempotentClient named in
the signature.
It carries rpc and none of the other four kinds, because rpc is the only kind
all four surfaces declare: an ActorRef and an IdempotentClient have no tell,
a Handle has no send or recv, ActorRef has no validly servable send,
and blob answers a Grant locally
against a BlobHandle — a content address plus the node that holds it —
remotely. One behaviour diverges across the bound, and it is documented rather
than papered over: a handler panic unwinds into the calling task against a
Handle, and arrives as Failure::Turn(Withheld::Panicked { .. }) against a
served surface. Failure cannot express an unwind, and TurnLost argues at
length why it should not try.
These are deliberate capability boundaries, not missing wrappers. A blank cell means the method is not generated on that surface.
| generated surface | rpc |
tell |
send |
recv |
blob |
error vocabulary |
|---|---|---|---|---|---|---|
Handle<A> |
yes | yes | yes | Unanswered; NotQueued for tell |
||
LocalRef<'_> |
yes | yes | yes | LocalError; LocalTellError for tell |
||
ActorRef<'_> |
yes | yes | yes | Error |
||
Client |
yes | yes | yes | yes | yes | Error; TellError for tell |
IdempotentClient |
yes | yes | yes | yes | Error |
|
blocking::Client |
yes | yes | BlockingError<Error>; BlockingError<TellError> for tell |
|||
blocking::IdempotentClient |
yes | BlockingError<Error> |
A successful local tell returns Queued: the mailbox admitted the message, but
its turn has not run. A successful remote tell returns Transmitted: the bytes
reached the caller's transport, but the peer may never receive, authorize, route,
queue, or handle them. Use rpc ... -> () when completion or an idempotency record
matters. That acknowledged form is why tell stays absent from ActorRef and both
idempotent clients: adding it there would hide failures those surfaces exist to
make observable.
The generated Calls trait carries only rpc, and is implemented for
Handle, ActorRef, Client, and IdempotentClient; it intentionally does
not erase LocalRef's transport-free LocalError or the blocking client's
synchronous execution model. Governed is present on the three served async
surfaces (ActorRef, Client, and IdempotentClient), not on a bare local
handle or local keyed reference.
keylink::blocking is a narrow singleton client for programs with no runtime.
Its plain generated client exposes unary rpc and tell methods. Its
blocking::IdempotentClient exposes only rpc: idempotency excludes tell,
and blocking excludes send, recv, and blob. Keyed addressing remains
async. The blocking module owns a private multi-threaded runtime — mandatory,
because a current-thread runtime only advances iroh's keepalives during
block_on, so the connection would stall between calls. Calling it from inside
a runtime returns BlockingError::BlockingInAsync from generated methods
instead of deadlocking. Node::enter lends that runtime to the calling thread,
which is what a synchronous program needs to construct an actor at all.
The crate compiles and links for wasm32-unknown-unknown with
--no-default-features. The web feature adds Body::from_web_file and
Body::from_web_stream, so a <input type=file> streams up the same way a
path does on a server. blocking, fs and env refuse the target by
compile_error! — each needs an OS facility a browser does not have. mise run wasm-browser loads the module into headless Chrome and runs
keylink/tests/browser.rs there: the ALPN derivation, a postcard round trip of
the reply type, both web constructors, and the compose boundaries. A page
cannot open a UDP socket, so no connection runs in a tab and none is claimed; see
Status.
Every piece of effectively-once for an rpc is already here. Composed, they are a
recipe rather than a feature:
- Put the key on the work, not on the attempt.
client.with_idempotency(key)records the reply under(peer, key). Every retry of that one piece of work carries that one key; different work gets a different key. - Call through
Peer<C>. It reconnects once when the call reportsTransportorClosedand re-issues the call against the new connection. Application and local errors are not replayed. The ledger belongs to the servedService, not to the connection, so the retry meets the same record. - Ask to attach.
Idempotency::attaching_for(key, budget)waits for a call already in flight under the key instead of being refused withInFlight— the difference between a retry that helps and a retry that races. The budget is capped by the servedLimits::attach, which defaults toMAX_ATTACH, 30 s, and the cap is the serving side's, not yours: the budget arrives from the network and a peer asking to be held for a year is asking for a QUIC stream to be held for a year. So a caller with a 90-second handler cannot wait it out — at the default it is refused at 30 s with the work still running, which isInFlightand still the truth. Ask again, or askkey_state. A service that will not admit an attacher at all passesAttaching::ForbiddentoLimits::attach, which answersInFlightat once and spends no waiting place — distinct fromAttaching::UpTo(Duration::ZERO), which admits the attacher and holds it for no time.
keylink/examples/reliable_call.rs is that recipe, compiled by the gate and run over two
nodes. keylink/tests/reliable.rs::a_retried_call_across_a_reconnect_runs_exactly_once is
the proof: the work runs, the serving side closes the connection that was carrying
its answer, and the retry — over a second handshake, against the same ledger — is
answered from the record without entering the handler again.
That is effectively-once for one rpc. Here is what it does not cover.
Limits 2, 3 and 5 are now partly the serving side's choice rather than only
keylink's: ServiceBuilder::keyable decides which methods accept a key at all, so
a key on any other is refused as NotKeyable instead of filling the ledger — which
also voids limit 2's remedy on every method outside the policy, because
cx.call() is then None and an Applied there never fires. And
ServiceBuilder::records can decline to hold the reply at all — reaching limit 5's
Unrecordable refusal by configuration rather than by the method's shape, which
keeps at-most-once and gives up the answer on purpose so that a caller's payload is
not held for the 600 s of limit 3. It narrows limit 4's window from MAX_TOMBSTONES
plus MAX_RECORDS of churn to MAX_TOMBSTONES alone. And ServiceBuilder::limits
replaces the numbers themselves: RECORD_TTL, MAX_RECORDS, MAX_RECORDED_BYTES,
MAX_RECORDED_FRAME and MAX_TOMBSTONES are the defaults of Limits fields
rather than fixed values, which is how a service bounds its own memory instead of only
its exposure. Not every constant these limits print: MAX_HEADER is the wire's own cap
and APPLIED_BOUND bounds a set in the caller's state, and neither is a Limits
field. The margin limit 4 names is the one to keep — it is a property of the
default, so narrowing it takes on the degradation that limit says a caller cannot
detect. See docs/rfc-08-serving-side-keys.md.
-
There is no reliable
tell. Atellhas no reply stream, so nothing can ack it, nothing can refuse it, and it does not accept a key — there is no reply to record and answer a repeat with.with_idempotencyreturns anIdempotentClient, which has notellmethod, so this limit is one the compiler enforces rather than one you have to remember. Work that has to arrive is anrpc. -
The ledger is memory by default, so across a restart the guarantee is yours to keep — or you buy it.
Records::Memorydies with the process.Records::Durablewrites each record throughkeylink-persist's entry port and survives one, which costs twofsyncs per keyed call: one before the handler is entered and one before its reply is written.Durable state is what made the default sharp rather than mild. While nothing survived a restart, re-running a retried call was correct — a fresh process had a fresh state. It stopped being correct the moment state became durable. A process that fsyncs turn n, records the reply in an in-memory ledger, and dies before the frame reaches the peer will, on restart, load a state that already contains n, find an empty ledger, and run n again. That is the pairing
Records::Durableexists for, and it is why a service with durable state should think twice before leaving the ledger at the default.What no store makes durable is the in-flight half. A key claimed when the process died comes back
Interrupted— unknown — because the caller it belonged to held a live stream, and no disk carries one of those forward.A store that refuses a write is answered, not swallowed, and the two refusals are on opposite sides of the handler. If it cannot write the claim, nothing ran and the caller is told
NotClaimed— the same key is safe to send again. If it cannot write the settle, the call ran and its reply is discarded rather than sent, which isUnrecordable; a reply this ledger cannot replay is a reply the caller would ask for again. Neither isInterrupted, which means the handler produced nothing at all.Records::Durableon a service that accepts no idempotency key is refused at startup, asConfigError::DurableWithoutKeys. No ledger is built, so the store would be held and never written — and its exclusive claim on the log file would refuse the next service that meant to use it.So effectively-once is a property of one process's lifetime, not of a key — for as long as the only thing that knows about the key is the ledger.
cx.call()changes that: itsCallIdbinds the peer and key to the same received-message, contract-label and principal/holder commitment as the ledger.Applied<T>is a bounded set of those bound calls, so an old answer cannot cross requests or principals after the ledger forgets its slot. Retries must preserve message bytes, negotiated label and authority; another contract encoding is another bound call. CallId v2 does not migrate old IDs; a reset of pre-v2 applied-ID state is required. Put one in your own state, and the dedupe is three lines:let call = cx.peer().and_then(Context::call); if let Some(&balance) = self.applied.get(call) { return balance; } self.balance += amount; self.applied.insert(call, self.balance);
The method has to be one your service keys.
Keyabledecides that, and on a method outside the policy a keyed call is refusedNotKeyablebefore the handler runs — socx.call()isNonethere always, and anAppliedin that handler never fires and never says so. Deduplicating in a handler means putting its method in the policy.Insert in the turn that applies, as above, and never in a turn that merely hears about the work: the two together are one snapshot and one
fsync, and that is the whole reason the set lives in your state rather than in keylink's. A#[defer]method's parking turn is the trap — park theCallIdbeside theDeferredand let the releasing turn record it, or a restart in between leaves a record with no work behind it.Applied<()>is the valueless form for exactly that shape, where there is no answer to replay.AppliedimplementsSerialize/Deserializeand noSchema, so it cannot reach an ALPN andpostcard::to_allocvec(self)keeps compiling when you add the field. Take that encoder rather than writing your own: a hand-rolled snapshot that omits the field compiles, passes every test, and starts empty after every restart, which is the silent at-least-once this whole mechanism exists to prevent.APPLIED_BOUNDis 4096 calls — and the unit is calls, not seconds. It means "4096 other pieces of keyed work may land between yours and its retry". Turning that into a duration needs your adapter's turn rate: overkeylink-persist's file store a turn is ~7.6 ms, so ~130/s and roughly 31 seconds — but the same turn with nofsyncbehind it is ~13 µs, where 4096 entries is well under a second. Measure and divide, and treat the count as the guarantee.That 7.6 ms is a macOS number and it does not carry to Linux.
std::fs::File::sync_dataissuesF_FULLFSYNCon macOS, which asks the drive to flush its own write cache — measured at about 12x a plainlibc::fsync, 5498 µs against 462 µs. On Linux the same call isfdatasync, so a Linux deployment is faster by roughly that order and every figure derived from 7.6 ms in this file is pessimistic there. Measure on the platform you will run on.And the set is a term in that turn rate, so measure at the bound you mean to run.
Persistre-encodes the whole state every completed turn, so anAppliedcosts its entries on every turn whether one changed or not — 4096 entries is ~92 µs, which is 1.2% of anfsyncturn and 6.5× a non-syncing one. That ratio divides by 14.0 µs, not by the ~13 µs above: both halves come from one run, and dividing by the tabled figure instead gives 7.2×.APPLIED_BOUND's rustdoc has the measured table and says why the two differ. Raise the bound for slow callers; lower it for cheap turns.The busiest window is the one right after the restart, which concentrates arrivals — every peer's backoff fires at once. What drains the set, though, is only first-time keyed work: a recognised retry returns before it touches the eviction order, so a herd of pure retries costs no places at all. A reconnecting client supplies both, and it is the new work that spends the bound.
What this does not buy, in full. The duplicate still enters the handler and still costs a turn — about 7.6 ms of
fsyncon macOS, less on Linux, on a state it did not change — because only a durable ledger could refuse it before dispatch, and that one is structural: the record lives in your state, so reaching it means dispatching to your state. And the dedupe is yours to write, because the correct version knows what your answer is and whether replaying it beats refusing.Separately:
key_state(key)answersFreshafter a restart on aRecords::Memoryservice, so a peer learns nothing about a key your domain is about to dedupe. This paragraph used to add as the tree stands today, and predicted that "a durable record store would answer it again, with nothing here changing". It does, and nothing here changed: onRecords::Durablethe query answersDoneafter a restart. What did not come back is the cost above it — a duplicate entering the handler is a consequence of the record living in your state, and no store on a disk alters that.It also rests on the calling peer keeping its identity. The peer is in the hash, so a client that reconnects on a fresh keypair is a different caller, is not recognised, and runs the work again — silently, and indistinguishably from a first call, which is the degradation limit 4 below names as the one a caller cannot detect. A long-lived
SecretKeyon the calling side is the precondition, not a deployment preference.keylink/tests/persist.rs::a_retried_call_across_a_restart_is_applied_onceis the proof, over two real nodes and the real file store: the process dies, the restarted ledger answersFresh, the retry is dispatched, and the work does not happen twice. -
A record answers for 600 s. That is
RECORD_TTL, alongside volume bounds ofMAX_RECORDS— 1024 entries — andMAX_RECORDED_BYTES— 1 MiB of held reply in total, across every peer, not per key. Past any of them the reply is dropped and the retry is refused asForgotten: at-most-once still holds, and the answer does not. Read the result out of your own domain; do not re-run the work.cx.call()is no help here, and it is worth saying so. AForgottenrepeat is refused before dispatch, so there is no turn, noContext, and nothing for a handler to recognise — the advice above is the caller's to act on, in its own domain, and limit 2's remedy is the serving side's. The two limits look adjacent and are not. -
Past
MAX_TOMBSTONES— 8192, eight times the record bound — a key isFreshagain and the retry runs. This is the one edge where the ledger's guarantee degrades: effectively-once, to at-most-once, to at-least-once. It is also the one state a caller cannot detect, becauseFreshis what "never used" looks like too. The wide margin over the record bound is what makes it unlikely rather than impossible.The real bound, beside it. The margin is over this ledger's traffic, so what it protects is the window between a call and its retry on one live process — and a restart resets it to nothing, which is limit 2. Two bounds, then, and they are different quantities:
MAX_TOMBSTONESis how many other keys may pass before this one is vacant again, andAPPLIED_BOUNDis how many other keys may pass before your own state forgets it. The second is the one that survives a restart, and it is the one you can raise.Undetectable degradation has two doors, and this is the second: a caller on a fresh keypair also gets
FreshfromApplied, for the same reason and with the same silence. -
A reply that carries a payload is never recorded. A
recvmethod answers with a body beside its reply, and a body is streamed rather than held, so the repeat is refused asUnrecordableinstead of being answered with a header whose payload never arrives — a permanent property of the method, not an eviction. Ablobreply is not affected: its bytes move on iroh-blobs and the reply itself is just a header carrying an address, so it records and replays like any other.So is any reply frame over
MAX_RECORDED_FRAME— 1 MiB, 1048576. This line used to nameMAX_RECORDED_BYTESand print no figure, which was wrong twice: the ledger's ceiling is a bound on one frame, and the pool is a separate number that happens to share its value today.The ceiling equals
MAX_HEADER, the wire's own cap, and that is not a coincidence.MAX_HEADERbounds the postcard payload, while the ledger stores the whole frame including its four-byte length prefix — so a maximal payload arrives as 1048580 bytes, which exceeds the 1 MiB pool on its own. A reply that cannot fit the pool under any conditions is permanently unrecordable, soUnrecordableis the right answer for it and the ceiling stops at the payload cap to say so. Raise the pool and that changes: a bigger pool can hold a maximal frame, and then the ceiling gains the four bytes.The number is worth setting lower than the default, and
Limits::frame_bytesis how: one large reply that fits the pool can evict many small ones. There is no floor to quote, because the smallest useful value is the encoding of one service's own response type. -
Interruptedis permanent, and it is the answer that needs a decision. A call whose handler was entered and produced no reply — a dead connection, a shutdown — settles the key asInterrupted. Whether the work took effect is genuinely unknown and no record can make it known, so retrying that key is refused forever, by design. What a caller does: reconcile against its own domain, then issue a new key.client.key_state(key).awaitasks without running anything, which is the only way to learn a state you are not willing to discover by executing. -
A
feedhas no delivery guarantee at all. It is a local stream into an actor rather than a call: nobody sent it, nothing acks an item, nothing refuses one, and no key applies. It never reaches the wire. A feed that dies takes the actor with it, which is the only report a feed can make.
Use the separate keylink-http package when a caller
cannot use Rust or the Keylink protocol. It serves generated rpc, send,
recv and blob declarations on a singleton surface, and rpc, recv and
blob on a keyed actor surface. tell has no route, and a keyed surface serves
no send. Its README carries the route table and the error codes with their
statuses; this section is the summary.
keylink-openapi is a separate opt-in package that
generates an OpenAPI 3.1 document and a standalone JSON Schema from the same
surface the gateway is mounted with. Its schema dependency reaches neither
keylink nor keylink-http.
The gateway calls the existing generated Client, so the Keylink server has one
serving pipeline.
Run the example:
cargo run -p keylink-http --example inventoryThe program prints a loopback URL. Call it with an argument object:
curl -i -X POST "$URL/rpc/Add" \
-H 'Authorization: Bearer allow' \
-H 'Content-Type: application/json' \
--data '{"left":2,"right":3}'The success body is only the declared return value. The response above is 5,
not a Keylink Response enum. A zero-argument RPC uses {}. A unit return uses
null. HTTP field names use the Rust argument names. Serde attributes on nested
domain types still apply.
Outer middleware must verify the HTTP identity and insert a Principal in the
request extensions. Principal is a stable, non-secret namespace. The gateway
then calls the mandatory async Access policy before it reads the body. The
Keylink server authenticates the gateway node's EndpointId, not the HTTP
principal.
Set finite request and success limits with JsonLimits. Both limits are
inclusive. Error envelopes have a separate fixed limit. JavaScript clients must
keep integers in the safe integer range, or use JSON-safe domain DTOs.
An optional Idempotency-Key is valid for one logical call. The gateway hashes
it with the principal namespace before it reaches Keylink. Two principals cannot
replay each other's result. Do not reuse one key for another method or argument
set. Replay records live for the Keylink server process lifetime; they are not a
durable exactly-once record.
Errors use this shape:
{"error":{"code":"forbidden","message":"this caller may not invoke this method"}}The stable codes are invalid_request, invalid_idempotency_key,
idempotency_not_supported, invalid_stream_metadata, range_not_supported,
unauthenticated, forbidden, method_not_found, actor_not_found,
method_not_allowed, blob_not_found, idempotency_conflict, blob_gone,
request_too_large, request_stream_too_large, unsupported_media_type,
range_not_satisfiable, idempotency_capacity, response_too_large,
response_stream_too_large, upstream_error, upstream_unavailable and
upstream_timeout. The package README carries the same
set with each status. the_readme_documents_every_error_code reads both files
and fails when a code the gateway can answer with is missing from either, so this
list cannot fall behind the enum. It compares codes alone: no test checks a status
or a description, and those are reviewed by hand.
A client disconnect drops the gateway-owned Keylink call future. Work that
already entered the backend can continue, and an idempotent call can settle as
Interrupted. The gateway does not promise rollback.
The gateway does not expose tell, feeds, __Status, WebSockets, actor
lifecycle, actor discovery, Keylink administration, or CORS. send, recv,
blob, keyed actors and OpenAPI generation all exist — the package README
carries the route table.
export KEYLINK_SECRET=$(openssl rand -hex 32) # 64 hex characters, one identity
cargo run -p demo -- id # the id peers dial
# machine A
KEYLINK_SECRET=<64-hex> KEYLINK_ALLOWED_IDS=<peer-id> cargo run -p demo -- serve
# machine B
cargo run -p demo -- connect <endpoint-id> [file-to-upload]
cargo run -p demo -- connect-sync <endpoint-id> # the same, without a runtimeSet KEYLINK_SECRET before you read an id. Every subcommand binds through
Node::bind_from_env, which falls back to a random identity when the variable
is unset — so demo -- id without it prints an address that the next command
will not answer on. Each side keeps its own secret and puts the other side's
printed id in KEYLINK_ALLOWED_IDS.
With one machine and no internet, set KEYLINK_OFFLINE=1 on both sides: the
demo binds without discovery, prints a <id>@<ip:port> line, and accepts that
form as the dial target.
Build both application versions from one commit to try the window. Leave the server on the default build (version 3), then connect from the other machine with cargo run -p demo --features demo/previous-contract -- connect <id>. Swap roles: serve with --features demo/previous-contract, then connect with the default build. Both directions print speaking 2 and finish with contract: OK. The newer client fills aisle: None and reports the version-3-only Count as Unsupported when talking to 2.
cargo run -p demo --features demo/swap-variants -- connect <id> builds a peer
whose Currency variants are swapped — identical shapes, so postcard would
decode Brl as Usd silently. The handshake refuses it instead.
The demo's Warehouse is served as an actor, so it also stands as the worked
example of that model against an unmodified client. Its upload is a second
service on a second ALPN, registered beside it on the same node, because a
send method cannot be served as an actor — the split every actor with a bulk
transfer now has to make.
One grant for a whole tree, between two machines on n0's public relays and
discovery (a bare EndpointId is enough to dial):
cargo run -p keylink --example closure_share -- serve # machine A, prints its id
cargo run -p keylink --example closure_share -- fetch <endpoint-id> # machine Bfetch ends with closure_share: OK or exits non-zero. With no arguments the
same flow runs in one process, which is what mise run examples does.
./scripts/install-hooks.sh # once per clone — the hooks are not automatic
mise run gate # the whole gate. There is no other oneCI is off by decision, and mise run gate is the gate. The workflow was
deleted from main rather than left sitting there switched off, because a workflow
nobody runs still looks like coverage — and the repository's Actions setting is off
as well, so two independent things hold it. CONTRIBUTING.md measures both. The
pre-push hook runs gate, so the check happens
before a push instead of after one — and ./scripts/install-hooks.sh is what
makes that true for a clone, which is why it is the first line above.
What that costs, stated rather than discovered: the gate now runs on one
machine, on one OS. CI ran Linux, and a developer on a Mac no longer has
anything checking that the tree builds on the other one. Nothing in the gate is
weaker than it was — wasm-guards, msrv and deny moved into gate in the
same change that removed the workflow, so the three checks only CI ran are still
run — but they are run wherever you are, not everywhere.
wasm-guards is also stronger than the loop it replaces. The workflow
asserted only that blocking, fs and env fail to build for wasm, which is
vacuous: they fail anyway, deep inside tokio, once the feature turns on
something a browser has no facility for. Disabling the fs guard left the CI
check green. The task now requires the failure to quote keylink's own
compile_error!, which is the claim the workflow's comment made and did not
check.
Install mise and rustup, then mise install. That
fetches the five pinned tools — cargo-deny, ripgrep, sccache,
wasm-bindgen-test-runner and chromedriver — and rust-toolchain.toml adds the
wasm32-unknown-unknown target and the llvm-tools component. The second is
not decoration: ring links its C through ar, and Apple's writes an archive
index rust-lld cannot read.
Four things mise does not install, each named here because a missing one fails somewhere unhelpful:
- Google Chrome, for
wasm-browser. The chromedriver pin is coupled to the installed browser's major version, so a Chrome that auto-updates past the pinned one reddens the task. Move the pin, never bypass the task. python3, which five gate steps are written in —anchors,anchors-selftest,headings,replay-anchorsandtask-lists.curl, which both HTTP smokes drive the running gateway with.- A second Rust toolchain,
1.91, formsrv. About 200 MB on first use. That cost used to sit on CI; it is paid once per clone now, because an MSRV nobody compiles is a promise broken silently.
The steps run cheapest-failure-first, and [tasks.gate] in mise.toml is the
list — not repeated here, because a second copy is a copy that goes stale.
mise run pre-commit is the cheap subset a commit pays, and that one is
pinned: mise run task-lists fails when any copy of it disagrees with
mise.toml.
See CONTRIBUTING.md for the wire-contract pin, the public API
rules, and when --no-verify is legitimate. The RFCs record
the decisions, including the reversed ones.
Early. The API will move. Not published to crates.io.
Contract windows support frozen application versions with explicit conversions, for singleton services and keyed registries. The application owns conversion policy; keylink's own wire is not yet versioned.
Greenfield: nothing here has ever run in production, so an ALPN move costs nothing and no compatibility shim is owed. Changes are taken in their correct form rather than their compatible one — every call site moves and the old path is deleted. That stops being true the day somebody deploys this, which is the argument for fixing the sharp things now.
A moved contract hash is therefore not a cost, and nothing here is designed
around avoiding one. No change is scheduled, grouped or reshaped to keep a pin
still, and a pin move is not reported as a price paid — a rename, a new variant
and a reshaped struct are all free. The two tests that pin the literals are
#[ignore]d, so not even the bookkeeping is owed. What still runs is the pair of
claims about the mechanism rather than the value: that two type sets derive two
different contracts, and that a peer built from other types is refused at the
handshake. This holds until keylink has a stable version, which is when the pins
come back and reversing it is the point of reaching one.
Known gaps, in rough priority order:
-
Trust is a local view. Revocations reach other nodes only through the application's feeder. Accepted records are permanent and never pruned. Each signer and proven upstream issuer has an accepted-revocation budget; exhausting a key permanently refuses its subtree rather than evicting tombstones. Path accounting also bounds churn through freshly delegated revoker keys. Adding new trusted roots supplies new operator-authorized budgets; it does not reset existing keys. Idle expired connections close only when the application applies
trust::Change::Expire; expiry is checked on every call regardless. Already-dispatched calls finish after revocation. The host clock is trusted, with no skew allowance. -
Blob withdrawal is an explicit binding.
Trust::with_blobsmust receive the same store as the services before any grants are issued. Without it, closing RPC admission does not retract grants. With it, any lost authority proof withdraws all of a known holder's grants, including narrowed and disconnected holders. Expired grants wait for the application'sChange::Expiretimer. Delivered bytes cannot be taken back. -
Signed values are replayable. The application supplies freshness in
T. Its shape is not part of a service contract hash; drift is refused atopen. Idempotency records remain per device key, not per delegated principal. -
Trust closes every tracked connection for a revoked peer. Connections are tracked independently across service ALPNs. The open presentation protocol has byte, depth, per-holder and total retained-chain caps, but no rate limit. New entries fail loudly at the total cap; feeders must handle capacity refusals. Pending unauthorized revocations are bounded separately; accepted tombstones are bounded by issuer-path budgets and are never evicted. Browsers can verify chains and signed values, not serve guarded QUIC connections.
-
No connection has ever run in a browser.
mise run wasm-browserloads the module into headless Chrome and runs every test inkeylink/tests/browser.rsthere — the schema hash, apostcardround trip ofWireResponse, bothwebbody constructors, and the compose patience and lossy-broadcast boundaries — so the module is no longer merely linked. What is untested in a tab is everything that needs a socket: noNode, noServer, noClient, no actor turn. A page cannot open a UDP socket, so QUIC is out of reach whatever keylink does. Threaded wasm is refused outright: the local mailbox holds a thread-affine task handle behind aSend + Synctype. -
A turn is never cancelled. A deadline bounds the caller's wait and sheds work that has not started; slow work goes off the turn with
Deferred::away. A deadline's answer to a caller does not stop a turn that is running, so it sheds nothing once the turn has started, and a wedged keyed turn keeps its keyRunningpast the caller's deadline, bounded by the per-peer running cap. It does record the turn's real reply under the key, so the retry receives it.QueueTimeoutand a deadline both refuse a message that has not started, and leave its key claimable, because nothing ran. -
Protocols cannot be added or removed after
spawn. iroh-examples'custom-routershows a runtime-mutable alternative worth adopting. -
Idempotency records are in memory by default, so a retry across a restart runs the handler again.
Records::Durablekeeps them on disk instead, at the storage cost and with the in-flight recovery rule Reliable delivery above describes, along with the other limits of the guarantee. -
A keyed instance's
Registry::ended(&id)future must be subscribed while the id is mounted; it returnsNonefor an id that is not live and never inserts one. The future reports the entry's final loop-or-teardown ending, whileRegistry::stop(&id, budget)separately reports whether that shutdown drained or had to kill it. -
Supervision restarts one actor, and no strategy reaches its siblings.
LocalActor::spawn_supervised,LocalActor::spawn_supervised_deep, andrun_keyed_supervisedtake anIntensityand put a fresh incarnation on the same task and the same mailbox, so a caller whose reply died mid-restart is toldRestartingrather thanGone. There is noSupervisortype, noOneForAll/RestForOneand no parent/child tree; RFC-07 §9.8 refuses each one and gives the reason.compose::Groupis the part of that caveat which was built — shared fate across several actors, retired in reverse insertion order — and its two verdicts areForgetandStopAll, so nothing there restarts a sibling either. Restarting is opt-in, so the rest of this bullet is the unsupervised default, and under a supervisor it is what one incarnation costs. What a fault costs depends on which input caused it: a panickingrpcturn costs that message, the actor survives, and the caller is told rather than left to infer it. A panic in afeedhandler ends the actor, and so does a feed that simply stops — nobody is waiting on a feed item, so dropping its fault would carry on over a half-applied state.Endedsays which happened. RFC-04 §8 has the original argument against supervision; RFC-07 §5.1 has the argument for an ending that names itself. -
Error::SchemaMismatchis classified from the rendered error string, because the TLS alert code is several private layers down. Narrow and tested, but it would break if iroh reworded its message.
Read CONTRIBUTING.md first, then run
./scripts/install-hooks.sh so mise run gate fires before your push rather
than after it. There is no CI to catch what the hook misses — see
Working on it for why, and for what that costs.
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT license (LICENSE-MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.