Repository navigation
netstack: every NIC driver is a process of its own, over a link (P4) - #1106
Conversation
Review of #1106 at 4105fe8 (P4 / LAN stage 3), round 1ArchitectureThe shape is right. The protocol lives in its own library ( I checked the full-ring handshake. Each side does one Every ring index, length, link word, group count and group encoding is bounded before it is used. Counters are display-only. The stack's Size: production is about +1.9k/−0.5k lines. FindingsBLOCKER 1. A driver's hello can panic netstack.
BLOCKER 2.
BLOCKER 3. The fault loop has no bound (risk a). Bound it now.
BLOCKER 4. A restarted driver can be refused for good.
BLOCKER 5. False or stale comments.
BLOCKER 6. Issue records this change makes inaccurate.
SUGGESTION 1. Silent pending links in SUGGESTION 2. The hello's label. It is any UTF-8 string up to 32 bytes, newlines included, and it is printed into the netstack lines that judges match ( SUGGESTION 3. Stale counters on a reopened link. On a new SUGGESTION 4. The stop pipe. Now that the I219 driver is a Risk (b): the 30 s wait on a machine with no NICThis is acceptable as a recorded temporary gap. It is in Evidence
SEND BACK |
… link ends its driver, and the I219 does not restart - A hello carries an `IndividualMac`, and a group's address, or a name that is not ASCII letters, digits and `-`, is refused `Malformed` at decode, so no driver's hello reaches a panic in netstack. - A link whose driver wrote what no driver writes ends, and that address's next hello is refused `Faulted`; the driver ends on it, so the supervisor's restart budget bounds a driver that keeps doing it. - netstack refuses `OneLink` only a hello with another address, or with its own while that link is alive, read on the connection then: the kernel closes a dead driver's handles before its end is published, so a restarted driver's hello finds the old link hung up. - The I219 rows take no `restart`: its start resets the part, which over a ring a link change left it is not known to be safe. - Counters are read only for an ask the link was given; a pending driver connection in the first wait is let go after the handshake bound. - Comments that described the rejected `--drivers` and a restarting stack are corrected, and the stop and room records name the driver that now owns their subject; the stop record's slug says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 1 answered at b5f9cc0 (records in 559a69d)
Suggestions taken:
Not taken: 4, the stop pipe, which belongs to the renamed record's follow-up. Evidence gap: the PR now states that the restart claim is shown by two tests composed, not by one. Checks at b5f9cc0: the five network guest tests and |
Review of #1106, round 2, at 559a69d (fixes in b5f9cc0)I reviewed only what round 2 changed. Everything settled in round 1 stays settled. The round 1 blockers
The suggestions taken
New findingBLOCKER:
The T14 rows not rerunThe reasoning is sufficient, though the PR body describes the change too narrowly.
Evidence
SEND BACK for the one new blocker. Everything else is resolved. |
…converts it `toyos-link` depended on netstack's wire crate for one six-byte type, which put Ethernet, IPv4 and TCP wire formats and toyos-crypto into both drivers. The link keeps its own checked `Address` instead, an individual one refused otherwise at decode; a driver checks its device's address against it, and netstack builds its `IndividualMac` from it once, as it hears the hello, refusing `Malformed` should that ever fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Orchestrator: r2's one blocker checked against 559a69d..4864cf7. toyos-link has its own checked individual Address with no netstack/wire dependency, and netstack converts it to IndividualMac once at the hello, refusing Malformed on failure. The five network guest tests pass at the head, and the PR body covers round 2's netstack changes. The T14 rows at 8bdf491 stand for the reason r2 accepted. Landing. |
netstack held every NIC driver and the stack in one process, behind `Card`, and each image's row named the card its machine has. A driver's fault ended every connection, no driver could start again without the stack's state, and a machine nobody wrote a row for had no network. `toyos-link` is the protocol between them, on `toyos::transport` alone: a driver connects to netstack's `link` port and sends a region it made with its hello. The region's header holds a ring of frame lengths each way and the words each side states and asks with: the driver's link, the multicast groups the stack joined, and the counters `inspect` asks for. Frame `n` is in slot `n` of its ring, so a slot is its producer's again when the consumer releases it, and each side copies a frame once. A doorbell is a byte on the connection; a full ring's producer sets a want that the releasing consumer takes, each side by one AcqRel swap of the want's word, so neither misses the other; `tests/ends_loom.rs` checks it and reds with the swaps Relaxed. Everything the peer writes is bounded before it is believed; the counters are displayed and never acted on. The I219/82574 driver (`system/i219`, with `toyos-i219` moved under it) and the virtio driver (`system/virtio-net`) are `drives` rows: one process per function a row's parts match by class, vendor and device, restarted by the supervisor when one ends other than by exit 0. Each runs `toyos-link`'s pass over its device and connects again when netstack hangs up. netstack claims no device. Its node is made with the first driver's address, so it waits for that driver's hello, and a machine none says hello on within 30 s has no network and netstack leaves. A link that ends leaves every connection to the node, and a hello with that address is opened again; any other is refused `OneLink` until P5 runs several links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both rows passed on one benchcase boot at 8bdf491, the orchestrator's, judged from the readback; what P4 still owes is S2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… link ends its driver, and the I219 does not restart - A hello carries an `IndividualMac`, and a group's address, or a name that is not ASCII letters, digits and `-`, is refused `Malformed` at decode, so no driver's hello reaches a panic in netstack. - A link whose driver wrote what no driver writes ends, and that address's next hello is refused `Faulted`; the driver ends on it, so the supervisor's restart budget bounds a driver that keeps doing it. - netstack refuses `OneLink` only a hello with another address, or with its own while that link is alive, read on the connection then: the kernel closes a dead driver's handles before its end is published, so a restarted driver's hello finds the old link hung up. - The I219 rows take no `restart`: its start resets the part, which over a ring a link change left it is not known to be safe. - Counters are read only for an ask the link was given; a pending driver connection in the first wait is let go after the handshake bound. - Comments that described the rejected `--drivers` and a restarting stack are corrected, and the stop and room records name the driver that now owns their subject; the stop record's slug says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes over one link region moved 1,514-byte frames on the T14 at about 3,300 TSC cycles of one CPU a frame with a doorbell per frame, 920 per batch of 8 and 620 per batch of 32, both directions alike; full rate needs 200,163 frames a second each way. Reported, deciding nothing by itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…converts it `toyos-link` depended on netstack's wire crate for one six-byte type, which put Ethernet, IPv4 and TCP wire formats and toyos-crypto into both drivers. The link keeps its own checked `Address` instead, an individual one refused otherwise at decode; a driver checks its device's address against it, and netstack builds its `IndividualMac` from it once, as it hears the hello, refusing `Malformed` should that ever fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… heap The merge queue's `mutate-want-relaxed` control found no FAILED line: the deadlock it should report aborted the test binary first. Two causes, each enough on its own. The model held its page and doorbell in loom's `Arc`, whose drop panics while a deadlocked model unwinds (issues/loom-arc-panics-in-its-drop-while-a-failed-model-unwinds.md); and its 768-word page, built inline, left too little of loom's 32 KiB coroutine stack to unwind, which on Linux overflowed and aborted before the harness printed its verdict. The model now shares one `Rc`, as transport_loom does, and collects the page on the heap. With `Rc` and an inline page the coroutine still overflows (printed on macOS, fatal on the CI runner). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ches its verdict
|
Orchestrator: re-queueing. The merge queue's red was the link loom control's test aborting (stack overflow, then loom's Arc drop panic during unwind) before it printed its verdict. 6807935 shares the page by Rc and builds it on the heap, as transport_loom does. Host CI is green with |
Each direction of a connection is a kernel message queue: the bytes of its messages in the frames #1074 takes as a pipe fills, and beside them the messages' lengths and their handles, in the order they were sent. A send admits the whole message under the queue's lock or refuses it with nothing of it queued and every handle back at its number; a receive takes the head message whole into the caller's buffers, its handles installed under the same lock, or leaves it queued. No stream position exists any more, so nothing pins a batch of handles at one. The bounds are the approved ones, per direction: 2 MiB of bytes, taken as used, 256 messages and 64 handles. Without room the send waits, or answers WouldBlock; a receiver's buffer too small for the head message is InvalidArgument and a table without room for its handles is ResourceExhausted, both leaving it queued; a peer gone is Gone. A connection polls writable once the largest message would fit, so a refused non-blocking send retried on readiness lands. ABI: SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone, and plain read and write on a connection are refused PermissionDenied: a connection is never a byte stream. SDK: a frame is a 4-byte message type and its payload, the kernel reporting length and handle count, so IpcHeader goes. FrameRx is a buffer and one receive, non-blocking (pump) or blocking (recv), and replaces recv_header/recv_payload/recv_bytes/recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer instead of a zeroed 8 KiB one. Every consumer moves to it in this commit. netstack's Client::gone asks with a non-blocking receive instead of a one-byte read. The link (#1106) rang its doorbell as a byte on its connection, the last such doorbell; it rings an event each way now, as the block service's do. The hello carries the bell the driver rings, the answer the bell the stack rings, and a bell whose waker is gone is the link's end, so the connection goes once the link is open. Deleted: kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch, try_read_within, the pipe's written/read counters and Landing (a pipe's write landing is its own rule now), the connection arms of the pipe read and write paths, the acceptor's close_now of each queued connection (a pending connection's receiver goes with it). Tests: the pure queue (order, wholeness, each bound, a refused receive taking nothing, a 50,000-step random run against the order sent); the SDK's framing and refusals; messages_are_whole, replacing message_carries_its_handles, for the syscalls' refusals and the bounds as a sender meets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each direction of a connection is a kernel message queue: the bytes of its messages in the frames #1074 takes as a pipe fills, and beside them the messages' lengths and their handles, in the order they were sent. A send admits the whole message under the queue's lock or refuses it with nothing of it queued and every handle back at its number; a receive takes the head message whole into the caller's buffers, its handles installed under the same lock, or leaves it queued. No stream position exists any more, so nothing pins a batch of handles at one. The bounds are the approved ones, per direction: 2 MiB of bytes, taken as used, 256 messages and 64 handles. Without room the send waits, or answers WouldBlock; a receiver's buffer too small for the head message is InvalidArgument and a table without room for its handles is ResourceExhausted, both leaving it queued; a peer gone is Gone. A connection polls writable once the largest message would fit, so a refused non-blocking send retried on readiness lands. ABI: SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone, and plain read and write on a connection are refused PermissionDenied: a connection is never a byte stream. SDK: a frame is a 4-byte message type and its payload, the kernel reporting length and handle count, so IpcHeader goes. FrameRx is a buffer and one receive, non-blocking (pump) or blocking (recv), and replaces recv_header/recv_payload/recv_bytes/recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer instead of a zeroed 8 KiB one. Every consumer moves to it in this commit. netstack's Client::gone asks with a non-blocking receive instead of a one-byte read. The link (#1106) rang its doorbell as a byte on its connection, the last such doorbell; it rings an event each way now, as the block service's do. The hello carries the bell the driver rings, the answer the bell the stack rings, and a bell whose waker is gone is the link's end, so the connection goes once the link is open. Deleted: kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch, try_read_within, the pipe's written/read counters and Landing (a pipe's write landing is its own rule now), the connection arms of the pipe read and write paths, the acceptor's close_now of each queued connection (a pending connection's receiver goes with it). Tests: the pure queue (order, wholeness, each bound, a refused receive taking nothing, a 50,000-step random run against the order sent); the SDK's framing and refusals; messages_are_whole, replacing message_carries_its_handles, for the syscalls' refusals and the bounds as a sender meets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ue stays open The bucket held max(IW, 1 ms at the rate). At srtt up to about 2 ms in slow start, or 1.25 ms in avoidance, a millisecond at the rate is at least cwnd, so the pacer never held there. HyStart++ still took L = infinity there, which is the unpaced L = infinity RFC 9406 §4.3 rules out. The bucket is now max(IW, min(1 ms at the rate, cwnd / 2)). A window past the initial one always leaves in more than one burst, at any srtt. The cost is more wakes on short round trips: a half bucket every srtt / 8 in slow start. The module header now gives the bucket's two bounds and their floor, and says where the millisecond departs from RFC 9002 §7.7's initial-window burst. It also names the data that spends but is never held, each with its reason: retransmissions, new data in SACK recovery, and a loss probe of new data. `hold`'s doc says when the held data is offered again. The ramp issue is no longer deleted. The A/B ran on trees from before #1106 and with the old bucket, so it is recorded as relative evidence (P over C, N = 2 over 1.25) at its full evidence hash. Parity waits on a boot of this head. - a_bucket_holds_at_most_half_the_window: at srtt 1 ms the bucket is 50 of a 100-segment window. - a_fast_bucket_holds_a_millisecond now uses srtt 8 ms, where the millisecond is under half the window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each direction of a connection is a kernel message queue: the bytes of its messages in the frames #1074 takes as a pipe fills, and beside them the messages' lengths and their handles, in the order they were sent. A send admits the whole message under the queue's lock or refuses it with nothing of it queued and every handle back at its number; a receive takes the head message whole into the caller's buffers, its handles installed under the same lock, or leaves it queued. No stream position exists any more, so nothing pins a batch of handles at one. The bounds are the approved ones, per direction: 2 MiB of bytes, taken as used, 256 messages and 64 handles. Without room the send waits, or answers WouldBlock; a receiver's buffer too small for the head message is InvalidArgument and a table without room for its handles is ResourceExhausted, both leaving it queued; a peer gone is Gone. A connection polls writable once the largest message would fit, so a refused non-blocking send retried on readiness lands. ABI: SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone, and plain read and write on a connection are refused PermissionDenied: a connection is never a byte stream. SDK: a frame is a 4-byte message type and its payload, the kernel reporting length and handle count, so IpcHeader goes. FrameRx is a buffer and one receive, non-blocking (pump) or blocking (recv), and replaces recv_header/recv_payload/recv_bytes/recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer instead of a zeroed 8 KiB one. Every consumer moves to it in this commit. netstack's Client::gone asks with a non-blocking receive instead of a one-byte read. The link (#1106) rang its doorbell as a byte on its connection, the last such doorbell; it rings an event each way now, as the block service's do. The hello carries the bell the driver rings, the answer the bell the stack rings, and a bell whose waker is gone is the link's end, so the connection goes once the link is open. Deleted: kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch, try_read_within, the pipe's written/read counters and Landing (a pipe's write landing is its own rule now), the connection arms of the pipe read and write paths, the acceptor's close_now of each queued connection (a pending connection's receiver goes with it). Tests: the pure queue (order, wholeness, each bound, a refused receive taking nothing, a 50,000-step random run against the order sent); the SDK's framing and refusals; messages_are_whole, replacing message_carries_its_handles, for the syscalls' refusals and the bounds as a sender meets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each direction of a connection is a kernel message queue: the bytes of its messages in the frames #1074 takes as a pipe fills, and beside them the messages' lengths and their handles, in the order they were sent. A send admits the whole message under the queue's lock or refuses it with nothing of it queued and every handle back at its number; a receive takes the head message whole into the caller's buffers, its handles installed under the same lock, or leaves it queued. No stream position exists any more, so nothing pins a batch of handles at one. The bounds are the approved ones, per direction: 2 MiB of bytes, taken as used, 256 messages and 64 handles. Without room the send waits, or answers WouldBlock; a receiver's buffer too small for the head message is InvalidArgument and a table without room for its handles is ResourceExhausted, both leaving it queued; a peer gone is Gone. A connection polls writable once the largest message would fit, so a refused non-blocking send retried on readiness lands. ABI: SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone, and plain read and write on a connection are refused PermissionDenied: a connection is never a byte stream. SDK: a frame is a 4-byte message type and its payload, the kernel reporting length and handle count, so IpcHeader goes. FrameRx is a buffer and one receive, non-blocking (pump) or blocking (recv), and replaces recv_header/recv_payload/recv_bytes/recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer instead of a zeroed 8 KiB one. Every consumer moves to it in this commit. netstack's Client::gone asks with a non-blocking receive instead of a one-byte read. The link (#1106) rang its doorbell as a byte on its connection, the last such doorbell; it rings an event each way now, as the block service's do. The hello carries the bell the driver rings, the answer the bell the stack rings, and a bell whose waker is gone is the link's end, so the connection goes once the link is open. Deleted: kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch, try_read_within, the pipe's written/read counters and Landing (a pipe's write landing is its own rule now), the connection arms of the pipe read and write paths, the acceptor's close_now of each queued connection (a pending connection's receiver goes with it). Tests: the pure queue (order, wholeness, each bound, a refused receive taking nothing, a 50,000-step random run against the order sent); the SDK's framing and refusals; messages_are_whole, replacing message_carries_its_handles, for the syscalls' refusals and the bounds as a sender meets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each direction of a connection is a kernel message queue: the bytes of its messages in the frames #1074 takes as a pipe fills, and beside them the messages' lengths and their handles, in the order they were sent. A send admits the whole message under the queue's lock or refuses it with nothing of it queued and every handle back at its number; a receive takes the head message whole into the caller's buffers, its handles installed under the same lock, or leaves it queued. No stream position exists any more, so nothing pins a batch of handles at one. The bounds are the approved ones, per direction: 2 MiB of bytes, taken as used, 256 messages and 64 handles. Without room the send waits, or answers WouldBlock; a receiver's buffer too small for the head message is InvalidArgument and a table without room for its handles is ResourceExhausted, both leaving it queued; a peer gone is Gone. A connection polls writable once the largest message would fit, so a refused non-blocking send retried on readiness lands. ABI: SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone, and plain read and write on a connection are refused PermissionDenied: a connection is never a byte stream. SDK: a frame is a 4-byte message type and its payload, the kernel reporting length and handle count, so IpcHeader goes. FrameRx is a buffer and one receive, non-blocking (pump) or blocking (recv), and replaces recv_header/recv_payload/recv_bytes/recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer instead of a zeroed 8 KiB one. Every consumer moves to it in this commit. netstack's Client::gone asks with a non-blocking receive instead of a one-byte read. The link (#1106) rang its doorbell as a byte on its connection, the last such doorbell; it rings an event each way now, as the block service's do. The hello carries the bell the driver rings, the answer the bell the stack rings, and a bell whose waker is gone is the link's end, so the connection goes once the link is open. Deleted: kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch, try_read_within, the pipe's written/read counters and Landing (a pipe's write landing is its own rule now), the connection arms of the pipe read and write paths, the acceptor's close_now of each queued connection (a pending connection's receiver goes with it). Tests: the pure queue (order, wholeness, each bound, a refused receive taking nothing, a 50,000-step random run against the order sent); the SDK's framing and refusals; messages_are_whole, replacing message_carries_its_handles, for the syscalls' refusals and the bounds as a sender meets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A NIC driver's hello carried the bell the driver rings, and netstack used it typed on every receive pass. A handle of another kind in that slot, even from a plain driver bug, ended the whole stack with a handle fault, and with it every stream and the healthy second link #1106's failover keeps. No check settles an event's kind, so netstack cannot refuse it. The serving side now makes every bell, as blockserve and netstack's own stream bells already do: Session::open makes both pairs and sends the driver its waker and its bell with MSG_OPENED (OPENED_HANDLES). The hello carries only the region, which Region::lent checks, and HELLO_HANDLES goes. netstack holds only events it made; only the driver trusts received bells, as diskserver does the block server's. The driver's departure is still its waker's last handle, so a hang-up keeps its meaning. Also the review's suggestions: the receive bounds' comment reads, the pipe-lock record is rewrapped with its page-population sentence on both copies, a double blank line goes, and response_bytes says its buffer must hold an error answer too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each direction of a connection is a kernel message queue: the bytes of its messages in the frames #1074 takes as a pipe fills, and beside them the messages' lengths and their handles, in the order they were sent. A send admits the whole message under the queue's lock or refuses it with nothing of it queued and every handle back at its number; a receive takes the head message whole into the caller's buffers, its handles installed under the same lock, or leaves it queued. No stream position exists any more, so nothing pins a batch of handles at one. The bounds are the approved ones, per direction: 2 MiB of bytes, taken as used, 256 messages and 64 handles. Without room the send waits, or answers WouldBlock; a receiver's buffer too small for the head message is InvalidArgument and a table without room for its handles is ResourceExhausted, both leaving it queued; a peer gone is Gone. A connection polls writable once the largest message would fit, so a refused non-blocking send retried on readiness lands. ABI: SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone, and plain read and write on a connection are refused PermissionDenied: a connection is never a byte stream. SDK: a frame is a 4-byte message type and its payload, the kernel reporting length and handle count, so IpcHeader goes. FrameRx is a buffer and one receive, non-blocking (pump) or blocking (recv), and replaces recv_header/recv_payload/recv_bytes/recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer instead of a zeroed 8 KiB one. Every consumer moves to it in this commit. netstack's Client::gone asks with a non-blocking receive instead of a one-byte read. The link (#1106) rang its doorbell as a byte on its connection, the last such doorbell; it rings an event each way now, as the block service's do. The hello carries the bell the driver rings, the answer the bell the stack rings, and a bell whose waker is gone is the link's end, so the connection goes once the link is open. Deleted: kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch, try_read_within, the pipe's written/read counters and Landing (a pipe's write landing is its own rule now), the connection arms of the pipe read and write paths, the acceptor's close_now of each queued connection (a pending connection's receiver goes with it). Tests: the pure queue (order, wholeness, each bound, a refused receive taking nothing, a 50,000-step random run against the order sent); the SDK's framing and refusals; messages_are_whole, replacing message_carries_its_handles, for the syscalls' refusals and the bounds as a sender meets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A NIC driver's hello carried the bell the driver rings, and netstack used it typed on every receive pass. A handle of another kind in that slot, even from a plain driver bug, ended the whole stack with a handle fault, and with it every stream and the healthy second link #1106's failover keeps. No check settles an event's kind, so netstack cannot refuse it. The serving side now makes every bell, as blockserve and netstack's own stream bells already do: Session::open makes both pairs and sends the driver its waker and its bell with MSG_OPENED (OPENED_HANDLES). The hello carries only the region, which Region::lent checks, and HELLO_HANDLES goes. netstack holds only events it made; only the driver trusts received bells, as diskserver does the block server's. The driver's departure is still its waker's last handle, so a hang-up keeps its meaning. Also the review's suggestions: the receive bounds' comment reads, the pipe-lock record is rewrapped with its page-population sentence on both copies, a double blank line goes, and response_bytes says its buffer must hold an error answer too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each direction of a connection is a kernel message queue: the bytes of its messages in the frames #1074 takes as a pipe fills, and beside them the messages' lengths and their handles, in the order they were sent. A send admits the whole message under the queue's lock or refuses it with nothing of it queued and every handle back at its number; a receive takes the head message whole into the caller's buffers, its handles installed under the same lock, or leaves it queued. No stream position exists any more, so nothing pins a batch of handles at one. The bounds are the approved ones, per direction: 2 MiB of bytes, taken as used, 256 messages and 64 handles. Without room the send waits, or answers WouldBlock; a receiver's buffer too small for the head message is InvalidArgument and a table without room for its handles is ResourceExhausted, both leaving it queued; a peer gone is Gone. A connection polls writable once the largest message would fit, so a refused non-blocking send retried on readiness lands. ABI: SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone, and plain read and write on a connection are refused PermissionDenied: a connection is never a byte stream. SDK: a frame is a 4-byte message type and its payload, the kernel reporting length and handle count, so IpcHeader goes. FrameRx is a buffer and one receive, non-blocking (pump) or blocking (recv), and replaces recv_header/recv_payload/recv_bytes/recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer instead of a zeroed 8 KiB one. Every consumer moves to it in this commit. netstack's Client::gone asks with a non-blocking receive instead of a one-byte read. The link (#1106) rang its doorbell as a byte on its connection, the last such doorbell; it rings an event each way now, as the block service's do. The hello carries the bell the driver rings, the answer the bell the stack rings, and a bell whose waker is gone is the link's end, so the connection goes once the link is open. Deleted: kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch, try_read_within, the pipe's written/read counters and Landing (a pipe's write landing is its own rule now), the connection arms of the pipe read and write paths, the acceptor's close_now of each queued connection (a pending connection's receiver goes with it). Tests: the pure queue (order, wholeness, each bound, a refused receive taking nothing, a 50,000-step random run against the order sent); the SDK's framing and refusals; messages_are_whole, replacing message_carries_its_handles, for the syscalls' refusals and the bounds as a sender meets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A NIC driver's hello carried the bell the driver rings, and netstack used it typed on every receive pass. A handle of another kind in that slot, even from a plain driver bug, ended the whole stack with a handle fault, and with it every stream and the healthy second link #1106's failover keeps. No check settles an event's kind, so netstack cannot refuse it. The serving side now makes every bell, as blockserve and netstack's own stream bells already do: Session::open makes both pairs and sends the driver its waker and its bell with MSG_OPENED (OPENED_HANDLES). The hello carries only the region, which Region::lent checks, and HELLO_HANDLES goes. netstack holds only events it made; only the driver trusts received bells, as diskserver does the block server's. The driver's departure is still its waker's last handle, so a hang-up keeps its meaning. Also the review's suggestions: the receive bounds' comment reads, the pipe-lock record is rewrapped with its page-population sentence on both copies, a double blank line goes, and response_bytes says its buffer must hold an error answer too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
netstack held every NIC driver and the stack in one process, behind
Card, and each image's row named the card its machine has (devices = ["pci:02:00:00/1af4:1041"]). A driver's fault ended every connection on the machine. A driver could not start again without the stack's state. A machine nobody wrote a row for had no network: the shipped image named virtio alone, so the T14's I219 came up only inbenchcase. This is stage 3 ofissues/the-lan-is-not-yet-production-grade.md, which is P4 ofissues/there-is-no-wifi.md. W5 needs its match table.What changes
toyos-link(lib/link) is the protocol between netstack and each NIC driver. It is built ontoyos::transportand adds no transport primitive.linkport and sends a region it made with its hello (MAC address and name). netstack answersOPENEDorREFUSED. After that the connection carries only doorbell bytes, and its hang-up ends the link. This is blockring's shape.nis in slotnof its ring, so a slot is its producer's again exactly when the consumer releases its entry. There are no return queues. Each side copies a frame once (the model S2 measures).AcqRelswap of the want's word, so whichever swap comes second in that word's order sees the other.tests/ends_loom.rschecks this under loom, and a gate control reds it with the swapsRelaxed.Link::decode, and groups by count and encoding. A fault ends the link by name. Counter values are displayed ininspectand never acted on, so no arithmetic touches them. That was the orchestrator's condition for putting counters in the region.system/i219drives the I219 and the 82574, withtoyos-i219moved under it unchanged but for comments.system/virtio-netis the formernetstack/src/virtio_net.rs, without itsRefCells.drivesrow. The supervisor's existing match table binds one process per function by class, vendor and device, so the supervisor itself is unchanged.restart = true: the supervisor starts it again when it ends other than by exit 0.issues/no-machine-has-read-an-intel-nic-across-a-link-change.mdrecords a reset over a ring left by a link change as not known to be safe, with a T14 hang on record. That issue's rule now names the I219 row.toyos_link::driver::drive: one pass per wake, at most a ring of frames each way. When netstack hangs up, the driver connects again with a fresh region and its device stays up.Card,card.rsandCARDSare gone.toyos-link's own checkedAddress, an individual one. A group's address, or a name that is not ASCII letters, digits and-, is refusedMalformedat decode.IndividualMacfrom that address once, as it hears the hello, and refusesMalformedshould the conversion ever fail. No driver's hello reaches a panic in netstack, and the drivers do not depend on netstack's wire crate.OneLinkby name. So is a hello with the node's address while that link is still alive, which netstack reads on the connection at that moment. The kernel closes a dead driver's handles before its end is published, so a restarted driver's hello always finds the old link already hung up.Faulted. The driver ends on that refusal, so the supervisor's restart budget (3 in 10 s, then given up by name) bounds a driver that keeps doing it.inspectasks the driver for its counters and answers once it has them, or after 1 s without them.images/system.tomldeclares both drivers, so the shipped image now drives the T14's I219 with no row naming the T14.netcasedeclares both drivers,benchcasethe I219, andmetalcasevirtio. The newlinkcasedeclares none, so its job runs the driver itself.netstack-ends-for-the-boot-on-a-nic-it-cannot-drive-onis closed: its exit is the restart test below.netstack-drives-one-cardsay what is built and what is owed.Decisions
issues/netstack-drives-one-card.md), and pre-empting it here would be redone. So netstack serves nothing until the first hello, and leaves afterLINK_BOUND(30 s) on a machine where no driver says hello.issues/netstack-drives-one-card.md, with "a node that serves before any link, soLINK_BOUNDgoes" added to P5's exit.--drivers <n>, as it does for the block service. It would have removed the delay, but it couples the supervisor to netstack, P5 deletes it, and no guest job could then start the driver itself for the restart test.linkchanges along with blockring, with no bridge.Checks
All runs are on macOS (arm64 host), QEMU x86-64. Every log, its command, revision and exit code is indexed in README.
cargo run -- --ci host, 107 steps, includingtoyos-link's host tests and the new loom controlcargo test --test toyos-build -- netstack_holders_exported_to_children a_link_ends_under_a_held_stream_and_comes_back udp_receive_times_out a_virtio_function_without_access_platform_is_refused iommu_virtio_platformcargo test -p toyos-link --test ends_loomAcqRelswaps--features mutate-want-relaxed, which reds with loom's deadlock (a lost wake)--metal-readbackcommand:i219_carries_netstacks_linkandi219_leaves_ulp_unarmedpassed, the I219 up at 1000 Mb/s full duplex 2,658 ms after its driver came uplib/link/tests/ends_loom.rs(&*pageto&page, a clippy finding), which no image contains.linkcasedid not ship/system/bin/virtio-net.iommu_virtio_platform's no-unit arm stopped waiting before the supervisor refused the driver row, which the supervisor now starts last.Round 2: the review's six blockers
cargo run -- --ci host, 107 stepsFaulted,OneLinkonly for another address or a live link, and pending links let go after the handshake bound, the first wait included.inspectwaits is asked.restartremoved from the I219 rows.82574link on QEMU'se1000eruns the samei219binary and the same netstack code, and it passed at b5f9cc0 and again at 4864cf7. The T14's individual address and the namei219pass the hello's checks. Removingrestartchanges nothing on a boot that passes.Round 3: the link's address is its own type
The round-2 review's one blocker was that
toyos-linkdepended onsystem/netstack/wirefor one six-byte type. That pulled Ethernet, IPv4 and TCP wire formats andtoyos-cryptointo both drivers.lib/linknow has its own checkedAddress, andcargo tree -p i219no longer namestoyos-net-wireortoyos-crypto.cargo test -p toyos-link(wire tests of the address and name)Round 4: the link's loom control reaches its verdict on Linux
The merge queue dropped the PR:
mutate-want-relaxedprinted noFAILEDline on the Linux runner (run 40782639843). The model put its 768-word page of loom atomics on loom's 32 KiB coroutine stack. Unwinding the deadlock overflowed that stack and aborted the test binary. With the page on the heap, a second abort followed: loom'sArcpanics in its drop while a failed model unwinds (issues/loom-arc-panics-in-its-drop-while-a-failed-model-unwinds.md). The model now shares oneRc, astransport_loomdoes, and collects its page on the heap. Each fix alone still aborts; the probes are logged. The issue's exit now names this model too.cargo run -- --ci hostat 6807935 on origin/main 0c7e62c: exit 0, 107 steps,control mutate-want-relaxed: 1 verdict(s) reached.Integration tests and where they run
a_link_ends_under_a_held_stream_and_comes_back(new)netstack_holders_exported_to_children, now also over QEMU'se1000e(newHeadlessE1000eprofile)i219_carries_netstacks_link(new metal row)i219_leaves_ulp_unarmed's boot. Its new joblan_leasewaits on the served log for netstack's lease, then the judge reads that the link opened, came up and carried a lease, and never ended.benchcasenow serveslogto its runner.Owed
T14 boot. Done: both rows passed (above).
AMD bench. No row: the AMD bench has no wired NIC ToyOS drives.
S2: done. It is a scout and never lands. On the T14 (TSC 2.42 GHz), 2^18 frames of 1,514 bytes crossed the link between two processes, each side copying each frame once (S2, judge).
Risks
a_link_ends_under_a_held_stream_and_comes_backkills a driver its job started and starts another, and the stream survives. The supervisor's own restart of adrivesrow is held by the lifecycle crate's host tests. No single test shows the supervisor restarting a NIC driver under a held stream.i219:,82574:andvirtio-net:. Updated: the iommu judges, the I219 judge, and the issues that quote a line.🤖 Generated with Claude Code