[Japabu]

Phase 2 of the owner-approved IPC migration, on main after #1108 (phase 1, events and SYS_HANDLE_WAIT). Draft. The call (SYS_MESSAGE_CALL) is out of this PR by the orchestrator's decision. It goes to a follow-up that carries parks_next through a poll's delivery and is measured on the fs rows. The scheduler commit b24b37a56 (#1111) that was cherry-picked for the call is gone with it.

Problem

A connection was two byte pipes plus two handle queues that pinned each batch of handles at a stream position. Plain read and write worked on it, and a write could land partly. The SDK rebuilt [type][len][handles] frames out of partial reads, which made a blocking receive two or three syscalls: header, payload, and the handle receive. A send over 64 bytes zeroed an 8 KiB buffer. The fs-rtt split (T14) puts about 5 µs of a 9.56 µs file round trip in this transport: the send body before the kick 1.84 µs, the post-resume finish 1.43 µs, and the extra frame reads 1.76 µs. This PR removes the extra reads; the call's follow-up targets the wake.

Result

A connection carries whole messages (a connection carries whole messages):

• Kernel. Each direction is a message queue. It holds the messages' bytes in the frames kernel: a pipe keeps its bytes in 4 KiB frames it takes as it fills #1074 takes as a pipe fills, and beside them each message's length and its handles, in the order sent. A send admits the whole message under the queue's lock, or refuses it with nothing queued and every handle back at its own number. A receive takes the head message whole into the caller's buffers and installs its handles under the same lock, or leaves it queued.
• Approved bounds, per direction: 2 MiB of bytes, taken as used; 256 messages; 64 handles; 16 KiB per message.
• Refusals. With no room, a send waits, or answers WouldBlock when non-blocking. A buffer too small for the head message is InvalidArgument, and a handle table with no room is ResourceExhausted; both leave the message queued. A peer gone is Gone, after whatever it sent before going has been taken.
• Writable means the largest message fits now, so a refused non-blocking send retried on readiness lands.
• ABI. SYS_MESSAGE_RECV takes one message. SYS_HANDLE_RECV, HandlesPending and MAX_QUEUED_BATCHES are gone. Plain read and write on a connection answer PermissionDenied.
• SDK. A frame is a 4-byte message type and its payload, and the kernel reports its length and handle count, so IpcHeader goes. FrameRx is a buffer with one receive, pump (never waits) or recv (waits). It replaces recv_header, recv_payload, recv_bytes, recv_handles and the partial-frame state machine. A send frames into an uninitialised buffer.
• Consumers. Every consumer moves in the same commit: about 60 files across the SDK, sdk/window, lib/*, system/* and the guest tests.
• A receive lands where the caller keeps it. SYS_MESSAGE_RECV takes two places: the message's first bytes fill head, the rest bytes, and a message longer than both together is refused whole. The ABI gives the two places no meaning. The SDK receives the type word apart from the payload. Connection::recv_into takes a payload straight into the caller's own buffer, so a caller that keeps its payload pays no copy out of a FrameRx; osbench's peer and toyos::net::response_bytes use it. A FrameRx is a caller's buffer too.
• The link (netstack: every NIC driver is a process of its own, over a link (P4) #1106) was the last doorbell rung as a byte on a connection. It now rings an event each way. As in the block service and netstack's own stream bells, the serving side makes every bell: the hello carries only the region, which Region::lent checks, and MSG_OPENED hands the driver the waker it rings the stack with and the bell the stack rings it on. netstack therefore holds no event a driver sent, so a driver that sends the wrong handle cannot end the stack (no check can settle an event's kind, issues/a-received-handle-has-no-knowable-type.md). A bell whose waker has gone is the link's end.

Deleted

• kernel/pure/message.rs's batches and windows, HandleQueue, AtBatch and try_read_within.
• The pipe's written and read counters and Landing.
• The connection arms of the pipe read and write paths.
• The acceptor's per-connection close_now: a pending connection's receiver goes with it.
• Connection::{read_nonblock, write_nonblock}, IpcHeader and the SDK's blocking recv_* functions.
• libc's EPROTO mapping.

Checks

Head b45676390, rebased onto main 4753fb875 (#1123):

• 91b68e796: whole messages.
• ab0209bca: the session_lifecycle repair.
• 8aa8ccf2b: the console-loss issue record.
• aadce7fab: the receive into the caller's buffer.
• b45676390: review r2, netstack makes both of a link's bells, plus the review's suggestions.

Renumbered after #1104 and #1123. SYS_MESSAGE_RECV is 159: #1104 took 152 and #1123 took 153–158. SYSCALL_PROFILE_BINS is 161, and the compile guard checks SYS_MESSAGE_RECV. The ABI file conflicted in both rebases, and nothing else did. At b45676390: cargo run -- --ci host exit 0, 108/108; cargo run -- --build-only exit 0 (evidence).

The QEMU and metal results below were taken at a16f09c3a. The rebases add syscall numbers and bring in #1104's clock and #1123's paged objects. Host CI and the image build at the new head show that #1123's code compiles against whole messages, but no guest test has run at b45676390.

Actuator members on QEMU at a16f09c3a. The eight ACTUATOR_TESTS members need SYS_DEBUG, and --debug boots no kernel that has it. A measurement-only commit (60f3dd99c, branch ipc-messages-actuator-qemu, never to land) made --debug boot the boot-actuators,test-actuators kernel instead. All eight exited 0: abuse_kernel_addr, handle_lifetime, shm_release_reclaims, spawn_returns_the_census, spawn_lands_claimed, spawn_child_ends_first, counters_silent and ring0_timer_in_syscall. The control is review round 1's run on a kernel without SYS_DEBUG, where the six that check for it failed their own vacuity guards (evidence).

The AMD memcpy issue record is dropped: #1133 finds and fixes its cause in libc.

The rebase conflicted only in netstack's link opening, where #1106's follow-ups give a link its node iface in place of its MAC: Session::open takes the iface.

Evidence: first round, review round 1, round 2, ipcwake, round 4, landing A/B and rebase, review round 2.

At a16f09c3a: cargo run -- --ci host: exit 0, 107/107, and cargo run -- --build-only: exit 0, both run at ccaf1b1cd, which differs only by the dropped issue file. QEMU x86-64 (remote runner) at a16f09c3a, the netstack registrations, which the link's bells and the rebase's conflict touch: traffic_leaves_by_the_second_link_while_the_first_is_down (#1106's failover, new on main), a_link_ends_under_a_held_stream_and_comes_back, netstack_holders_exported_to_children, stream_holders, a_client_killed_mid_connect_gives_its_place_back and udp_receive_times_out: 6/6 green.

Landing A/B on metal (base 18060b521, head c715ebfb5, osbench, four interleaved boots per bench, all passed. It measures this PR's delta on its pre-rebase base, and the rebase does not change that delta except in netstack's link opening, which osbench does not drive; rule stated before the run: mean p50 per arm within 5% of base on ipc.reqrep.{8,4096,65536}, ipc.reqrep_handle.8 and every fs.*.direct.* row):

row	T14 base → head	AMD base → head
ipc.reqrep.8	7176 → 6883 (−4.1%)	6668 → 6327 (−5.1%)
ipc.reqrep.4096	8758 → 8352 (−4.6%)	7775 → 7255 (−6.7%)
ipc.reqrep.65536	41336 → 41475 (+0.3%)	31631 → 30058 (−5.0%)
ipc.reqrep_handle.8	8757 → 8048 (−8.1%)	7970 → 7179 (−9.9%)
fs.disk.direct.* (5 rows)	−0.8% … +0.4%	+8.3% … +10.7% by the mean
ipc.connect_accept	−20.5%	−21.3%

• The AMD ipc.reqrep.65536 regression is gone: 66 µs at whole messages without the receive into the caller's buffer, now 30.1/30.0 µs against base's 31.6/31.6.
• AMD fs rows miss the rule by the mean; I attribute the miss to one boot's mode, not to this change. head-3 sits 1–3% under both base boots on every fs row; head-2 sits 20–26% over. On head-2, rows this change does not touch slow in proportion (pingpong.futex.same_cpu 1.30×, pingpong.pipe.same_cpu 1.31×; cross-core and tmpfs 1.0×). Round 1's base boot base-6, a tree without whole messages, shows the same ratios (fs direct 1.22–1.26×, futex same-CPU 1.32×, pipe same-CPU 1.35×, cross-core 1.01×). That the mode is unrelated is an inference from those matching signatures; the rule as written is not met, and one more AMD pair from the same images would settle it.

At c715ebfb5 (before the rebase; its one conflict was netstack's link opening, whose registrations are rerun above; the rest of main's changes merged without conflict and are covered by host CI and the build):

• cargo run -- --ci host: exit 0, 107/107.
• QEMU x86-64, the 18 registrations below: 18/18 green.
• The --debug shared boot: messages_are_whole (with the split receive's new check), connect_before_serve, netstack_gone_mid_bind, pipe_other_end_gone, abuse_shared_grant, abuse_connect_flood, cpal_drop_unreleased, mutual_kill and window_refusal, all green.

Earlier results, at the trees named:
Host: cargo run -- --ci host at 343a655b1: exit 0, 107/107. The test repair touches only a guest test and a test image, which host CI does not build.

Pure and SDK tests:

• The queue: order, wholeness and each message's own handles; a refused receive takes nothing; each bound; writable; a 50,000-step random run.
• The SDK: framing, the refusals a server steps on, and the errors a waiting client gets.
• Negative controls, each applied alone and red: q1, q2, q3 (the random test was strengthened after it stayed green), s1, b1.

QEMU x86-64 registrations (remote runner, 343a655b1), 18 of 19 green:

• netstack and both links;
• the kernel's objects;
• launch and shutdown;
• disks, audio, the compositor and a window;
• osbench's capability groups and the console;
• libc_children, memory_floor_holds_the_controls, usbd_drives_the_spare. libc_children's tty_interrupt stalled earlier on trees carrying the scheduler commit and passed in this run; the stall also reproduces on main (1 of 4), filed as issues: libc_children's tty_interrupt can stall at the nested shell's line #1127, so neither run attributes it.

The red was session_lifecycle; see the repair below.

QEMU x86-64, the --debug shared boot (runner, d878eb102, then 2ee4db1e3): every shared member green but the six actuator members, which need a SYS_DEBUG kernel. Those differ from 343a655b1 by the call and the scheduler commit; neither is something a shared member drives, apart from the call test now gone.

session_lifecycle repair (edc8770e4):

• The job took the holder as ready once the roster listed it. A process that has not yet listened for its quits is ended by the first one, so a stop or sign-out asked in that window killed the holder before it listened, and the machine rebooted under the job.
• Whole messages made the job fast enough to land there: red at 343a655b1 twice, base green.
• The job now detaches the holder itself, through the session's launcher, with its stdout a pipe, and waits for the line the holder says once it listens.
• A second red remains, and it is not this change: the forced reboot's last userland lines (the job's marker, or the supervisor's forced-stop line) sometimes never reach the console. The kernel's records of the forced stop do.
  • Interleaved runs on the runner: base 18060b521 1 red of 4, head 3 red of 6.
  • Faster connections make it likelier but did not create it. Recorded as issues/a-forced-reboots-last-userland-lines-miss-the-console.md.

Metal, T14 and AMD, every boot passed (images at 4fc78ac97, 245f7b9af, 2269ea100):

• messages_are_whole and connect_before_serve exit 0 on both.
• osbench A/B, whole messages against base (then 4fc78ac97), two boots per arm:
  • T14 better on every IPC and fs row (ipc.reqrep.8 7222 → 6462 ns, fs.disk.direct.seek 9010 → 7740 ns).
  • AMD better on ipc.reqrep.8, ipc.reqrep_handle.8 and the fs rows.
  • AMD ipc.reqrep.65536 doubles (31.1/51.0 → 66.3/66.2 µs), and ipc.reqrep.4096 sits at base's slow mode on all four boots. Fixed by the receive into the caller's buffer; see the landing A/B above.

Why metal: the A/B is timing, which QEMU cannot establish, and the members are the kernel boundary's refusals and handle moves, on both benches.

Risks and open items

• AMD fs rows, one slow boot. See the landing A/B: the gate is met on every row except AMD's fs direct rows by the mean, attributed by signature to a boot mode also seen on a base boot. Not verified beyond that match.
• T14 wake.onto_busy staging. Whole-message trees had 3 of 6 boots refused before. A placement A/B (head and base, two boots each) refused on none, so the refusal is not shown to be this PR's. On both head boots, though, osbench's bench thread migrated to CPU 0, the IRQ CPU, during staging (base: 0 of 2), and one head boot's row went unmeasured. That migration is attributed to the scheduler moving an unpinned bench thread, an inference handed to the scheduler track (evidence).
• The AMD userland copy cost (2645 against the T14's 134 ns per 8 KiB copy_from_slice) is 4K store-to-load aliasing in libc's copy, found and fixed by libc: a forward copy a short way above its source modulo 4 KiB copies down #1133.
• session_lifecycle fails about half the time at the head (3 of 6 runs; base 1 of 4) on the pre-existing console loss above. Its fix belongs to the console/stop path. It is not a required merge check: main's ruleset requires only host and image, and the guest suite runs nightly.
• MAX_FRAME_LEN stays 8192 under the kernel's 16 KiB message bound.
• A FrameRx sized MAX_FRAME_LEN is 8 KiB, zeroed once, at the blocking client sites that take answers of any length.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1121 at 2269ea100 (base 4fc78ac97)

Architecture

The design is the approved one. Each direction is a single queue (kernel/src/message.rs over pure kernel::message::Queue), with bytes, lengths and handles under one lock. That deletes the stream positions, HandleQueue, AtBatch, Landing::Partial and the SDK's partial-frame state machine, so the deletion is real. The frame code moves out of pipe.rs into frame_bytes.rs, and pipe and queue now share it rather than each keeping a copy. Net change: kernel and ABI about +270 lines (pure tests included); production plus unit tests +100; guest tests +170. No bridge is left: recv_header, recv_payload, recv_bytes, recv_handles, IpcHeader, HandlesPending, handle_recv, MAX_QUEUED_BATCHES and post_boosted appear nowhere in code. The link moves to events in both directions. The approved bounds and named refusals are implemented as stated.

What I checked and found sound:

• Send. It checks the receiver, then fits (bytes, messages, handles), then FrameBytes::write (all frames or none), then push, all under one hold. Every refusal returns the batch, so transfer puts each handle back at its number.
• Receive. It asks for table room before pop, installs the handles and copies the bytes under the same hold. A refusal takes nothing.
• Peer death. Gone comes only when the queue is empty and no sender is left. The last receiver's release drops the queued messages and handles outside the lock and posts the write watch. The acceptor's close_now correctly goes away with the receivers.
• Charging. The heap is charged at its bounds, and the frames are counted to the connecting job, as before.
• parks_next. At kernel/src/syscall/ipc.rs:473 it is receives && !conn.rx().readable(), evaluated by the kernel after the request is queued. No flag in the ABI or SDK can assert it. If a reply races in first, the only cost is a placement hint that turns out wrong; correctness is unaffected. This meets the approved requirement.
• Negative controls. q1, q2, q3 and s1 are each red, on 8fb61777d, and focused. The q3 follow-up is right: the random test now checks that what it takes fits the reader, and is red under q3 (q3-short-buffer-taken-2.log).
• Scheduler commit. 4fc78ac97 is byte-identical to b24b37a56. sched: a wake its poster parks or exits behind runs where it was posted #1111 has since been rebased to 0e756f34e plus a tripwire commit, and keeps the same WakeCause::parks_next API.

Findings

BLOCKER 1. sdk/toyos/src/ipc.rs:460 (Refused::blocking): a peer's oversized answer panics the fs client.

• Mismatch. The kernel answers InvalidArgument when the head message is larger than the receiver's buffer or handle slots. blocking turns that into IpcError::Syscall(InvalidArgument). But FrameRx::recv (:500) and IpcError::Malformed (:114) both document that case as Malformed, and step does map it to Malformed.
• Consequence. In toyos::fs (Dir::call :831, hello :1264), transport passes Syscall(e) to Error::kernel, which panics on InvalidArgument (fs.rs:374). So a file server whose reply is longer than Reply now kills its client. Before this change, recv_payload answered Malformed.
• Fix. Map Kernel(InvalidArgument) to Malformed in blocking, and settle ResourceExhausted there explicitly. Add the blocking twin of a_receive_refused_is_the_step_a_server_acts_on. The existing test covers only step, which is why this is not caught.

BLOCKER 2. sdk/abi/src/syscall.rs:2217-2240 and kernel/src/syscall/ipc.rs:435: a call that carries handles cannot tell whether they moved.

• Ambiguity. message_call accepts request handles. But Gone, InvalidArgument and ResourceExhausted can each come from either half: from a refused send (nothing moved) or from the receive after the request was queued (the handles are the peer's).
• Consequence. The caller cannot meet its own # Safety contract. With the SDK's close-on-Err pattern (move_batch), it would close numbers it no longer owns, and those numbers may already be reused.
• No user. No caller sends handles with a call: Connection::call passes &[], and so does osbench.
• Fix. Make the call carry no request handles: drop the parameter and have the kernel refuse a non-zero count. The other option, an answer that says which half refused, is machinery for an imagined caller.

BLOCKER 3. sdk/libc/src/errcode.rs:70 and sdk/std/sys/pal/mod.rs:35: stale comments. Deleting the HandlesPending arms left their comments behind. "A read of a connection met handles…" now sits above _ => EINVAL and above the closing };. Both are false; delete them.

BLOCKER 4. issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md:13: an issue record this change directly affects is now stale.

• Stale name. It names PipeRef::try_read_within, which this PR deletes. The copies now live in FrameBytes::read and FrameBytes::write (kernel/src/frame_bytes.rs), which pipe and queue share.
• New fact to record. A connection's copy is now bounded at 16 KiB per message. On receive it runs under the queue lock nested inside with_process_data, which only the send path did before.
• Fix. Update the record to the current names and this fact.

BLOCKER 5. PR body, "Checks": material changed guest code has neither run nor been staged.

• Claim. The body says the two members messages_are_whole and connect_before_serve are what QEMU did not boot.
• Actually unrun. All of the following were changed in this PR, and none appears in qemu-p2-1.log / qemu-p2-2.log (which ran at 455a1822e):
  • Shared-boot members: netstack_gone_mid_bind, pipe_other_end_gone, abuse_shared_grant, cpal_drop_unreleased, mutual_kill, window_refusal. netstack_gone_mid_bind's ordering argument is newly rewritten: the receiver is dropped before the sender, so Gone now means "can no longer be sent to".
  • Actuator members: shm_release_reclaims, spawn_lands_claimed.
  • Code driven by their own registrations: memory_floor (memory_floor_holds_the_controls), libc_recv_beside_close (libc_children), usbd_spare (usbd_drives_the_spare), process_tree (metal), and master.rs's users.
• Not staged either. The staged metal testcases boot carries only those two jobs (h2-prep.log: "testcases: 2 job(s)"), so none of the above is covered there.
• Wider exposure. Every unchanged shared member also now runs over the new transport, and none has booted.
• Cheapest fix. One x86-64 QEMU --debug shared boot once its paths are free (it boots every shared and actuator member), plus a filtered run of memory_floor_holds_the_controls, libc_children and usbd_drives_the_spare. Alternatively, stage testcases and shared-debug with the full job list.
• Body correction. The PR description should say what has actually run.

Pending, not a finding. The call (2269ea100) is conditional on the queued A/B (h2 against h1, beyond one syscall entry and exit). It cannot land before that result, whatever happens to the blockers above.

Suggestions

• tests/guest/rust-tests/src/bin/messages_are_whole.rs gone(): this is a one-shot Poller::new(1), which phase 1 replaced with SYS_HANDLE_WAIT. toyos::poller::wait_one, which diskserver uses, says the same in one call.
• kernel/src/message.rs QueueRef::recv: the binding let taken = match …; taken inside the block is redundant.

SEND BACK

[Japabu]

Review round 1 answered at 2ee4db1e3 (C1 40f5cc88a, C2 2ee4db1e3)

Evidence: https://github.com/ToyOSOrg/evidence/tree/72d897c7e9256b2a368446de785370586d538f25/ipc-messages/review-r1-2ee4db1e3

Blockers:

1. Fixed (C1). Refused::blocking maps the kernel's InvalidArgument, a frame larger than the receiver holds, to Malformed, as step does. A table without room (ResourceExhausted) stays Syscall(ResourceExhausted): that is this side's failure.
   • New host test: a_receive_refused_is_the_error_a_client_acts_on.
   • Control: the old mapping restored makes it red (control-b1.log).
2. Fixed (C2). A call carries no request handles.
   • ABI: message_call lost its handles parameter. It sends none, so it is now a safe fn.
   • Kernel: SYS_MESSAGE_CALL refuses a non-zero send.count with InvalidArgument.
   • Both docs say why: an error from either half would leave the caller not knowing whether the handles had moved.
   • No guest test, because the ABI type rules it out and the raw syscall entry is private to the ABI crate.
3. Fixed (C1). Both stale comments are deleted (libc's errcode.rs, std's pal/mod.rs).
4. Fixed (C1). The issue record names FrameBytes::read/write (frame_bytes.rs), which pipes and queues share. It records the connection's per-message bound (16 KiB), the copy under the queue lock nested inside with_process_data on both send and receive, and that nesting in its exit condition.
5. Run.
   • x86-64 QEMU --debug shared boot on the remote runner: all 79 shared members, driven one at a time (remote/debug-results.txt, at d878eb102).
     • 72 exit 0, including every member you named.
     • 6 actuator members exit 101 for want of SYS_DEBUG, which this boot's kernel does not carry: abuse_kernel_addr, handle_lifetime, shm_release_reclaims, spawn_lands_claimed, spawn_child_ends_first, counters_silent. They are still unrun on an actuator kernel. Their edits are the receive rename only.
     • abuse_connect_flood exposed one more miss: a plain write on an accepted connection. It sends a one-byte message now, and passes in a second boot at 2ee4db1e3, together with messages_are_whole.
   • Filtered runs at d878eb102: memory_floor_holds_the_controls and usbd_drives_the_spare pass. libc_children is red: tty_interrupt stalls at "the nested line held".
     • It is not this PR. The same stall appears at 4fc78ac97 (phase 2's parent).
     • libc_children passes on Wakeups are kernel events, and a one-handle timed wait is SYS_HANDLE_WAIT #1108 18060b521 and on main f45dc5e3b, so it arrives with the scheduler's b24b37a56 (sched: a wake its poster parks or exits behind runs where it was posted #1111), cherry-picked here. One run of each.
     • Reported for sched: a wake its poster parks or exits behind runs where it was posted #1111; not fixed here.
   • The PR body now says what has run.

Suggestions:

• gone() in messages_are_whole uses wait_one: taken.
• The let taken = …; taken binding is not redundant. Without it the build fails: a tail match's temporaries outlive guard (E0597). It now carries a one-line comment saying so.

Checks: host CI at d878eb102: exit 0, 107/107. Since then only abuse_connect_flood.rs changed, a guest file the host does not build.

The metal A/B and profile have run (both benches, every boot passed; the table is metal-judged/ab-table.txt). Means of p50, two boots per arm:

• Whole messages (h1 vs base):
  • T14: better on every IPC and fs row, for example ipc.reqrep.8 7222 → 6462 ns and fs.disk.direct.seek 9010 → 7740 ns.
  • AMD: better on ipc.reqrep.8, ipc.reqrep_handle.8 and the fs direct rows.
  • AMD regression: ipc.reqrep.65536 goes from 31.1/51.0 µs (base's two boots) to 66.3/66.2 µs (h1's two), and h2 matches h1. ipc.reqrep.4096 sits at base's slow mode on all four boots.
    • The T14 improves on both rows (65536: 41.1 → 38.1 µs).
    • Unexplained. My inference, not measured: the receiver now takes each 8 KiB frame in one syscall instead of two. It catches up with the sender and parks between frames, so each of the 16 frames pays a wake, and on the AMD that wake crosses an LLC. The next check would be per-op wake counts on that row.
• The call (h2 vs h1):
  • ipc.reqrep.8: −0.40 µs on the T14 and −0.33 µs on AMD. That is about one syscall entry and exit.
  • ipc.reqrep.4096: −0.97 µs on the T14, −0.33 µs on AMD.
  • fs direct rows: no gain. Each bench had one h2 boot in its slow mode, and the other h2 boot equals h1.
• The profile explains the fs result. In reqrep8 the call's wake never kicks: the server runs on the caller's CPU, as designed. In fsseek it kicks on 905/1000 (T14) and 999/1000 (AMD) calls, because the file server waits in an inbox poll and parks_next does not reach a poll's delivery. As it stands, the call is one syscall for every real server.
  • Send body (T14 fs call): enter→resolved 0.49, resolved→queued 0.40, queued→kick 0.36, kick 0.08, then 0.04 to posted (µs).
  • Post-resume (T14): recv resumed→taken 0.58 + 0.09; the server's inbox resumed→delivered 0.91.
• The call stays gated. By the rule it should go. It would earn its place only if parks_next reached a server waiting in an inbox, and that is a design question for the owner, not a change for this PR. My recommendation is to drop C2 from this PR unless the owner wants that extension.

[Japabu]

Round 2 state, head 73cf52240, re-review not yet requested

Evidence: https://github.com/ToyOSOrg/evidence/tree/d2b062ab56692fe08bedb6cf51d6afdd3b0a3bcb/ipc-messages/round-2-73cf52240

• The call is out. So is sched: a wake its poster parks or exits behind runs where it was posted #1111's b24b37a56, which only the call's parks_next used. IPC phase 2: a connection carries whole messages #1121 is whole messages on Wakeups are kernel events, and a one-handle timed wait is SYS_HANDLE_WAIT #1108 alone, and libc_children (tty_interrupt) passes again. The call and parks_next through a poll's delivery are the follow-up PR.
• session_lifecycle repaired (edc8770e4): the job took the holder as ready before it listened for its quits, and whole messages made it fast enough to land in that window. It now waits for the holder's own "holding" line.
  • A second red is pre-existing: the forced reboot's last userland lines sometimes miss the console. Base 1 of 4, head 3 of 6.
  • Recorded as issues/a-forced-reboots-last-userland-lines-miss-the-console.md.
• Host CI at 343a655b1: 107/107. QEMU: 18 of 19 registrations green; the red is session_lifecycle, above.
• The AMD ipc.reqrep.65536 regression is still open. The existing readbacks cannot attribute it. A measurement-only AMD batch is staged: base and whole messages, interleaved, with diary counts per round trip and the decision rule stated before the run (amd-wake/request.txt). I'll ask for the focused re-review once it is answered.

[Japabu]

Correction to round 1's reply: tty_interrupt's stall is not attributed to #1111's b24b37a56. It also reproduces on main (1 of 4 runs) and is filed as #1127. Its absence once that commit was gone is consistent with that, but proves nothing. The PR body now says so.

[Japabu]

From the scheduler agent, about this PR's AMD ipc.reqrep.65536 regression (31/51 → 66 µs). The coordinator asked me to tell you the plan.

• The finding: on the AMD bench, a request/reply pair split across the two LLCs reads about 25% slower. Today no placement keeps such a pair in one LLC. osbench: a row says where its processes ran #1126 gives every ipc and fs row a ran=name.pid@cpuN/llcL:share% field. In a 9-boot diagnostic, each ipc.reqrep.8 and fs.disk.read_4k_at value followed where its pair landed (placement table). Your bimodal base (31 against 51 µs) looks like the same split.
• The change: branch sched-wake-llc, on top of sched: a wake its poster parks or exits behind runs where it was posted #1111/sched: a thread runs only on the CPUs its process's mask names #1116/sched: a spawn wakes the shallowest of its equal free-core sleepers #1117, moves a task woken twice in a row from another LLC into that LLC.
• The A/B: 4 AMD boots per side, both with osbench: a row says where its processes ran #1126's field. ipc.reqrep.65536 is in those benchcase boots, so we will read its placement and p50 with and without the change. That measures main's ipc, not this PR's whole messages.
• For this PR: cherry-picking osbench: a row says where its processes ran #1126 (6adab470b) onto this branch would show whether its 66 µs runs are split. It applies to main cleanly, with one export-line conflict in osbench's arch/x86_64.rs.

[Japabu]

AMD ipcwake judged: neither the per-frame park hypothesis nor LLC placement

Evidence: https://github.com/ToyOSOrg/evidence/tree/87bf3a5ac862a2ac8ed2499fe51e2259f1353809/ipc-messages/round-3-ipcwake

• One park per side per round trip in both arms, at 4096 and 65536.
• Every pair ran inside LLC 0 on all four boots.
• So by the rule stated before the run, the AMD extra time (+1.9 µs at 4096, +35 µs at 65536) is in the copies, about 1 µs per 4 KiB frame and 2.2 µs per 8 KiB frame.

Next: one T14 and one AMD boot of a copy-split profile on ipc-messages-ab-copy (eb58854e8). Its rule chooses between fixing the kernel's queue copy and receiving into the caller's buffer. Re-review waits for that fix.

[Japabu]

Copy split judged; the second copy is gone (c715ebfb5)

Evidence: https://github.com/ToyOSOrg/evidence/tree/fa23028231cb89f038894931a7a00898fb18379e/ipc-messages/round-4-c715ebfb5

• AMD against T14, per 8 KiB frame: kernel copy out 591 vs 1074 ns, kernel copy in 591 vs 791 ns, userland copy out of the FrameRx 2645 vs 134 ns.
• By the rule stated before the run, the cost is the userland copy, and the fix is a receive into the caller's buffer.
• SYS_MESSAGE_RECV takes two places, a head and the rest. The whole-message boundary needs no intermediate buffer: the type word and the payload land apart, and a message too large for both together is refused whole. Connection::recv_into uses it; osbench's peer and toyos::net::response_bytes no longer copy.
• Checks at c715ebfb5: host CI 107/107, 18/18 QEMU registrations, and 9/9 debug-boot members.
• Separately, AMD userland memcpy runs at about a fifteenth of the T14's, recorded as an issue.

Next: the landing A/B, base 18060b521 against head c715ebfb5 on each bench, interleaved, with the rule in its request. The focused re-review follows its result.

[Japabu]

Review r2 of #1121 at 93adf1c83 (base main 3add613c1)

Scope: 4387feee3 (receive into the caller's buffer), e6e2fe789 (session_lifecycle repair), the round-1 fixes in d252db359, and the netstack conflict resolution. The call's move to a follow-up and the landing A/B judgment are settled and not reopened.

Architecture

4387feee3 is the right fix, and it is small: +138/−101. The bytes were already validated and copied once under the queue lock, so splitting that copy across two caller windows removes the intermediate FrameRx without adding any state. The ABI gives the two windows no meaning. The SDK uses them for the type word and the payload. FrameRx, Wire::bytes_mut and osbench's RefCell<Box<FrameRx>> go away.

The netstack rebase resolution is coherent: Session::open takes main's iface and sent together with this PR's bells. The one architectural problem is the ownership of the link's doorbells, below.

Verified

• Bounds (kernel/src/syscall/dispatch.rs:678-692).
  • head_len is clamped to MAX_MESSAGE_LEN, and len to MAX_MESSAGE_LEN - head_len. That cannot underflow, and the sum is at most 16 KiB.
  • Both windows go through View::window (in_user_half with checked_add, then allows) at their clamped lengths before anything is taken.
  • pop(head.len() + body.len(), …) refuses a longer message whole (kernel/pure/message.rs:101).
  • first = min(len, head.len()) and len - first <= body.len() both follow from that check, so pieces' bound assert is the kernel's own arithmetic and cannot be reached from user input.
  • A message shorter than the head writes only first bytes and nothing to the body. A 1–3-byte message is Refused::Malformed in the SDK, and its batch is dropped closed.
• Partial work. FrameBytes::read cannot fail: a page the process unmapped drops its piece and consume still advances, so the two reads in sequence leave the queue consistent. Handles are installed before the copy, under the same hold, as in round 1.
• Peer death. A message is admitted whole under the lock, so a sender that dies after its send leaves complete messages. Gone comes only with an empty queue and no sender. Nothing in the split changes that.
• Aliasing. message_recv takes head and bytes as two &mut slices, so safe callers cannot overlap them. An overlap through a raw syscall only overwrites the caller's own memory.
• Round-1 blockers.
  1. Refused::blocking maps InvalidArgument to Malformed (sdk/toyos/src/ipc.rs:478-489). fs::transport therefore never reaches Error::kernel's panic for a peer's oversize reply, and ResourceExhausted becomes OutOfResources. The host test is present.
  2. The call is out (settled).
  3. Both stale comments are deleted.
  4. The pipe-lock record names FrameBytes::{read,write} and the nesting in the process lock (formatting nit below).
  5. Settled by the reported runs.
• Evidence.
  • git show c715ebfb5 and git show 4387feee3 are identical apart from the index and hunk lines, and the same holds for edc8770e4 against e6e2fe789. The c715ebf runs therefore carry over: host CI 107/107, 18/18 QEMU registrations, and the debug-boot members, including the new split assertion in messages_are_whole.
  • At 93adf1c83: ci.exit, build.exit and remote-qemu.exit are 0, ci.rev matches, the QEMU log says the revision equals HEAD, and 6/6 netstack registrations pass.
• e6e2fe789. hangupholder prints holding only after notice.listen(), and it ignores write failures, so dropping the read end after the first line cannot end it. If the holder dies before speaking, the job gets EOF and fails at a clear assertion. The sessioncase row's starts is updated with the job.
• Fork. The pinned rust fork (8004d62da) uses none of the removed items.

Findings

BLOCKER 1. system/netstack/src/link.rs:65-67 and :252-253, lib/link/src/driver.rs:149-153, lib/link/src/wire.rs:28-32: netstack uses, typed, a handle a NIC driver sent it, so one driver can end the whole stack.

• What it does. The hello's second handle is wrapped Bell::from(rung) with no check. Every receive pass calls rung.try_take(), which is SYS_EVENT_TAKE → get::<EventWait>.
• Failure scenario.
  • A driver sends anything other than an event's wait end in that slot. Even a plain bug qualifies, such as swapping ring and stack_hears in driver.rs:149.
  • netstack's next pass takes HandleError::WrongType, which ends it with a handle fault (issues/a-received-handle-has-no-knowable-type.md). Any other refusal is Bell::take's panic.
  • That ends every TCP stream and every link, including the healthy second link that main's netstack: every NIC driver is a process of its own, over a link (P4) #1106 failover (traffic_leaves_by_the_second_link_while_the_first_is_down) exists to keep.
  • On main the hello's only handle is the region, which Region::lent checks by fstat. So this PR creates the exposure.
• No knowable-type check exists for events. fstat settles only shared memory and connections, so netstack cannot check this handle itself.
• Simpler ownership removes the problem.
  • In both existing precedents the serving side makes every bell and hands the client its ends:
    • blockserve: lib/blockserve/src/sessions.rs:198;
    • netstack's own stream bells: system/netstack/src/serve.rs:196-198, the design's "netstack makes it, hands the holder the signal side".
  • Do the same for the link: Session::open makes both pairs and sends the driver its waker and its bell with MSG_OPENED. The hello then carries only the region, and HELLO_HANDLES goes.
  • netstack then holds only handles it made. Only the driver trusts received bells, as diskserver does from the block server.
  • The driver's departure is still its waker's last handle, so Ended::HungUp keeps its meaning.
• PR body. "It now rings an event each way, as the block service does" is false in exactly this respect. It needs correcting either way.
• If declined. It must at least appear under "Where it is reachable today" in issues/a-received-handle-has-no-knowable-type.md, which this PR already edits.

Suggestions

• kernel/src/syscall/dispatch.rs:675-677: "Room past any message's or batch's holds nothing" does not parse. Suggested wording: "No message is longer than MAX_MESSAGE_LEN or carries more than MAX_TRANSFER_HANDLES, so only that much of the two byte windows (head first) and of handles is taken."
• issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md:29 and :54: the edited lines are not rewrapped (about 110 columns). "Each 16 pages of it…" now hangs off the connection paragraph, though it describes both copies.
• sdk/toyos/src/ipc.rs:236-237: a double blank line before recv_into.
• sdk/toyos/src/net.rs:389 (response_bytes): the semantics changed from truncating to Io when the answer is larger than buf, and an error answer now has to fit in buf.
  • Its one caller, dns_lookup's 256-byte buffer, is safe because netstack bounds the answer to 256 (MAX_ANSWERED).
  • The doc should say that buf must hold an ErrorResponse too, or the function could take its buffer at a size the protocol fixes.

SEND BACK

[Japabu]

Review r2, answered at a16f09c3a.

Blocker 1, fixed as suggested. netstack makes both of a link's bells. Session::open makes both pairs and sends the driver its waker and its bell with MSG_OPENED (OPENED_HANDLES = 2). The hello carries only the region, which Region::lent checks, and HELLO_HANDLES is gone. netstack now holds no event a driver sent. Only the driver trusts the bells it receives, as diskserver does the block server's. A hang-up still means the same thing: the driver's departure is still the drop of its waker's last handle. The toyos-link and netstack module docs and the PR body ("as the block service does") are corrected.

Suggestions, all taken:

• the receive bounds comment uses your wording;
• the pipe-lock record is rewrapped, with the page-population sentence moved ahead of both copies;
• the double blank line is gone;
• response_bytes documents that buf must also hold an ErrorResponse.

Separately, the AMD memcpy issue record is dropped, because #1133 fixes its cause.

Checks:

• --ci host: exit 0, 107/107.
• --build-only: exit 0.
• Both ran at ccaf1b1cd, which differs from the head only by the dropped issue file.
• Remote QEMU x86-64 at a16f09c3a, six netstack registrations including netstack: every NIC driver is a process of its own, over a link (P4) #1106's failover: 6/6, exit 0.

Evidence: https://github.com/ToyOSOrg/evidence/tree/86f317df1c9f6b87b9d534272e3b1ec8737df6ee/ipc-messages/review-r2-a16f09c3a

[Japabu]

wake.onto_busy placement A/B, T14, four boots, all passed. Head is this PR at 93adf1c83 plus placement notes; base is 3add613c1 plus the same notes. The rule was set before the run.

• Refusals: head 0 of 2, base 0 of 2. By the rule, the refusal is not shown to be this PR's.
• A head-only signature outside the rule: on both head boots, osbench's bench thread migrated to CPU 0, the IRQ CPU, during staging. That happened from partner 2 on one boot and from partner 32 on the other. On base it happened on 0 of 2 boots.
  • On the first head boot, every round was skipped as waker-on-IRQ, so the row went unmeasured.
  • The other head boot measured p50 3870 ns, against base 3838 and 3898 ns.
• My inference, not verified: the earlier refusals are this same migration. The scheduler moves an unpinned bench thread onto an idle CPU 0, and this PR changes only how busy that CPU's other work keeps it. The notes go to the scheduler track.

Evidence (bench and note lines only): https://github.com/ToyOSOrg/evidence/tree/9261cc8e8d211555f26bd786eb2de91f96eb146a/ipc-messages/placediag-t14

[Japabu]

Landing check at b45676390 (after the #1123 rebase), QEMU x86-64 on the remote runner, all exit 0:

• paged objects: a file's pages are the kernel's, read by its holders without a request (memory stage 7a) #1123's members in one --debug testcases boot: a_served_file_reads_through_its_paged_object, served_file_offset, a_pager_s_end_answers_its_waiting_holders_gone, std_fs, file_mtime. All five passed, with messages_are_whole alongside.
• The six netstack registrations: 6/6 PASS.

Evidence: https://github.com/ToyOSOrg/evidence/tree/6c6239cdacdda5bc8f0f8fbdf0b5589045198b4b/ipc-messages/land-check-b45676390