Repository navigation
socket export: the race's peer connects once a racer answered its last connection; a failed job reports the peer's verdict - #1083
Conversation
Review r1:
|
…ecord goes back as base had it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r2:
|
…t connection, and a failed job reports the peer's verdict netstack_holders_exported_to_children's race stalled now and then on a macOS host, and the record blamed a lost wake between the two holders. No wake is lost. The harness's peer made its two race connects back to back. QEMU's user network listens on a forwarded port with a queue of one (QEMU 11.1.1, `netstat -Lan` reads maxqlen 1). A macOS host resets a connect that finds that queue full, so when slirp has not yet accepted the first connection, the second connect fails with ECONNRESET (os error 54). That happens when QEMU's loop is busy: under TCG, every observed stall fell at netstack's mDNS claim. The guest's racer then waits, correctly, for a connection that never comes. The peer's thread ended with the error, but the job's failure dropped the peer's result, so the ceiling reported a silent guest. Evidence, in the PR's evidence: a packet dump of a stalled round carries one SYN for the round's two connects; the peer's own error is "a racer's connect: Connection reset by peer (os error 54)"; a suspended QEMU's forwarded port holds one connection unaccepted (1/0/1) and resets the second connect; a backlog-1 listener on macOS does the same. The peer now connects one at a time, the second once a racer has written its line on the first. That is an event the guest answers, not a delay. The datagrams still go two at once. A job that fails reports what the peer's thread had come to, so a step the peer could not take is named as the peer's, not the guest's. Closes issues/two-holders-racing-on-an-exported-socket-can-both-stall.md; the queue-of-one fact moves to SOCKET_EXPORT_PORT's comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers over one queue are modelled on the host on both sides The holder half of the several-holder wake contract had no host check: consuming the signal immediately before the request and never after it. That half is `look` together with the drain calls in `stream::accept` and `recv_from`, and only the guest race exercised it. - `look` hands each attempt a single-use `Drain`. Spending it reads the bell and yields the `Drained` that alone sends an ACCEPT or a RECEIVE. A request without a read, and a second read after the request, no longer compile. The drain stays inside the request closures, after every step that can fail, so a call refused before its request still leaves the ring in the bell. - `look` is generic over the end's bell (`EndBell` in production), so a host test runs it for two holders over one queue against a modelled server that keeps netstack's per-holder rule. The second arrival lands after the server answers holder 1 "nothing" and before holder 1 reads that answer. Negative control: `look` reading the bell after an ask answered nothing fails the test with "holder 1 waits for good with [2] queued". - The node's datagram test gains a third arrival after a holder's empty receive: both holders are rung, the one that found nothing included. Negative control: re-arming a holder only on a receive that takes a datagram fails it, (2, 2) against (2, 3). - The harness's inner comment that repeated its doc comment is gone, and the queue of one is said to be the host port's. The commit before this one says the mDNS claim is when QEMU's loop is busiest. That is an inference from the correlation, not a measurement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d what its type guarantees is said exactly The previous commit said a request without a read, and a second read after it, no longer compile. That is stronger than the types. ACCEPT and RECEIVE are public, because netstack's server parses them, so a raw `ipc::signal` of either compiles. Inside socket.rs, a hand-built `Drained` or another read of the bell compiles too. What the types now guarantee: - `look` hands each attempt a `Drain` that is spent once. A second `drain.now()` in one attempt is E0382, recorded in the PR's evidence. - Spending it gives the `Drained` that `Drained::accept` and `Drained::receive` send their request with. Those name the message themselves, where `signal`/`signal_with` took any message type. - A `Drained` cannot be made outside socket.rs, and stream.rs reaches the listener's bell only through its `Drain`. - The bell is read in one place, `EndBell::drain`; the free `drain` is gone. A raw `ipc` send of ACCEPT or RECEIVE bypasses the token. The doc comment says so; review catches it, and the recorded attempt compiles with an unused-variable warning. `Drain` is `#[must_use]`. Rebased onto 717baf3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r3:
|
…ent leave to read it The `#[must_use]` on `Drain` never fired. A `Drain` exists only as `ask`'s argument, and an ignored parameter draws `unused_variables` instead. The value that matters is the `Drained`. `drain.now();` on its own reads the bell and sends nothing. That loses the ring, because the server still counts the holder as told and does not ring it again for the waiting item. With the attribute on `Drained`, that statement warns "the bell was read for a request: send it" (recorded in the PR's evidence). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ecord goes back as base had it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
netstack_holders_exported_to_children's race (tests/guest/rust-tests/src/bin/socket_export.rs,race) stalled now and then on a macOS host: 9 stalls in 64 soak batches across three revisions (orchestrator's soak, evidencepr-1074r4).issues/two-holders-racing-on-an-exported-socket-can-both-stall.mdsuspected a lost wake between two holders of one socket queue.What it actually is
No wake is lost in ToyOS. The harness's peer made the race's two connects back to back. QEMU's user network listens on the forwarded port with a queue of one. A macOS host resets a connect that finds that queue full. When QEMU's loop has not accepted the first connection yet, as happens under TCG at netstack's mDNS claim, the second
connect()fails withECONNRESET. The peer's thread ended there. The guest's second racer waited, correctly, for a connection that never came. The job's failure dropped the peer's result, so the harness reported a silent guest after its 700+ s ceiling.Observed (evidence below):
a racer's connect: Connection reset by peer (os error 54).netstat -Lanreads QEMU 11.1.1's forwarded listener asmaxqlen 1. With QEMU suspended, the first connect sits unaccepted (1/0/1) and the second is reset with error 54, every time.Inference, not measured here: a Linux host drops a SYN to a full queue rather than resetting it (
tcp_abort_on_overflowoff by default), so its client retransmits. That would explain why the stall was seen on macOS only.Change
racingbefore it calls accept, so the peer cannot know; likely under TCG, as before this change), and the one that does not take it must wait on its bell again and be woken by the second. The datagram half still has two datagrams waiting together for two holders.peer_verdict), so a step the peer could not take is named as the peer's.SOCKET_EXPORT_PORT's comment.lookhands each attempt aDrainspent once; spending it gives theDrainedthatDrained::acceptandDrained::receivesend their request with. A rawipcsend ofACCEPTorRECEIVE, which netstack's server needs public, bypasses the token; review catches it.lookis generic over the end's bell (EndBellin production), so a host test runs two holders over one queue against a modelled server. The node's datagram test gains a third arrival. No change in what the SDK sends or when.The cause needed no guest, netstack or kernel change; the SDK change is the review's coverage of the contract below. The test's environment and targets are unchanged: QEMU x86-64
netcase.toml, user network with one forwarded port.Checks
Evidence: https://github.com/ToyOSOrg/evidence/tree/a5c49b359811d9d1a19a76906a41195ebdfcdd32/pr-1083/r1 (README lists each file, command and exit code). Host: aarch64 Mac, QEMU 11.1.1, x86-64 guest under TCG, test alone, three boots in parallel.
cargo run -- --ci hostat this tree: exit 0, 104 steps green.bde3e08ba: 40/40 exit 0. Run without--nocapture, so these logs do not show whether the race met the claim.--nocapture, same session and setup:bde3e08ba+ loop: 12/12 boots pass, 480 races; netstack's mDNS claim fell inside the rounds in 10 boots.ECONNRESETat connect. Two end onEINVALfromset_read_timeoutstraight after a connect; that this is the same reset is inferred, not measured. Two earlier base loop boots stalled at rounds 7 and 4, and the plain base test stalled on its first run.maxqlen 1.Round 3 (reviews r1 and r2), evidence https://github.com/ToyOSOrg/evidence/tree/b7dfd4045520217c8ea54f21cd17d25d62f90c9f/pr-1083/r3, at head
b6a717df8on717baf3ac; every log begins with thatHEADand an emptygit status --porcelain:cargo run -- --ci host: exit 0, 104 steps green.lookreading the bell after an ask answered nothing fails the SDK model test ("holder 1 waits for good with [2] queued"); re-arming a datagram holder only on a take fails the node test,(2, 2)against(2, 3).drain.now()in one attempt is E0382; a rawipc::signalofRECEIVEcompiles.netstack_holders_exported_to_childrenonce with the SDK rebuilt into the guest: exit 0. The 40-round loop was not rerun: since r1 the harness changed only in comments, and the SDK sends the same requests in the same order.91b62446bmoves#[must_use]fromDraintoDrained, so a baredrain.now();, which reads the bell and sends nothing, warns (recorded).(cd sdk/toyos && cargo check && cargo test)at that head: no warnings, all tests pass. No guest rerun: an attribute changes no code. Evidence: https://github.com/ToyOSOrg/evidence/tree/60a99591018817e70c42bcd5ee493d6be07ddeb4/pr-1083/r3.socket.rsthat differs from9a26dea93by doc comments only (r2-control-trees.diff). Round 2's host and guest runs recorded no tree; the runs above at a recorded head supersede them.No metal run: the behaviour belongs to the host and QEMU's user network, not to hardware.
Risks
every_holder_takes_from_one_queue_and_the_loser_finds_nothing).issues/two-holders-of-one-socket-queue-can-wait-for-good.md, a record on that branch that this mechanism explains; it should not land.The wake contract for several holders of one queue
A coalesced event bit is enough only with 1 to 3. One event shared by all holders, or a server that signals only when items arrive, can lose a waiter: with three items arriving together and two holders, the third is never signalled.
What checks each part, after this PR:
system/netstack/node/tests/listeners.rsevery_holder_takes_from_one_queue_and_the_loser_finds_nothing(theP3step: both holders rung after looks)system/netstack/node/tests/datagram.rsevery_holder_receives_from_one_queue_each_datagram_once_and_whole, now with a third arrival after a holder's empty receive:(2, 3)(2, 2)Drainis spent once; aDrainedcannot be made outsidesocket.rs;stream::acceptandrecv_fromsend throughDrained::accept/Drained::receive. Review covers the rest: a rawipcsend ofACCEPT/RECEIVE, or another read of the bell insidesocket.rs, compilesdrain.now()in one attempt: E0382 (recorded); the raw-ipcbypass compiles, with warnings (recorded)look's loop, never reading after the asksdk/toyos/src/net/socket.rsthe_holder_that_found_nothing_takes_the_next_arrival: two holders, one queue, a modelled server keeping rules 1 to 3; the second arrival lands between the server's "nothing" and the holder reading itlookreading the bell after an ask answered nothing fails it: "holder 1 waits for good with [2] queued"stream::accept)netstack_holders_exported_to_children)The host tests are independent of the bell's mechanism: the node's
Belltrait and the SDK'slookover aBelltrait. They carry the contract through the IPC migration; the guest race remains the end-to-end check.🤖 Generated with Claude Code