Skip to content

Wakeups are kernel events, and a one-handle timed wait is SYS_HANDLE_WAIT - #1108

Merged
Japabu merged 6 commits into
mainfrom
ipc-events
Oct 31, 2026
Merged

Japabu merged 6 commits into
mainfrom
ipc-events

Conversation

@Japabu

@Japabu Japabu commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A wakeup carries no data, yet every one in ToyOS was a pipe. That meant two kernel objects and a pipe's four watches. Once rung it also took a 4 KiB frame (since #1074), a byte per wake and a 64-byte drain loop to take it. EOF stood for "the waker is gone" and a refused write for "the bell is gone". Block sessions and TCP streams also wrote raw doorbell bytes onto their connections (Connection::{read,write}_nonblock, netstack's RING frame). Separately, about a dozen one-handle timed waits each built a fresh poller: an inbox page made, mapped and given back for one wait. mio's ToyOS waker wrote to a pipe its selector never drained, so a woken waker stayed readable for good (issues/mio-s-toyos-waker-never-drains-its-pipe.md).

What changes

  1. Kernel event object (kernel::event, pure and host-tested; kernel/src/object/event.rs, kernel/src/syscall/event.rs).
    • EventSignal and EventWait share one Event: a pending bit, each side's end and one watch.
    • A signal is one atomic or. It posts only on the idle-to-pending edge, never blocks or allocates, and answers Gone once no wait side is left.
    • A take is one atomic and.
    • A wait side whose signalers are all gone is readable for good, and its take answers Gone, never a phantom signal.
    • Readiness observes and takes nothing.
    • The two authorities are separate types.
    • A real-time signaler lends its window through the same scheduler::post_lending path pipe wakes use, bounded by the same quantum.
  2. SYS_HANDLE_WAIT: park on one handle's one condition with a timeout. It is answered from the same readiness an inbox watch reads, including the notice-listener rule, so a one-handle wait needs no inbox. A claim is refused NotSupported.
  3. ABI: ObjectKind::{EventSignal, EventWait}, SYS_EVENT_CREATE, SYS_EVENT_SIGNAL, SYS_EVENT_TAKE, SYS_HANDLE_WAIT.
    • SDK: toyos::wake::{pair, Waker, Bell} keeps its names over events, with Hungup for the other side's end. It also adds poller::wait_one.
  4. Every pure-wake pipe is an event. This covers:
    • window's Waiter, compositor session opens, soundserver's command, device-loop and per-client bells;
    • the supervisor's app-end watchers and file worker (a counter beside the event, since an event coalesces);
    • each driver's DRIVER_STOP, each program's log ALIVE, and libc's wake and control signals;
    • netstack's holder bells and each stream's receive and send bells.
  5. Block sessions: the doorbell bytes are two events the driver makes and sends with MSG_OPENED. The connection is dropped once the session is open.
  6. TCP streams: RING is deleted. netstack makes a per-holder "rung" event, gives the holder its signal side, and watches the wait side instead of the holder's connection once the connect is answered. The holder's departure is that signal side's last handle going. Until the answer, netstack itself holds the signal side it will hand over, so it watches the first holder's connection instead, whose hang-up is the departure (review r1). EXPORT stays a message, followed by a ring. Each place still has one watched handle, so netstack's place bound is unchanged.
  7. One-shot pollers become SYS_HANDLE_WAIT: libc's interruptible sleep, diskserver's block-answer wait, inspect, the supervisor's logkeeper flush, hda's acceptance and the terminal's shell. This is a separate commit (080528966) so it can be reverted if the A/B says so (approved condition: it must win or tie against a cached poller).
  8. mio: the waker is an event, and the selector takes it on the waker's token. The pin is ToyOSOrg/mio fe42def on the fork's toyos branch. This closes the mio issue.

Deleted: the bell pipes and their drain loops, netstack's pipes.rs, stream::SERVER_WATCH and RING, and a poller per socket end. A netstack holder's memory bound drops from four pipes to two.

Consequential decisions

  • The server makes every event it will act on. A handle of the wrong type ends the caller, so a server must never use an event handle a client chose. The netstack rung event and the block doorbells are therefore made server- or driver-side and handed out.
  • Approved before implementation: the syscall rather than a cached per-thread poller; signalers-gone reads ready while its take answers Gone; mio fork commit and pin in one landing.
  • Rebase onto kernel: a pipe keeps its bytes in 4 KiB frames it takes as it fills #1074: kernel: a pipe keeps its bytes in 4 KiB frames it takes as it fills #1074 primed each wake pipe with one byte so that a later wake could not be refused for memory. An event holds no bytes, so the priming and its ResourceExhausted arm are gone with the pipe.

Checks

Head 18060b521, on main f45dc5e3b (#1106). The first four commits have the tree of 6250464e8, rebased, which is where the checks below were run unless noted; the fifth and sixth are review rounds 1 and 2. A connection's raw byte calls, read_nonblock and write_nonblock, stay until the whole-message receive (phase 2). So does lib/link's doorbell byte, which phase 2 moves to events.

Rebase onto #1106 (18060b521) (evidence): cargo run -- --ci host exit 0 (107/107). QEMU: netstack_holders_exported_to_children (virtio-net and 82574 arms) exit 0. QEMU: a_link_ends_under_a_held_stream_and_comes_back, stream_holders, a_client_killed_mid_connect_gives_its_place_back, udp_receive_times_out and kernel_objects_refused_alone green on virtio-net.

Review round 1 (044a7ae23) (evidence):

  • cargo run -- --ci host: exit 0, 105 of 105.

  • QEMU (x86-64): stream_holders, netstack_holders_exported_to_children and kernel_objects_refused_alone pass. The new a_client_killed_mid_connect_gives_its_place_back passes (exit 0).

  • Its negative control is the rung watched while the connect is unanswered, with netstack rebuilt; the test fails (exit 1). A first version with one child passed that control, because the connect ends on its own within seconds. The test now kills one of two children and requires the place back while the survivor's connect is still pending, which only the kill can explain.

  • The event charge and the comment and record fixes need no boot: refusal under exhaustion is kernel_objects_refused_alone's, rerun above.

  • cargo run -- --ci host: exit 0, 105 of 105 steps green.

  • cargo run -- --build-only: exit 0.

  • QEMU, cargo test --test toyos-build -- <9 names>: exit 0, 9 of 9 passed. The registrations are stream_holders, libc_children, netstack_holders_exported_to_children, udp_receive_times_out, the_data_partition_beside_the_stick_is_designated, a_host_disk_beside_the_stick_is_left_alone, session_lifecycle, hda_plays_to_the_controller_that_reaches_an_output and kernel_objects_refused_alone. These were rerun after the rebase because kernel: a pipe keeps its bytes in 4 KiB frames it takes as it fills #1074 changed the pipe that connections still use.

  • Negative controls, each a patch applied and restored in one run, on the pre-rebase tree (f3fe154). The rebase changed none of kernel::event or kernel/src/object/event.rs.

    • No post when an event becomes pending: the boot never reaches ===READY===.
    • Readiness that takes the signal: the boot never reaches ===READY===.
    • An end read as a signal: two host tests go red.
  • Metal, images built at 6250464e8 and a5b330f39 and run by the orchestrator, judged with the harness at a tree identical to 6250464e8 (evidence):

    • Members and audio, T14 and AMD: event_sides, mio_waker_coalesces, cpal_drop_unreleased, hda_tone and hda_tone_resampled all pass on both (exit 0, 5 of 5 each).
    • mio control, T14: the head with mio at its old pin f762092 fails mio_waker_coalesces (exit 1) on the defect itself: "a poll after the wake was reported reported [Token(7)]".
    • osbench A/B, two base boots and two head boots per bench, every boot passing the osbench judge. Means of p50, head against base:
      • T14: ipc.reqrep.8 -1.8%, .4096 -2.1%, .65536 -0.3%, ipc.reqrep_handle.8 -1.9%; pingpong.pipe.* and pingpong.futex.* within 3%.
      • AMD: head1 matches base on every IPC row (ipc.reqrep.8 6613 against 6593 ns). head2 is slower on all four reqrep rows (ipc.reqrep.8 8546 ns, minimum 8256 against about 6500) and is also an outlier on wake.periodic.idle and time.sleep_late.1ms, at the same clock as the others. This change does not touch the connection path reqrep measures, and head1 runs the same code at base speed, so I read head2 as one boot's machine state, not a regression. That is an inference from two boots: the cause is not established (cross-CCX placement would fit, unverified).
      • Other rows past 5% are the T14's persist.*, which is bimodal across base boots too (1.0 against 7.0 ms), and single-boot outliers.
    • SYS_HANDLE_WAIT against a cached poller (measurement-only waitone rows, head boots, p50 ns):
      • T14: ready 189/183 against 752/776; pingpong 5308/4704 against 5912/5859.
      • AMD: ready 140/140 against 581/581; pingpong 6472/6272 against 5400/7004.
      • A fresh poller, what the sites did, costs 9700 (T14) and 7100 (AMD) ready.
      • Decision: commit 080528966 is kept. handle_wait wins on every ready row (4x) and on the T14's pingpong, and ties on the AMD's, whose cached-poller pingpong itself spans 5400 to 7004 across the two boots.

Evidence: ToyOSOrg/evidence pr-1108 at e5371858a: logs with exit codes, control patches, the measurement-only waitone source and the metal request.

Why the guest tests need their environment

  • a_client_killed_mid_connect_gives_its_place_back needs a process's death and the real netstack's watch of it (tests/images/netcase.toml), which no host test has. Its peer is an address on QEMU's user network that nothing answers, so no public network. QEMU x86-64, order and counts, with QEMU's clock only a hang ceiling.

  • event_sides covers a parked wait woken by another process's ring and by its end, a watch that takes nothing, and an end that is never a ring. Host tests model the pure state, not cross-process parking. It runs on metal in the testcases shared boot (T14, AMD).

  • mio_waker_coalesces checks the fork itself: no image program builds mio for ToyOS. The check is 10,000 wakes before a poll coming back as one event, then nothing, then a later wake reported.

Risks and open items

  • AMD head2's slower reqrep is attributed to the boot, not the change, on two boots per side; another AMD pair would settle it if review wants it settled.
  • issues/a-tcp-connect-zeroes-twelve-mebibytes.md on main still cites the issue kernel: a pipe keeps its bytes in 4 KiB frames it takes as it fills #1074 deleted (a-pipe-is-a-two-mebibyte-page-...). That is outside this change and left as is.

🤖 Generated with Claude Code

@Japabu

Japabu commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #1108 at f961df917 (tree identical to 6250464e8), against origin/main

Architecture

The design matches what was approved, and it is the right shape.

  • Why a kernel object. An event is a kernel object because userland cannot build one. A futex word is not inbox-watchable, and a futex cannot report a peer's end through handle lifecycle.
  • Authority. The signal and wait authorities are separate types. Rights are WRITE|DUP|TRANSFER against READ|WAIT|DUP|TRANSFER, and the SDK narrows a holder's rung to DUP|WRITE.
  • Pending bit and wakes.
    • One AtomicU8 holds the state, with a post only on the idle-to-pending edge. The lost-wake argument is the existing Watch's: wait_until registers, then re-reads ready().
    • A signal-side end sets SIGNALERS_GONE before it posts.
    • A take after the end answers Gone only when nothing is pending. The host tests and event_sides both hold this.
  • Observe versus take. Readiness is a plain Acquire load, so a poll observes and does not consume.
  • RT loan. The loan goes through post_lending, the same path and quantum pipes use. lend_rt can be granted to the take after the one it was marked for, which is the same race the pipe's rt_boost_pending already has. It stays bounded at one window per RT raise.
  • SYS_HANDLE_WAIT.
    • It clones the object before parking, as kernel/CLAUDE.md requires.
    • It reuses read_watch/has_data and watched_readable, and refuses claims.
    • Its deadline arithmetic saturates (Instant + Duration).
    • SYSCALL_PROFILE_BINS = 153 is correct for numbers up to 151.
  • Servers make the events. Every event a server acts on is server-made: block doorbells, netstack's rung, soundserver's signals. The ALIVE and stop events are supervisor-made.
  • No bell pipes left. I found no remaining pure-wake pipe in system/, lib/, sdk/ or apps/. The remaining pipe_pair callers are real byte streams.
    • Pinned socket2 (2aa6b00) and the in-tree sdk/std reach bells only through the SDK.
    • Connection::{read,write}_nonblock stay until phase 2, which is within the approved migration.
  • mio fork.
    • fe42def is a direct child of the old pin f762092, and refs/heads/toyos on ToyOSOrg/mio points at it. It is pinned in both Cargo.lock and tests/guest/rust-tests/Cargo.lock, and depends on toyos-abi by version.
    • The change is upstream-quality: a Kind tag in the token, and the take happens before the token is reported, so no wake is lost.
    • The control fails on the defect itself ("a poll after the wake was reported reported [Token(7)]").
  • Size. Production and tests: kernel +377/−21 (about 70 of those lines are host tests, and about 120 are exhaustive match arms in ops.rs), system +320/−318, sdk +324/−220, tests +254/−71, issues +9/−30.

Blockers

  1. system/netstack/src/serve.rs:119, :380, :581: a client that dies during a pending connect is not noticed.

    • Where the waker sits. hold_stream stores the holder's three handles in Stream.answer until settle hands them over. That includes the rung waker. Meanwhile watch() watches holding.rung, and no longer watches the holder's connection. While the connect is pending, netstack itself holds a signal side of the bell it watches, so the bell can never read "every waker gone".
    • On main. The connection's hang-up was watched from hold_stream onwards. It became Heard::Leaving(None) → departed → StreamEvent::Closed. The settle arm at :522 still says "Its client left first".
    • Now. The death is seen only when the connect resolves:
      • on success, hand_over's try_send fails and the dropped waker reads as the departure;
      • otherwise, with timeout_ms == 0, after limits::SYN_GIVE_UP = 180 s (system/netstack/shard/tcp/src/lib.rs:141).
    • Consequence. A killed connecting client keeps its place, region mapping and handles for up to three minutes. The PR's claim that "the holder's departure is that signal side's last handle going" is false for this window.
    • Fix. Watch the first holder's connection for its hang-up while answer is pending, and the rung after it is handed over. That keeps one watched handle per place. Alternatively, keep no signal side in netstack.
    • Evidence. No test covers a death mid-connect. Either an inspection argument in the reply or a guest test against the real netstack is needed: connect to a peer that never answers, kill the client, and read netstack's held-place count under a ceiling.
  2. kernel/src/object/event.rs:38–44: the event's shared memory outlives its charge.

    • Arc<Event> and its Arc<Watch> are charged to the signal side's ObjectCore. That core and its charge drop when the last Arc<EventSignal> goes, but the memory lives on as long as any wait side does.
    • That is the normal "every signaler gone" state: a dead client's ring held by the driver until it closes, a sounded stop bell, ALIVE bells that logkeeper holds until it sweeps them, netstack's rung after its holder leaves.
    • So a job can keep uncharged event memory indefinitely, about the size of an Event and a Watch per orphaned wait side.
    • Pipe keeps its Charge in the shared object (kernel/src/pipe.rs:201–238). Do the same in Event, and delete the comment "the signal side is charged what they share".
  3. Commit history says something false. Main keeps branch commits (see 153e85d07 "review r1" on main).

    • 8b3200605's message says "Deleted: Connection::{read_nonblock, write_nonblock}". Both are still in sdk/toyos/src/ipc.rs:329–336.
    • b777405ce and its revert f961df917 would land as a no-op pair.
    • Reword 8b3200605 and drop the pair. The implementer owns this remote branch.
  4. Stale or unreadable comments.

    • system/supervisor/src/sessions.rs:193–195: "An end reaches the loop through one pipe". It is an event now.
    • tests/guest/libc/spawn_children_cap.c:1–4: "beside its control pipe", "the wake pipe's". Both are events now.
    • system/netstack/src/lent.rs:139: "A bell whose holder is gone has a holder its own bell reports leaving." This does not parse. Say what happens: a ring into a bell nobody holds is dropped, and the holder's leaving is its rung bell's end.

Suggestions

  • AMD head2 (needs no new pair). Inspection shows reqrep's path is unchanged:

    • sdk/toyos/src/ipc.rs and kernel/src/pipe.rs are untouched;
    • wake_pipe_readers makes the same calls through post_lending;
    • head1 matches base on every IPC row.

    The same boot is also out of line where this change cannot reach. time.sleep_late.1ms reads 44.9 µs against 5.5 µs in the other three AMD boots. That row is thread::sleep, which is SYS_NANOSLEEP, untouched here. wake.periodic.idle is also out of line. That points to the boot's machine state, not the change. If the AMD is booted for the fixes anyway, add one head benchcase boot to that batch. Otherwise it is not worth a reboot.

  • T14 fs.disk.*. These are the rows the block-doorbell change touches. "Single-boot outliers" does not describe create_write_unlink, where both heads (161k, 160k) match base2 (158k) and base1 is the low one. Name these rows and their bimodal base explicitly in the evidence.

  • issues/closing-a-pipe-end-ends-every-holders-polls-on-it.md. Events now share this defect: close_ends_polls(EventWait) is true, and a listener's descriptor is a dup of an event wait side. The record should say so.

  • Open records that still describe the soundserver signal or command pipe. a-client-cannot-tell-soundserver-it-paused.md:21, null-sink-applies-one-connect.md:22,30,57, cpal-backend-hardcodes-the-format.md:18 and hda-client-stall-prints-a-stream-error-and-passes.md:11.

  • Reflow.

    • lib/quiesce/src/lib.rs:87–89 and sdk/toyos/src/net/stream.rs:14 have broken line wraps.
    • mod bells; is out of order in system/netstack/src/main.rs:51.

Evidence checked

  • Host and build. ci-host-3.log: 105/105, exit 0. build5.log: exit 0. The aarch64 kernel check passes. AArch64 has no guest coverage, but the suite is x86-64 only (qemu::SUITE_ARCH), so no target is missing.
  • QEMU. qemu-3.log: 9/9, exit 0, at 6250464e8, x86-64.
  • Negative controls. c1, the dropped post, and c2, readiness that takes, each stop the boot before ===READY===. c3, an end read as a signal, turns two host tests red. These are coarse, but they are real controls of the mechanism.
  • Metal members. 5/5 on the T14 and on the AMD, at the stated tree.
  • mio control. On the T14 it fails with exit 1, on the defect itself.
  • Merge. The branch merges cleanly with current origin/main (28fdfd7e7), and the osbench changes there touch no API this changes.

SEND BACK

@Japabu

Japabu commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review round 1, head 044a7ae23.

  1. A client dying mid-connect. netstack now watches the first holder's connection until the answer is handed over, and the rung after it, so there is still one watched handle per place (Sockets::watch). There is a new QEMU test on the real netstack, a_client_killed_mid_connect_gives_its_place_back. Two children connect to an address nothing answers, and one is killed. Its place must come back while the survivor's connect is still pending. With the old watch (rung only), the test fails. A one-child version passed that control, because the connect ends on its own within seconds, which is why the survivor is there.
  2. The event charge is the Event's now, as the pipe's is the Pipe's, and the comment is gone.
  3. History. 8b3200605 is reworded (now 2537f0c5b), the b777405ce/f961df917 pair is dropped, and the branch is force-pushed with lease. The first four commits have the measured tree.
  4. The comments are fixed as listed, the lent.rs sentence now says what happens, and stream.rs states the pre-answer rule too.

Suggestions taken:

  • The closing-a-pipe-end record now covers an event's wait side.
  • The four soundserver records now say event where it is one now, keeping their history.
  • The two broken wraps are reflowed, and mod bells is back in order.
  • The T14 fs.disk.* rows are now named against their bimodal base in the metal evidence (1a7b8481c).

No boot is needed for these fixes. Evidence: https://github.com/ToyOSOrg/evidence/tree/0af72a5d05789abaa11d8d52e1e5551d27054b7c/ipc-events/review-r1-044a7ae23

@Japabu

Japabu commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 2 of #1108 at 044a7ae23

This round checks the four round-1 blockers and the new commit. Everything else is settled.

Architecture

The fixes stay inside the approved design.

  • netstack's watch. netstack still watches one handle per place. It watches the first holder's connection while Stream.answer is pending, and the rung after the answer is handed over. That restores the hang-up detection main had, with no second mechanism.
  • The event's charge. It now lives in the shared Event, the way the pipe's lives in Pipe.
  • Size. The fix commit is +207/−45:
    • +101 for the new guest test;
    • +35 for its harness registration;
    • +37/−22 in issue records;
    • the rest is code and comments.

Round-1 blockers

  1. A client killed mid-connect: fixed.
    • Watch. Sockets::watch (system/netstack/src/serve.rs:584–593) picks holding.connection while stream.answer.is_some(). Netstack rebuilds its watch set every pass (main.rs:433 through Armed), so the watch moves to the rung once settle takes the answer.
    • Hang-up path. A hang-up while the connect is pending runs through holder_answered. There try_take on the rung reads nothing, pump gives Eof, and heard turns that into Leaving(None). That leads to departed and then StreamEvent::Closed. The Closed arm still finds the stream, because answer holds a strong Rc<Holders>, so the stream is not pruned and the unreachable! cannot fire.
    • Two-child test. The test discriminates correctly, and I checked why:
      • Both children park on the same unresolved neighbour (stack.rs:1357) and fail in the same serve.
      • On a refusal, answer_refused drops the bells taken from answer, including netstack's own rung waker. So under the defect both rungs read gone in the same pass, and the count goes from +2 straight to 0. It never passes +1.
      • The control log shows exactly that: "holds Some(U64(0))".
    • Control. control-c4.log removes only the new match arm, rebuilds netstack (its mtime is logged) and goes red with exit 1. qemu-6.log is green with exit 0.
  2. The charge: fixed.
    • kernel/src/object/event.rs:26–28,43–51: Charge::current runs first, so a later new_core failure gives the charge back when it drops.
    • The misleading comment is gone.
  3. History: fixed.
    • The trees of e39e43d3b, 2537f0c5b, 6e1305ad3 and 20e0c546c match the old e39e43d3b, 8b3200605, 080528966 and 6250464e8 (20e0c546c^{tree} = 61021fde1, the measured tree).
    • The no-op pair is gone.
    • 2537f0c5b now says the raw byte calls stay until phase 2, which is true.
  4. Comments: fixed in sessions.rs, spawn_children_cap.c and lent.rs. The stream.rs and serve.rs headers now state the rule before the answer.

Taken suggestions:

  • the issue records are updated;
  • the two wraps are reflowed and mod bells is back in order;
  • the T14 fs.disk.* rows are named against their bimodal base in the evidence (metal-6250464e8/README.md:27–32).

All of these check out.

Blocker

  1. tests/guest/rust-tests/src/bin/netstack_connect_killed.rs:31 and tests/toyos.rs:656 — the ceiling comments are false.
    • What they say. "A hang ceiling under QEMU, far inside the connect's own give-up" and "QEMU's clock a hang ceiling far inside the connect's give-up".
    • What is true. The connect here never reaches SYN_GIVE_UP. It fails when neighbour resolution gives up: nud.rs:414, BROADCAST_SOLICIT = 3 requests at RETRANS = 1 s, so about 3 s. The module header's own "within seconds" agrees. The 20 s ceiling is far outside that.
    • Consequence. The test is only valid if the guest gets through spawning both children, seeing both places held, and the kill within about 3 s of guest time. That bound is the test's real constraint. The comments tell a reader the opposite, and anyone who later loosens the timing or misreads a red trusts the wrong number.
    • Fix. Correct both comments. The ceiling only bounds a hang. The kill has to land within the ~3 s resolution (nud::BROADCAST_SOLICIT × nud::RETRANS), and a run that misses it ends red at "a child ended before it was killed". This is a comment fix and needs no new run.

Suggestions

  • A peer that resolves and stays silent. With one, the connect would end only at SYN_GIVE_UP (180 s). A one-child version would then be enough, and the test would no longer depend on guest progress inside a 3 s netstack timer. That is a QEMU-speed dependence the testing policy avoids. Adopt it only if the user network makes it hermetic. Otherwise the two-child design with honest comments is acceptable.
  • system/netstack/src/lent.rs:139–140. "which netstack reads apart" is unclear. Ending the sentence at "its rung bell's end" loses nothing.
  • issues/null-sink-applies-one-connect.md:30,44–45. Two lines run past the wrap after the edit.

Evidence checked

  • Host. ci-host-6.log: 105/105, exit 0, at 044a7ae23.
  • QEMU. qemu-6.log: the new test passes, exit 0. Its tree differs from 044a7ae23 only by a header comment. qemu-4.log covers the three affected neighbours: stream_holders, netstack_holders_exported_to_children and kernel_objects_refused_alone. The later edits are the test's lease wait and comments, none of them in netstack, so that result still holds.
  • Control. control-c4.log: red as required, with netstack rebuilt.
  • Metal. No metal run is needed. The netstack change only restores the pre-answer hang-up watch, and the charge move does not touch the hot path.

SEND BACK: one comment fix (blocker 1). Everything else is ready.

@Japabu

Japabu commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

Round 2 answered at 77cc3e323

Blocker 1 (fixed). The two comments are corrected:

  • netstack_connect_killed.rs (CEILING): the ceiling only bounds a hang. The kill has to land before netstack gives the address up, which takes about 3 s (nud::BROADCAST_SOLICIT requests nud::RETRANS apart). A run that misses it is red at "a child ended before it was killed".
  • tests/toyos.rs (the registration): says the same.

Suggestions:

  • lent.rs:139–140: taken. The sentence now ends at the rung bell's end.
  • issues/null-sink-applies-one-connect.md: taken. The edited lines are rewrapped. Two long lines that were already on main stay as they are.
  • A peer that resolves and stays silent: declined. On QEMU's user network, slirp answers only for its own service addresses (gateway, DNS). A TCP connect to the gateway is forwarded to the host's own stack, so the result would depend on the host and the test would not be hermetic. This comes from slirp's documented behaviour; I did not run it. The two-child design stays, now with accurate comments.

Checks. The fix touches only comments and Markdown, so I ran nothing. The evidence from round 1 (ci-host-6, qemu-4, qemu-6, control-c4) still covers the code.

Japabu and others added 6 commits October 31, 2026 18:06
…nd a wait on one handle

A wakeup carried no data, yet every one was a pipe: two objects, a
pipe's four watches and, once rung, a frame, a byte per wake and a
64-byte drain loop to take it, EOF standing for "the waker is gone" and
a refused write for "the bell is gone". And a wait with a timeout on one
handle built a poller for it: an inbox page made, mapped, and given back
through the all-CPU flush shared memory takes.

An event is `EventSignal` and `EventWait` over one `Event`: a pending
bit, each side's end and one watch (`kernel::event`, host-tested). The
signal side raises it with one atomic or, posting only as it goes from
idle to pending, and never blocks or allocates; it answers `Gone` once
no wait side is left. The wait side takes it with one atomic and; a
wait side whose every signaler has gone is readable for good, and its
take answers `Gone`, never a signal nobody raised. Readiness observes
and takes nothing. The two authorities are separate by type: a wait
handle has no signal path, nor a signal handle a take. A real-time
signaler lends its window through the one path pipe wakes lend theirs
(`scheduler::post_lending`), bounded by the same quantum.

`SYS_HANDLE_WAIT` parks on one handle's one condition with a timeout,
answered from the readiness an inbox watch looks at, the notice's
listening included, so a one-handle timed wait needs no inbox. A claim
lends its watch to polls alone and is refused `NotSupported`.

ABI: `ObjectKind::{EventSignal, EventWait}`; `SYS_EVENT_CREATE`,
`SYS_EVENT_SIGNAL`, `SYS_EVENT_TAKE`, `SYS_HANDLE_WAIT`. SDK:
`toyos::wake` keeps its names over events, with `Hungup` for the other
side's end, and `poller::wait_one`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pipes that carried nothing but a wake are events now, each made by
the side that hands the other its part, so no server ever acts on an
event handle a client chose:

- window's `Waiter`, the compositor's session opens, soundserver's
  command and device-loop bells and each client's signal, the
  supervisor's app-end watchers, its file worker (a counter beside the
  event, which coalesces), each disk driver's stop and each program's
  log `ALIVE`, whose waker's end is the program's, and libc's wake and
  control signals.
- netstack's holder bells of listeners and datagram sockets, and a
  stream's receive and send bells.
- A stream holder's `RING` frame is gone: netstack hands each holder a
  third bell's waker and watches that bell, rung for a cursor or a want
  and after an `EXPORT`; its wakers' end is the holder's leaving, so
  one watched handle a place stays one.
- A block session's doorbell bytes on its connection are two events the
  driver makes and sends with `MSG_OPENED`; their wakers' end is the
  session's, and the connection goes once the session is open.

Deleted: netstack's `pipes.rs`, `stream::SERVER_WATCH` and `RING`, the
64-byte drain loops, and a poller per socket end. A connection's raw
byte calls stay: `FrameRx`, netstack's `Client::gone` and a guest test
still use them, until the whole-message receive. A holder's bells hold no page, so
netstack's places cost two pipes, not four. A guest test,
`event_sides`, holds what the host tests cannot: a parked wait woken by
another process's ring and by its end, a watch that takes nothing, and
an end that is never a ring; `kernel_objects_refused` is refused an
event too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libc's interruptible sleep, diskserver's wait for a block service's
answer, inspect's for a snapshot, the supervisor's for logkeeper's
flush, hda's for soundserver's acceptance and the terminal's for its
shell each made an inbox for one wait and gave it back after it. A
refused wait now says so where the poller's would have spun or been
ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fork's ToyOS waker wrote a byte per wake to a pipe its selector
never read, so a woken waker stayed readable for good. It is an event
now, and the selector takes it as it reports the waker's token.

No program in the image builds mio for ToyOS, so the guest test that
holds it, `mio_waker_coalesces`, depends on the fork itself: 10,000
wakes before a poll are one event, the next poll reports nothing, and a
wake after it is reported. This closes
`issues/mio-s-toyos-waker-never-drains-its-pipe.md`. Only mio's source
line in Cargo.lock moves: `cargo update -p mio` re-resolves windows-sys
for four unrelated crates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d while it lives

Until a connect is answered netstack holds the waker of the holder's rung
bell itself, so that bell cannot read as the holder leaving, and a client
killed meanwhile kept its place until netstack gave the connect up.
netstack watches the first holder's connection until the answer is
handed over and the rung after it: still one watched handle a place. A
guest test on the real netstack, `a_client_killed_mid_connect_gives_its_place_back`,
connects two children to an address nothing answers and kills one: its
place comes back while the other's connect is still pending. With the
rung watched instead, the killed child's place stays until the connects
end, and the test fails.

An event's shared state and watch were charged to its signal side, whose
charge went with the last signaler while a wait side kept the memory.
The charge is the `Event`'s now, as a pipe's is the `Pipe`'s.

Comments and records that still said pipe where an event is now, a
sentence that did not parse, broken wraps and a module out of order;
`closing-a-pipe-end-ends-every-holders-polls-on-it.md` now covers an
event's wait side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The connect to an unanswered address fails when netstack gives up resolving
it, about 3 s (nud::BROADCAST_SOLICIT requests nud::RETRANS apart), not at
the connect's own give-up. The comments said the 20 s ceiling sat far
inside the give-up; they now say the kill has to land within those 3 s and
that a run missing it is red at "a child ended before it was killed".

A peer that resolves and stays silent would remove that dependence, but on
QEMU's user network only the slirp gateway and DNS addresses resolve, and a
connect to them reaches the host's own stack, so it is not hermetic.

Also: lent.rs's ring comment ends at the rung bell's end, and the issue
record's edited lines are rewrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main f45dc5e3b (#1106 landed): head 18060b521

#1106 landed first, so the branch had conflicts. Resolved at 18060b521:

lib/link's doorbell is still a byte on its connection. The approved rule is that whoever lands second migrates the other onto events. A connection's plain write still exists in this PR, so the link builds and runs here unchanged. Phase 2 deletes those bytes, so that PR has to move the link anyway, and it moves the link to events there. I'm flagging it so the scope is visible.

One attempt reverted. I first moved #1106's I219 driver's timed interrupt wait off its one-shot poller onto SYS_HANDLE_WAIT. QEMU's e1000e arm went red: the driver panicked on NotSupported, because SYS_HANDLE_WAIT refuses a claim by design (a claim's release answers polls only). I dropped the commit, so the poller stays.

Checks at 18060b521:

  • cargo run -- --ci host: exit 0, 107 of 107 steps green.
  • QEMU, netstack_holders_exported_to_children, both the virtio-net and the 82574 arm: exit 0.
  • QEMU, five more tests: a_link_ends_under_a_held_stream_and_comes_back, stream_holders, a_client_killed_mid_connect_gives_its_place_back, udp_receive_times_out, kernel_objects_refused_alone. They passed at 7a4a921bc, which is 18060b521 plus the dropped I219 commit. They run on virtio-net and start no I219 driver, so the result holds for 18060b521.

Evidence: https://github.com/ToyOSOrg/evidence/tree/10a1940f4eeeb3959e5c2f93b3b651858c2fcc40/ipc-events/rebase-1106-18060b521

@Japabu

Japabu commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

Orchestrator: r2's comment blocker checked at 18060b5. The ceiling comments now state the ~3 s address give-up (nud::BROADCAST_SOLICIT), and both small suggestions are taken. The rebase onto f45dc5e (after #1106) is green: host CI 107/107, QEMU netstack and kernel tests including the virtio-net and 82574 arms. lib/link's byte doorbell migrates in phase 2 (#1121), which deletes byte writes on connections; that's within "whoever lands second migrates the other". Landing.

@Japabu
Japabu marked this pull request as ready for review October 31, 2026 21:44
@Japabu
Japabu added this pull request to the merge queue Oct 31, 2026
Merged via the queue into main with commit 3127437 Oct 31, 2026
10 checks passed
@Japabu
Japabu deleted the ipc-events branch October 31, 2026 22:09
Japabu added a commit that referenced this pull request Nov 1, 2026
session_lifecycle's remaining red: the job's last line and the supervisor's
forced-stop line, written just before the forced reboot, sometimes never
reach the serial console, while the kernel's records of the forced stop
do. It is red at #1108 18060b5 too (1 of 4 runs on the remote QEMU
runner), so it predates whole messages, which make it likelier (3 of 6 at
e396bb5) by shortening the time from the terminate to the stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit that referenced this pull request Nov 1, 2026
session_lifecycle's remaining red: the job's last line and the supervisor's
forced-stop line, written just before the forced reboot, sometimes never
reach the serial console, while the kernel's records of the forced stop
do. It is red at #1108 18060b5 too (1 of 4 runs on the remote QEMU
runner), so it predates whole messages, which make it likelier (3 of 6 at
e396bb5) by shortening the time from the terminate to the stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit that referenced this pull request Nov 1, 2026
session_lifecycle's remaining red: the job's last line and the supervisor's
forced-stop line, written just before the forced reboot, sometimes never
reach the serial console, while the kernel's records of the forced stop
do. It is red at #1108 18060b5 too (1 of 4 runs on the remote QEMU
runner), so it predates whole messages, which make it likelier (3 of 6 at
e396bb5) by shortening the time from the terminate to the stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit that referenced this pull request Nov 1, 2026
session_lifecycle's remaining red: the job's last line and the supervisor's
forced-stop line, written just before the forced reboot, sometimes never
reach the serial console, while the kernel's records of the forced stop
do. It is red at #1108 18060b5 too (1 of 4 runs on the remote QEMU
runner), so it predates whole messages, which make it likelier (3 of 6 at
e396bb5) by shortening the time from the terminate to the stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant