Repository navigation
osbench: the rows that compare ToyOS with Linux on one machine - #1053
Conversation
Review r1: PR #1053 at ada5bf2ArchitectureExtending The UDP fix has the right owner. netstack writes the datagram into the socket's pipe and then answers, so a client that gave up on its own would race that write. Putting the deadline in Checked and fine:
FindingsBLOCKER 1: the ToyOS crunch runs its all-core phase with no stop, which is a change to the owner's condition that the owner hasn't approved.
BLOCKER 2:
Move the three lines back onto BLOCKER 3: a sub-millisecond read timeout becomes an unbounded wait.
BLOCKER 4: the r4 reading of the crunch rows is not like for like, and it doesn't say why. Evidence
Suggestions
VerdictSEND BACK. Blockers 2 and 3 are one-line code fixes. Blocker 4 is an evidence-prose fix. Blocker 1 needs either the default change in (a) or the owner's explicit acceptance in (b). |
Review r2: PR #1053 at 12d0ebaScope: round 1's four blockers, the cool-down question, and whether the round added a defect. The round is +99/−59 over 9 files: one code commit and the r5 evidence. ArchitectureThe round keeps the shape round 1 accepted. With no temperature, the program now refuses to do the risky thing instead of falling back to the CPU's own throttling. That is the simpler and more honest design: no blind timer, and the gap's exit is the kernel sampler that Round 1's blockers1. Unguarded all-core crunch: resolved on ToyOS, and the cool-down holds on Linux.
2. 3. Sub-millisecond timeout: resolved as asked, but see new blocker A.
4. Not-like-for-like marks: mostly resolved, but see new blocker B.
New findingsBLOCKER A:
BLOCKER B: evidence r5 tables and README: two false causal claims.
BLOCKER C:
Suggestions
VerdictSEND BACK. Blocker 1's guard and the Linux cool-down hold, and blockers 2 and 3 are fixed with a working control. Blockers A, B and C are each a few lines of code or prose. None needs a new guest or metal run. |
Review r3: PR #1053 at c5dfa51Scope: r2's blockers A, B and C, the two suggestions the implementer took, and an identifier check of the r6 evidence. The round is one commit, +36/−18 over 5 files, plus the r6 evidence. Everything r2 accepted stays settled. ArchitectureThe round keeps the design. Zero-timeout handling now sits in the one conversion every socket timeout passes through, so there is one rule in one place, as the other std platform layers have it. The issue and evidence corrections are prose only. r2's blockersA. Zero timeout: resolved.
B. Evidence corrections: resolved.
C. Nonblocking issue: resolved, and the claim is true.
The accepted suggestions
Evidence identifiersI scanned all seven r6 files, including the 700 KB ci-host log, and found:
The FindingsBLOCKER D.
VerdictSEND BACK. A, B and C are resolved, and the evidence leaks nothing. D is a few lines in |
The issue asked for one program, built for Linux and for ToyOS, measuring memcpy, pipes, a remote munmap, a 1 ms timer, NVMe reads, an idle minute's energy and the machine's identity; the owner added the wired and Wi-Fi LAN and a short CPU crunch that does not overheat the laptops. osbench is that program already, so it gains the missing rows rather than a second program being written: - mem: `mem.memcpy.<size>` from 64 B to 64 MiB by fours, which replaces `mem.bw.memcpy`; and `mem.munmap_touched_remote.4k`, a page written here, read by a sibling spinning on another core and unmapped here. Each side reads its APIC ID (CPUID.0BH:EDX, 01H:EBX where 0BH enumerates nothing) before and after each of ten windows, and `osbench_rules::pair::admit` refuses a window whose sides share an ID or either of which moved. - system: pipe throughput at 64 B, 256 B, 1, 4, 16 and 64 KiB writes. - time: `time.timer_1ms.late`, the periodic wake at a 1 ms period. - nvme: 4 KiB random and 1 MiB sequential reads of the first NVMe namespace, opened read-only with O_DIRECT. Linux needs root and says so unprivileged; ToyOS mints no grant to a disk's blocks outside a file server's role or the idle slot, so it reports unsupported. - energy: the powercap package-0 counter read each second over an idle minute, mW, its wrap through `osbench_rules::energy::span`. ToyOS counts no package energy: unsupported. - identity: DMI model, microcode, PCI, USB, NVMe and cpuidle from sysfs on Linux, with no serial, UUID or network identifier; PCI from the kernel's inventory on ToyOS, and the rest unsupported by name. - lan: the bench broadcasts for an `osbench-peer` and times TCP to it, connect, a 64 B round trip, and 5 s uploads and downloads, each result named for the link that carried it (`lan.wired.*`, `lan.wifi.*`). The bench opens every connection, so ToyOS is measured although the host cannot reach it (issues/the-host-cannot-reach-the-t14-while-it-runs- toyos.md), and neither end is configured with an address or prints one. ToyOS's Wi-Fi rows are unsupported (issues/there-is-no-wifi.md). - crunch: deflate of a fixed 1 MiB corpus, toyos-crypto's SHA-256, a 160x160 f64 product and syn parsing a generated Rust program, 4 s each on one thread and then on every CPU, with an idle cool-down after each phase. Where the package temperature reads (Linux's coretemp or k10temp), a phase stops at TjMax less 15 C, TjMax being the sensor's critical temperature or, for k10temp, which reports none, AMD's published 105 C for the 4800U; each run names the source. ToyOS reads no temperature: it says so and the phases run their length, the CPU's own thermal control standing behind them. The pure rules are `tests/guest/osbench/rules` and both LAN ends are `tests/guest/osbench/lan`, nested crates because osbench builds for no host; the peer runs on any machine on the link. On ToyOS `net`'s loopback rows are now unsupported up front: netstack refuses 127/8, and benchcase now starts netstack on the T14's I219 for `lan`. The crypto rows come from the crypto oracle probe's published table rather than being measured again. The issue is renamed for what remains, the published runs, and lists what ToyOS cannot yet measure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first Linux runs (three per bench, f644cef) read crunch.all.parse at 2.8 s a unit on the T14's 8 threads and 3.1 s on the AMD bench's 16, against 59 and 73 ms on one. A unit is one thread's parse of the whole generated program, so the figure is no unit spread across threads. A counting allocator around the same syn and corpus finds 588,862 allocation and free pairs per parse. The same runs' mem.malloc_churn falls from 33.6 M to 3.2 M pairs a second in total on the T14 at 8 threads, and from 28.5 M to 1.8 M on the AMD bench at 16. At that contended rate, about 2.5 to 9 us a pair per thread, a parse's allocations alone take 1.5 to 5 s. That is musl's malloc serialising, which the static Linux build links; ToyOS std's dlmalloc has one lock too. The other three workloads allocate a few blocks a unit and scale. The row stays, since it is a real program's work on each system, and its header now says what it measures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The T14's ToyOS osbench boot at f644cef killed `osbench lan` at the list bound, 321 s into a group whose discovery is bounded at 60 s: with no peer on the link its `recv_from`, under a 1 s read timeout, never returned (issues/std-udp-socket-ignores-its-timeouts.md). That was netstack parking a datagram receive until a datagram came. #1052 has since made a datagram socket its holder's own and std's receive wait on it with the read timeout, which is that issue's fix; its exit asked for a guest test with no answering peer, and this is it. `udp_receive_times_out`, on netcase's real netstack: two receives under a 300 ms timeout with nothing sent each end `WouldBlock`, std's Unix word for a receive timeout, after at least 300 ms and inside a 30 s hang ceiling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- crunch: a system that reads no package temperature runs the one-thread phase alone and reports every all-core row `unmeasured`, so nothing is heated past a limit nothing reads. The owner asked that the laptops not be overheated, and on ToyOS the guard was the CPU's own throttling. The blind 30 s cool-down goes with the phase it followed. The gap entry cites issues/the-kernel-owns-cpu-performance-state.md, whose sampler is the recorded way to read both temperature and energy, and issues/the-t14s-thermal-modes-do-not-move-its-package-power-limits.md for the T14's all-core clock under ToyOS. The metal row's bound text drops the phases ToyOS no longer runs; the bound stands. - std: a non-zero timeout under a millisecond is one millisecond. The protocol reads 0 as no bound, so a 500 us read timeout truncated to 0 waited forever, on a datagram receive and on TCP's read, write and connect timeouts alike. `udp_recv_timeout` adds a 500 us receive; with the truncation restored it stalls at its hang ceiling. - osbench-lan: `download` no longer counts the first read's bytes, which arrive before its clock starts. - osbench-rules: `ASSUMED_STOP_MC` is derived from the table its doc names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- std: `timeout_ms`, the one conversion every ToyOS socket timeout and `connect_timeout` takes, refuses `Some(Duration::ZERO)` with `io::Error::ZERO_TIMEOUT`, as std's other platform layers do; the last commit's clamp had made a zero timeout a silent 1 ms one, and before it a wait without bound. `udp_recv_timeout` asserts `InvalidInput` from UdpSocket's two setters and from TcpStream::connect_timeout. - crunch: a cool-down with no starting temperature says so and returns rather than idling its whole bound. - osbench-lan: the inline comment that repeated `download`'s doc goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd stops when it goes quiet A sensor found at selection that then read nothing made `cool` return at once, and the all-core phase ran with a watcher that skipped every failed reading: unguarded, where the owner asked that the laptops not be overheated. The comment that said the next phase's stop still guarded it was false. - The group's first successful reading gates the all-core phase. With none, only the one-thread phase runs and every `crunch.all.*` is `unmeasured`, as on a system that reads no temperature. - `cool` takes that start reading as a value; it has no unread case. - The phase watcher stops the phase at a failed reading as at the limit, and the result says which (`stopped=no|hot|unread`), so a sensor that dies mid-phase leaves nothing running unguarded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1d15a4f to
627073e
Compare
Problem
issues/no-program-measures-toyos-against-linux-on-one-machine.mdasks for one program, built for Linux and for ToyOS, that measures each bench under its Ubuntu and under ToyOS. The owner added the LAN (wired, and Wi-Fi under Linux) and a short CPU crunch that does not overheat the laptops.Decision: extend osbench, not a second program
tests/guest/osbenchalready is that program (one source,x86_64-unknown-toyosand staticx86_64-unknown-linux-musl, oneosbench v=2line per result,unsupportedfor a missing facility). This PR adds the missing rows to it. Two nested crates hold what a host must build:osbench/rules(pure judgements, host-tested) andosbench/lan(both LAN ends; the peer binary runs on any host). osbench itself builds for no host, so its pure logic cannot carry host tests.Resulting rows
mem.memcpy.<64 B..64 MiB by 4x>(replacesmem.bw.memcpy)pipe.throughput.{64,256,1K,4K,16K,64K}; round trips are the existingpingpong.pipe.*mem.munmap_touched_remote.4k: sibling on another core read the page; 10 windows, each side's APIC ID (CPUID.0BH:EDX, 01H where 0BH enumerates nothing) before/after,osbench_rules::pair::admitrefuses a window with shared or changed IDstime.timer_1ms.late(periodic wake at 1 ms)nvme.read.4k.random,nvme.read.1m.seq: O_RDONLY|O_DIRECT, nothing writessudo osbench nvme), elseunmeasuredenergy.idle_60s.package: powercap package-0, per-second mW, wrap viarules::energy::spanidentity: DMI model, microcode, PCI, USB, NVMe, cpuidle; never serials, UUIDs, MACs or addressesinventorysyscap added to benchcase's runner); USB/NVMe/cpuidle unsupported by namelan.{wired,wifi}.tcp.{connect,rtt.64,upload,download}issues/there-is-no-wifi.md)crunch.{single,all}.{deflate,sha256,matmul,parse}, 4 s each, idle cool-down after each phasecrunch.all.*isunmeasuredCrypto rows are not re-measured: the crypto oracle probe's table is the evidence (
ToyOSOrg/evidenceatbed0b00efac82546b995f0841da54aea11c45694,r2/table.md).LAN peer
The bench broadcasts
osbench-lan/1 helloto 255.255.255.255:47321 and takes the answer's source as its peer, then opens every TCP connection itself. So no address is configured or printed anywhere, and ToyOS is measured even though the host cannot open a connection to it (issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md). The peer (osbench-peer) echoes, sinks with a count, or sources for a span. It bounds sizes, spans and concurrent sessions, and refuses anything else. I chose the development Mac as peer: one bench serving the other would put the AMD bench's Wi-Fi in the T14's wired path. The peer must be wired for a wired row to measure the bench's link.Heat limit
osbench_rules::heat::limit: TjMax is coretemp'stemp*_crit(the CPU's ownIA32_TEMPERATURE_TARGET). k10temp reports none, so the 4800U takes AMD's published 105 C. Each run printscrunch.limit_from. The 15 C margin is policy: stop before the CPU's own throttling. An unknown CPU stops at 85 C, labelled assumed.Other changes
benchcase.tomlstarts netstack withpci:02:00:00/8086:15fc(on the AMD bench netstack leaves). The runner receivesnetstackandinventory. On the T14 every other ToyOS group now runs beside netstack's DHCP/mDNS, as Linux's do beside its stack.neton ToyOS reports its loopback rows unsupported up front. With netstack running, loopback would otherwise be refused (netstack refuses 127/8 by design) and red the group.adds_ms480 000), derived intests/toyos.rs.no-published-run-compares-toyos-with-linux-on-one-machine, since the program now exists. Citations are moved. It lists every ToyOS gap.Checks and evidence (https://github.com/ToyOSOrg/evidence/tree/8c8291f65eff77753c74c538e314b8db394ce3e7/r1)
cargo run -- --ci hostat f644cef: exit 0, 103 steps. This includesosbench-rules(13 tests),osbench-lan(5, including a loopback discovery and every request end to end) and the osbench clippy shapes for both Linux targets.Linux musl build and peers: exit 0.
Negative controls (checked patches, restored in the same run):
a_pair_on_one_cpu_is_refusedred.QEMU TCG smoke (4 vCPU, no NIC) of identity, energy, nvme, lan, mem, time, system and crunch: every job exit 0. Every unsupported row named its reason. The munmap pair kept 10/10 windows. TCG numbers are meaningless.
Metal staged for T14 and AMD (exit 2, machine untouched).
Linux runs (orchestrator, three unprivileged
allper bench at f644cef): tables with spreads at https://github.com/ToyOSOrg/evidence/tree/3f183ae653e95bb941564cb725a232a6f54bf6a9/r2.lanrefused, because no wired peer exists yet;nvme/energyare unmeasured without root.crunch.all.parse, about 47x slower per unit than single-thread, is the allocator, not a measurement bug. A parse is 588,862 alloc/free pairs, and the same runs'mem.malloc_churnshows static musl's malloc collapsing under threads. Now said in the crunch header (75878e9).ToyOS boots at f644cef (orchestrator, no peer):
osbenchjudge.lanalone: killed at the 540 s list bound (below).compare-t14.md,compare-amd.md), with Linux's three-run median beside ToyOS's single run, a ratio, each side's spread, and every ToyOS-unsupported row marked.ada5bf2 (before the rebase): a datagram receive ends at its read timeout.
UdpSocket::set_read_timeoutwas inert (issues/std-udp-socket-ignores-its-timeouts.md). netstack parked a datagram receive until a datagram came, and the client waited without bound.udp_receive_times_outon netcase, and the issue's closure. On main a receive timeout isWouldBlock, std's Unix word, so the test asserts that.T14 ToyOS boot of ada5bf2 (no peer):
lanalone:osbench_lanexited 1, the clean refusal after 60 s. The hang is gone.crunchandpersistran. They are added to the T14 table at https://github.com/ToyOSOrg/evidence/tree/3d1a97292494141714862fb742b289978b8fda9b/r4 (compare-t14.md; the AMD table is r3's).Rebased onto main (2a5b446), after #1052, #1054, #1058 and #1060
connection.rssdk/libc/src/socket.rs,sdk/toyos/src/net.rs,system/netstack/src/main.rs,system/netstack/src/serve.rsand the nonblocking issue;sdk/std/sys/net/connection.rs(the zero refusal and the clamp, now on main'sduration_to_ms), the UDP test (WouldBlock) and its registration comment.udp_receive_times_outon main's datagram sockets;cargo run -- --ci host, 103 steps.files'servedrows, and netstack: a listener and a datagram socket are their holders; no request names a socket by a number #1052's datagram sockets servelan's peer search. A ToyOS run at this head would read those rows afresh. Linux's rows are untouched.Review r1 (SEND BACK, four blockers), answered in 12d0eba
crunch.all.*isunmeasured. The blind cool-down is gone. The gap entry cites the performance-state issue's sampler and the T14's refused PL1/PL2 writes.settle's doc is back onsettle. The rebase took main's netstack, so this no longer applies.duration_to_ms.udp_receive_times_outadds a 500 us receive. QEMU: exit 0.crunch.all,persist, huge-page memcpy and bw, 2 MiB first touch, realloc, 1 MiB churn, and yield.Suggestions taken:
downloadno longer counts the first read's bytes;ASSUMED_STOP_MCis derived from its table;Durationas imported; huge pages in the not-like-for-like marks; the nonblocking issue filed. Since dropped: main honours UDPset_nonblocking.Suggestion declined: a test of netstack's new wake in
main.rs. That stands on inspection: three lines that take the soonest deadline.Checks at 12d0eba:
cargo run -- --ci host: exit 0, 103 steps.udp_receive_times_out(QEMU): exit 0.unmeasured, exit 0.Review r2 (SEND BACK, three blockers), answered in c5dfa51
io::Error::ZERO_TIMEOUT, as std's other platform layers do. The check sits intimeout_ms, the conversion every ToyOS socket timeout setter andconnect_timeouttakes.udp_receive_times_outassertsInvalidInputfrom UdpSocket's two setters and fromTcpStream::connect_timeout. QEMU: exit 0.crunch.all.*marks are each machine's own, so the AMD table no longer cites the T14's PL1/PL2 issue. The T14's single-thread gap is now written as ToyOS's own unexplained 3,788 MHz ceiling, owned byissues/toyos-uses-what-modern-hardware-offers-for-speed.md. The idle-C-state link is labelled an inference: https://github.com/ToyOSOrg/evidence/tree/0f94abc7cd2e2e1fd663649dc06bfdcc51147b3e/r6WouldBlockat once.Both suggestions taken:
cool()returns when no starting temperature was read, and the duplicate comment is gone.cargo run -- --ci hostat c5dfa51, from a clean target: exit 0, 103 steps. An earlier run hit the Mac's full disk and was discarded.Review r3 (one blocker, D), answered in 1d15a4f
crunch.all.*isunmeasured.cool()takes that start reading as a plain value.stopped=no|hot|unread. This replaces thestopped_hotfield.cargo run -- --clippy(exit 0, 27 invocations clean) and the Linux musl build. Per the brief, no run: reading the code establishes it.Waiting on the owner
lan: the development Mac is on Wi-Fi and the AMD bench is Wi-Fi-only. Until a wired peer exists, the T14 metal judge is red onosbench_lanalone, and Linux'sallexits 1 for the same reason.sudo osbench nvmeandsudo osbench energyunder Linux. Without root these rows areunmeasured, and ToyOS reports both unsupported.Should "no peer" be unmeasured rather than a refusal?
My recommendation is no. Keep the refusal.
lansays the boot lacked its precondition.lanjob. The bench's environment should decide which boot runs, rather than a pass covering a missing measurement.Not run: a LAN measurement on either system; the Linux root rows. The Linux NVMe, powercap and
getifaddrslink paths have not executed past their permission or peer checks.Risks / open
maingiving netstack the T14's I219 (issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md: no T14 row reads netstack on the I219;issues/the-t14-hung-after-rebooting-with-its-i219-faulted.md). If netstack fails on metal,lanrefuses and the row reds.lanrefuses unlessosbench-peerruns on the LAN. Discovery depends on broadcast reaching the peer: AP client isolation or the macOS firewall can block it.issues/the-kernel-owns-cpu-performance-state.md(its sampler) owns it, and the comparison issue's gap entry says so.🤖 Generated with Claude Code