[Japabu]

ToyOS's persistent rows run on the bench's NVMe. #1079 staged this patch until ToyOS could designate its own DATA, which #1085 brought. Rebased on #1095 (370d7d9f6), so ToyOS boots off the same NVMe.

Problem and result

In #1079, persist compared ToyOS's boot stick with Linux's NVMe and read "the media differ". Now ToyOS's persistent filesystem is DATA's /home: the bench's TOYOS-DATA partition, through the DATA file server and the NVMe driver, medium nvme. Linux's is its home, ext4 on the same NVMe namespace. persist runs the durable rows and then files' operations as fs.disk.*, on both systems, so those rows can resolve. This is the disk and filesystem scoreboard toward Linux parity.

Changes

• osbench:

  • ToyOS's persistent_dir is /home with medium nvme, a constant that the metal row alone makes true;
  • storage runs durable rows, then files::suite(disk);
  • ToyOS's files keeps /tmp;
  • on Linux, fs.disk.mount carries the superblock options, and served_by names the device's write-cache mode;
  • the compare's SETUP reads fs.disk.served_by.

• The osbench metal row judges DATA with hostdisk::left_alone and the loader's volumes:

  • Mounted passes;
  • Formatted is the format boot, no measured run, with its duration stated;
  • Foreign or memory is refused.

• README: the disk rows' rules, as procedure:

  • one NVMe, and the compare refuses differing media;
  • each system's own filesystem;
  • one durability, which each side's recorded fields show;
  • one procedure for the cache: each row reads files it wrote in its run, no cache is dropped, and ToyOS's 32 MiB and 2 s write-back is part of what is measured.

  It also says that ToyOS's log volume shares the NVMe.

• Issues: each disk row ToyOS loses, with its evidence:

  • a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk;
  • a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk, whose host probe confirms the cause;
  • a-file-server-request-costs-three-to-twelve-times-a-kernel-file-call;
  • toyos-durable-writes-on-the-t14s-nvme-run-six-times-slower-in-some-boots;
  • a-created-fsynced-and-unlinked-file-costs-toyos-more-than-linux.

  The comparison record and the 64 MiB record are updated.

nvme_leaves_the_benchs_disk_alone's designated premise came with #1085, so it is not in this PR.

What the landing head's validation shows (069cbbeb8)

One ToyOS boot and two Linux runs per bench. Each was judged PASS, exit 0, from a tree pinned at 069cbbeb8 (r4). So no row resolves yet; the scoreboard's further pairs are where these become verdicts.

row	T14: ToyOS vs Linux	AMD: ToyOS vs Linux	reading
persist.fsync_clean	0.46 ms vs 23 µs	1.26 ms vs 64–66 µs	ToyOS's. A clean fsync syncs the whole volume, and Mounted::sync writes the superblock, so it costs a block write and a Flush. Linux's clean fsync takes tens of µs.
persist.write_4k_fsync	1.10 vs 1.01–1.04 ms	2.60 vs 2.99–3.58 ms	About parity: a durable write costs a Flush either way. The T14 ran six times slower in 2 of 5 boots, and partway through a third, with no storage change between; that is recorded as unexplained.
persist.append_4k_fsync	1.01 vs 0.91–1.20 ms	2.93 vs 3.30–3.78 ms	About parity, and ToyOS's lower on the AMD.
persist.create_fsync_unlink	1.17 vs 0.95–0.97 ms	3.53 vs 3.26–3.28 ms	ToyOS 8–22% dearer; filed, not yet attributed.
persist.seq_write_fsync	347 vs 527–585 MB/s	298 vs 544–548 MB/s	ToyOS's: mostly the server's copy, plus its flushes.
fs.disk.read_4k_at and the other cached ops	18.2 vs 0.72–0.74 µs	20.5 vs 1.46–1.47 µs	ToyOS's: the file server's round trip, 3 to 12 times ToyOS's own kernel /tmp.
fs.disk.seq_read.4096	431 vs 7,760–7,780 MB/s	151 vs 5,084–5,089 MB/s	ToyOS's: its cache. FIFO eviction of a file just over CLEAN_LIMIT; a host probe shows every pass missing.

Setup, and why it explains none of these:

• Linux's clocksource was tsc in every landing-head run.
• The two systems write different regions of one NVMe.
• Their free space differs, which would favour ToyOS's fresh volume.
• ToyOS's log writes share the device; every boot's persist completed.

Checks

Evidence: index.

• cargo run -- --ci host at b4121ccc0: exit 0, 105 steps green. Every commit since (069cbbeb8 to the head) changes only issue records.
• The osbench row at b4121ccc0:
  • the AMD's c8f3fbfb0 boot through toyos-metal boots every bench image off the bench's internal NVMe #1095's runner: PASS;
  • the AMD's fa7258221 designated boot: PASS;
  • the 2026-10-24 memory-DATA boot: refused;
  • the format substitution: refused.
• Metal at the landing head, 069cbbeb8: run on both benches, Linux, ToyOS, Linux, and the judge passed on each, exit 0 (r4). The c8f3fbfb0 runs passed too (r2, r3).
• The cache probe: its whole patch is republished as the probe that ran (r3, with a correction in the index).
• The Linux half's sha256 is 98d8b7f2…f796; both musl targets check.

Risks

• ToyOS is at or near Linux on the durable writes (write_4k_fsync, append_4k_fsync) on both benches, and loses every other disk row. Each loss is filed; this PR measures them and doesn't fix them.

🤖 Generated with Claude Code

[Japabu]

Review of #1097 at c8f3fbf

The branch has since moved to 8267716, which adds one issue record. I note it at the end.

Architecture

The design fits. A ToyOS program cannot ask what holds /home, so sys::persistent_dir states nvme as a constant, and the osbench metal row is the gate that makes the constant true. That is the smallest honest arrangement, and sys/toyos.rs says so. Moving fs.disk.* into persist and deleting files::targets also removes the only ToyOS-only target. The procedure is fair: both systems use one device, one row order, each its own filesystem, and every file is written in its run. Net change is +125/−53. Most of the growth is the judge (about 45 lines in tests/toyos.rs) and the README section.

Several things the README states as rules are false or unsupported. That includes two of its four fairness claims: durability and cache state. One of them is contradicted by the PR's own AMD validation.

Findings

BLOCKER tests/guest/osbench/README.md:160-161 and src/sys/linux.rs:509: the README says a nobarrier mount would show in fs.disk.mount. It would not.

• mount_of reads mountinfo's per-mount options (field 6) and drops the per-superblock options after - (field 11). ext4's barrier/nobarrier is a superblock option, so it can only appear in field 11. The AMD run confirms this: it recorded fs.disk.mount=ext4_rw,relatime, and fs.mem.mount shows no tmpfs size=.
• Consequence: the README's evidence for Linux's side of the one-durability rule cannot detect the setting it claims to detect.
• Fix: put the super options into mount_options; mount_of already splits that side of the line. Or delete the claim.

BLOCKER README.md:165-167 and src/files.rs:3-4: "fs.disk.* are cached reads on both, and no row reads the disk cold" and "on the disk these are the cached path" are false for ToyOS.

• Writes. DATA's cache flushes, including an NVMe Flush, once 32 MiB is dirty (system/fileserver/src/cache.rs:50, DIRTY_LIMIT). It also syncs the whole volume 2 s after an unsynced write (writeback.rs:8). So every round of the 64 MiB fs.disk.seq_write.* writes the disk on ToyOS.
• Reads, measured. The AMD run at c8f3fbf (evidence r2) read fs.disk.seq_read.4096 at 158 MB/s p50, with min 158 and max 426. In other words, four rounds out of five ran at about a third of the fifth. Cached 4 KiB transfers in the same boot ran at 16.9 µs for a seek plus a read (read_4k_at).
• Reads, cause (inference, not measured). CLEAN_LIMIT is 64 MiB (cache.rs:47), exactly the file's size. Eviction follows insertion order, and a hit does not refresh a block's place (cache.rs:140-146). Once write-back cleans the file's second half, a sequential read evicts the blocks it reads next.
• The procedure is fair: each system's write-back and cache policy is its own and part of what is compared. The README states outcomes, though, and the evidence refutes them.
• Fix: state the rule as procedure: each row reads files it wrote in its run, and neither system's cache is dropped. Say that ToyOS's write rows include its 32 MiB and 2 s write-back.
• Record the seq_read.4096 finding in issues/ with its evidence, as 8267716 does for fsync_clean (No silent debt). The cheapest check of the cause is a fileserver host test: a Cache over the counting disk the cache tests already use, 64 MiB written in 4 KiB calls, a flush, then five sequential reads, counting disk reads.

BLOCKER README.md:145-147, 150-153: two statements describe the medium check wrongly.

• "The compare refuses a verdict on any row where it is not [nvme]" is wrong. The compare refuses where the media differ (compare/src/lib.rs:429-430).
• "Every row's served_by and medium name the filesystem and the disk it measured" is false for ToyOS. There both are constants (sys/toyos.rs:395-397), so the compare's media check cannot see ToyOS's medium, and the metal row is the only guard. The code comment says this correctly; the README should say the same thing in one sentence.

BLOCKER src/files.rs:94-96: the rewrap broke the doc comment ("…and its medium. Each" / "one's comment is…"). Rejoin the line.

BLOCKER Evidence: no boot of this head's image has been judged by this head's row.

• The AMD validation boot was started by toyos-metal boots every bench image off the bench's internal NVMe #1095's NVMe runner, and this branch's harness refused its readback (judge-pre-1095-harness.log: "boot file names no stick_secs").
• By inspection, that kernel.log would pass both new checks: line 326 has the NVMe's TOYOS-DATA partition (type 064E3777…), line 344 has mounted the DATA volume, and there is no format line. That is inspection, not a run.
• The T14 run is still pending. The T14 is where the in-memory version of the 64 MiB write once stalled, so it is the one that retires the seq_write risk.
• On the AMD, persist completed in 8 s with no fileserver restart, so the risk is retired there for one boot.
• Needed before landing: the osbench row passing at the landing head on both benches, and the T14's persist completing. The PR's Checks list should then report r2; that correction is the orchestrator's.

SUGGESTION Boot medium. No code assumes a stick. data_on_the_nvme keys on the NVMe driver's partition line and works on either boot.

• Two things change under toyos-metal boots every bench image off the bench's internal NVMe #1095. The only validated ToyOS boot already had its LOG on the same NVMe (partition 7), so logkeeper's writes and the LOG server's write-back Flushes share the namespace with persist. On main's stick harness they go to the stick.
• tests/images/benchcase.toml:7 says "The stick has no DATA partition".
• Suggested: one README sentence on where ToyOS's log volume is, and "the image has no DATA partition" now. Both cost nothing, and both become required at rebase if toyos-metal boots every bench image off the bench's internal NVMe #1095 lands first.

SUGGESTION tests/toyos.rs:1770-1809: these duplicate tests/common/hostdisk.rs::left_alone.

• left_alone already classifies DATA as Foreign, Formatted or Mounted from the same lines. It also checks that the NVMe opened its session on DATA, where data_on_the_nvme argues from "the file server refuses two".
• Once toyos-metal boots every bench image off the bench's internal NVMe #1095's left_alone(log, ours) lands, Formatted means refuse and Mounted means pass, and both parsers could go.
• If they stay:
  • The format duration and since_boot serve no decision, and their only control produced 0.000 s by substitution.
  • The "it is absent this boot" test matches only the path that mounts and then cannot serve. The other refusals print "DATA is absent this boot" and are caught by the missing mount line instead.
  • The memory path's own line ("this machine has no DATA partition…") reports as "the file server said nothing of DATA".

SUGGESTION The durability bullet's Linux half also depends on the device reporting a volatile write cache. Recording /sys/block/<disk>/queue/write_cache beside the mount would make that half evidenced rather than assumed.

8267716 (after the requested head)

The new issue matches the evidence: fsync_clean read 1,233,821 ns under ToyOS against 69,703 and 56,649 under Linux, and write_4k_fsync read 2.9 ms against 3.0–3.6 ms. It also matches lib/bcachefs/src/fs.rs:927-939: sync writes the superblock unconditionally between two cache flushes, so a Flush is owed every time. No finding.

Verdict

SEND BACK

[Japabu]

r1 response, head 069cbbeb8 (rebased on #1095)

• B1, fixed. mount_options carries mountinfo's superblock options (field 11), where ext4's nobarrier is. Linux's served_by names queue/write_cache (suggestion taken).
• B2, fixed.
  • The README states the cache rule as procedure: each row reads only files it wrote in its run, and neither cache is dropped. It says ToyOS's write rows include its 32 MiB and 2 s write-back.
  • The seq_read.4096 finding is filed with its evidence and your probe. A cache over the counting disk, CLEAN_LIMIT + 16 blocks written in 4 KiB calls and flushed, made 16,400 disk requests on each of five passes; CLEAN_LIMIT - 16 made none (evidence r3 probe/, a patch applied and reverted). files.rs's header no longer claims the cached path.
• B3, fixed. The compare refuses differing media. On ToyOS the medium is a constant that only the osbench row makes true.
• B4, fixed.
• B5: the landing head's runs are staged for both benches. The T14 run at c8f3fbfb0 passed with persist complete in 11 s, with no file server restart.
• Suggestions taken:
  • the boot medium: the README says ToyOS's log volume shares the NVMe, and the image comment says "the image has no DATA partition";
  • one parser: the judge is hostdisk::left_alone(log, loader_named(loader)), with Mounted passing, Formatted the format boot and anything else refused. It passes the AMD's NVMe-runner and designated boots, and refuses the memory and format logs.

The T14 run also showed a durable write costing ToyOS six times Linux's on that disk. It is filed with the other three below-parity rows, as the owner's parity goal asks.

[Japabu]

Review r2 of #1097 at 069cbbe, with d6fc8be noted

This round was asked to review 069cbbe. The branch has since moved to d6fc8be, which replaces the T14 record and changes nothing else. Both heads are reviewed below. The metal runs at 069cbbe also cover d6fc8be, because d6fc8be touches only issues/.

Architecture

Unchanged since r1, and still a good fit. Moving the judge to hostdisk::left_alone(log, loader_named(loader)) removes the second parser. Source net is still small: tests/toyos.rs +41/−4, osbench +40/−30. Most of the growth is the README section and five issue records.

r1 blockers

• B1, resolved. mount_of now returns mountinfo's field 11. Both benches' Linux runs at r4 record fs.disk.mount=ext4_rw,relatime_rw and served_by=…, write cache write back, so the README's durability bullet is now backed by recorded fields.
• B2, resolved in code and README. The cache rule is now stated as procedure. The write-back limits match cache.rs:48-51 and writeback.rs. The files.rs header no longer claims the cached path. The seq_read record matches cache.rs:140-146 and :316-324: a hit does not re-queue a block, and blocks are evicted from order's front. One evidence defect remains (N3 below).
• B3, resolved. The README now says the compare refuses differing media, and that ToyOS's medium is a constant that only the row makes true.
• B4, resolved. files.rs:94-95 reads correctly.
• B5, resolved. The judge tree toyos-1097-check is at 069cbbe and clean. Both ToyOS boots log supervisor: build 069cbbeb878d… clean. On both benches the osbench row printed PASS osbench, with "1 passed, 0 failed". persist finished in 9 s on each with no file server restart, and fs.disk.seq_write.1048576 completed on both, the T14 included. The Linux half's sha256 98d8b7f2…f796 matches r3/checks/linux-osbench-SHA256SUMS. compare.md shows no media refusal on the disk rows.
• Suggestions taken: the log-volume sentence, the image comment and the single judge are all done correctly.

Rebase

Nothing this PR depends on broke. Readback::loader() and hostdisk::loader_named come from #1095. Host CI ran on the rebased tree at b4121cc (exit 0, 105 steps), and the judge compiled and passed at 069cbbe on real readbacks. Main has since moved to 9c677be (#1096, #1093); git merge-tree against it is clean, and their tests/toyos.rs hunk only adds a separate row.

Between c8f3fbf and 069cbbe, nothing under system/nvme, system/diskserver, system/fileserver or lib/bcachefs changed. The T14's acpiserver idle census is identical in both boots (MWAIT 0x31 253 µs + MWAIT 0x60 1048 µs).

Findings

BLOCKER (resolved at d6fc8be) issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md at 069cbbe: the landing head's own T14 run contradicts this record. ToyOS read write_4k_fsync 1,099,737 ns against Linux's 1,011,579–1,035,997, and fsync_clean 464,505 ns against 3,019,602 at c8f3fbf. d6fc8be restates the record with both boots, and I checked every number in its table against metal6/metal7. The two points in N1 remain.

BLOCKER N1 issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md:24-27: two statements go beyond the evidence.

• "The first followed DATA's format by less than a day, so the drive's own state is a candidate." This does not tell the two boots apart. fa72582's T14 boot at 12:13 still logs the DATA partition is not ours, and DATA then mounts at 16:08 (c8f3fbf) and at 16:47 (069cbbe). Both boots followed the format by hours. Either state what does distinguish the first boot (for example, that it was the first workload after the format, if a log shows that), or delete the sentence.
• "a superblock write and a Flush where Linux issues neither." Nothing measured what Linux sends to the device on a clean fsync. 23 µs only shows that it is cheap, and a Flush to a device whose cache holds nothing dirty can also be cheap. The same claim is in the PR body ("Linux's issues neither"). The cheapest fix is to state the measurement: "where Linux's clean fsync takes 23 µs". If the cause matters, read the namespace's flush counters in /sys/block/<disk>/stat before and after the row.

BLOCKER N2 issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md:22: "The disk adds nothing to these" is false for one of the five rows it covers.

• open_close reads 18,686 ns on the NVMe at c8f3fbf, against 14,178 on 2026-10-24's in-memory DATA (r2/amd/compare-2026-10-24-full-final.md:58), which is +32%.
• read_4k_at, stat, rename and seq_read.1048576 do agree within about 5%. r2's index was careful to name only read_4k_at, stat and rename.
• The title's "ten times" holds for read_4k_at (12.8x) and rename (9.7x), but not for stat (4.0x) or open_close (3.1x).
• Fix: restrict the sentence to the rows that agree and state the open_close difference as unattributed. Make the title match the table, for example "several times", or the commit message's "4 to 13 times".

BLOCKER N3 The evidence branch pr-1097, file r3/probe/probe-clean-limit.patch: the published patch is not the probe that ran.

• It is 10 lines long and adds only a blank line, so it cannot define probe_sequential_passes_against_the_clean_limit, the test that probe.log reports.
• The real patch, 31 lines, is still in the implementer's scratch mutations/probe-clean-limit.patch. It looks like the publishing step lost lines.
• The seq_read record cites this probe as the confirmation of its cause, so the published evidence has to contain its source. Republish the full patch and repoint the index.

BLOCKER N4 (orchestrator's correction) The PR body no longer matches the change or its evidence.

• It is still labelled Draft.
• "What the validations show" is the c8f3fbf table, and its T14 write_4k_fsync cell ("its Flush on that BC501 costs about 3 ms") is contradicted by r4.
• It names a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs, which d6fc8be deletes.
• Checks still say B5's runs are "staged … (exit 2)", and that "069cbbeb8 adds only an issue record", with d6fc8be unmentioned.
• The evidence link is pinned at 376e93d4, which has no r4. r4 is 9369068d on pr-1097.
• "The same rows on 2026-10-24's in-memory DATA read the same" carries N2's problem.
• "Linux's issues neither" carries N1's problem.
• The risk line's exception list is out of date: at r4 the AMD's append_4k_fsync is also below Linux's.
• Needed: r4's table and judge results, the evidence link at the r4 commit, and the current record names.

SUGGESTION tests/common/hostdisk.rs:107-121, through data_on_the_nvme: r1's suggestion assumed that left_alone checks the NVMe opened a session on DATA. Under #1095's NVMe boot it does not: the loader's boot and log sessions already satisfy "at least one session, each on DATA or ours". As a result, Mounted no longer requires DATA to be the NVMe's. Today this cannot misjudge: the image has no DATA, both benches' NVMe carry a TOYOS-DATA partition, and the file server refuses two. If a session on DATA is what the row means, require it when the fate is Mounted or Formatted. That is one any over opened.

SUGGESTION issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md:13-24: the /tmp column comes from the 2026-10-24 boot, but the c8f3fbf boot measured fs.mem itself (1,413 / 2,596 / 6,091 / 6,242 ns, 8,041 MB/s). Using the same boot for both columns would remove the "that boot's" sentence.

Verdict

SEND BACK. N1–N3 are wording or republishing fixes that need no new runs. N4 belongs to the orchestrator.

[Japabu]

Orchestrator: r2's N1–N4 checked against d6fc8be..ddc26a3 (issue records only, so the landing-head runs at 069cbbe still apply). N1: the T14 record states the measured 23 µs and drops the format-age candidate. N2: the record is renamed to three to twelve times, with both columns from one boot and open_close's +32% stated. N3: the full probe patch is republished at 38db2e75 with a correction note. N4: the body describes the final change and pins r4. The new create_fsync_unlink record states only its measured rows. Landing.