[Japabu]

Draft. Eight commits on main c1218e8 (after #1104): the osbench row (the A/B's base), 7a, review r1, review r2, the port of #1120 across the reserve/write-back split, review r3, the paged path's hole test and review r4. The metal A/B at r2 is in and judged (below); the rebase onto main and r3 are verified on the host and QEMU, and the metal testcases boots at 8153b4b passed on both benches; a testcases boot at the r4 head is staged.

Problem

A cached read of a file on DATA cost a round trip to its file server: 9.5–11 µs for read_4k_at against the kernel /tmp's 1.2–1.4 µs and Linux's 0.5–1.1 µs (fs-rtt r1, r2). The file server's block cache held file data in the server's own memory, so only the server could answer.

Change: memory design §7, stage 7a

The owner approved this stage plan, and two design reviews converged on revision 3. A file's pages live in a kernel paged object that its file server supplies. A reader holds a handle to the object and reads it through SYS_PAGED_READ without asking the server. 7a builds no user mappings; those come in 7b.

Kernel (kernel/pure/paged.rs, host-tested; kernel/src/object/paged.rs; kernel/src/syscall/paged.rs):

• Two kinds. Pager is a process's one pager, made only by a process started as one. It neither duplicates nor transfers, and its last handle ends every object it serves. A pager names its objects by id: every owner call takes the pager handle and the id SYS_PAGED_CREATE answered, so a server holds one handle however many files it keeps. An object goes by PagedOp::Release. Paged is a holder of one object, which can be narrowed and sent; each counts against its client's handle table, as any open file does.
• Ids are table slots, free again once an object's last holder goes, so no counter can wrap. A release checks, invalidates and clears its object in one hold of the lock, as the pager's end does, so a second release, or one racing the end, finds it released.
• A holder's read resolves its handle by kind: a handle a peer sent as a holder and is not one is refused InvalidArgument, never a fault of the reader. Every offset, length and size a pager names is checked against the last page a u64 can end at before the object sees it.
• Supply copies into fresh frames. A write runs under write pins, outside the lock. A frame leaves the object at withdrawal, failure, shrink or the object's end: it comes out of the table under the pager's lock and is freed at its last pin.
• A holder's read waits in its own request record, linked on its pager. The record allocates nothing. Only a supply, a failure, a shrink, the pager's end or a kill ends the wait. Past the pager's report time the read is counted and logged overdue, and it keeps waiting. A pager's own process is refused a holder's read.
• The kernel holds the one size. A write or dirty supply grows it in the same hold that makes the bytes visible: a write pin grows it at its unpin, after the bytes land. A clean supply never grows it.
• The index is one open-addressed table of the pager's capacity, keyed by (object, page) with backward-shift deletion, plus an intrusive list per object. It is made with the pager, and so is the object table, so no call on it allocates. Its cost scales with the capacity, never with an object's length. A radix tree per object would have a worst case on sparse objects that this table does not. This is an implementation choice within §7.5's rule.
• The tables are in pieces of 1 MiB less 8 KiB (kernel/pure/chunked.rs): two share a superframe, and the kernel heap panics on one allocation past 2 MiB less 4 KiB, which a capacity of an eighth of memory asked for (found by the linkcase run). A list of pieces past one piece is refused.
• No removal grows a list under the lock: withdrawal, failure, shrink, release and the pager's end give each frame back as the index answers it. Past 512 MiB in one object a list of frames asked the heap for 2 MiB and panicked the kernel (a guest control shows it).
• Write-back ranges are one table of 64 per pager, not eight per object, so an object's entry is small; DATA keeps an object for every 16 pages of its capacity, between 1,024 and 65,536.
• Notices are level state (requested, no_holders) read with SYS_PAGED_NOTICES. The pager handle reads ready while one is owed. A no_holders notice is given only while it is still true.

Authority:

• Rights::PAGER on a SysCap named by SpawnArgs.pager starts a child as a pager.
• The supervisor holds that right and confers it on a row the signed system.toml marks pager = true: the file servers' rows in all 16 configs, which the build now requires of a row with roles, and testcases' pagerprobe.
• No syscap name maps to it, and a package row is never a pager (both tested in lib/manifest).
• ROOT's server, whose files are immutable and belong to stage 4's sealed objects (memory design §6), has no pager. That decision was accepted.

File server (system/fileserver/src/pages.rs, host-tested against a model of the kernel object):

• One object per file, keyed by its volume identity, for DATA, LOG and BOOT. The object holds the volume's node for its life, so a dirty file stays open on the volume. It outlives the file's last close until room is wanted.
• An open to read of a regular file is answered with a holder.
• A reader's request is read off the volume and supplied, at most 512 pages per notice; a range the volume will not read, or holds none of, is failed, so no reader waits for ever.
• A write reserves room on the volume (allocates and sizes it, without writing the bytes), then lands dirty in the object. A partial page is merged with its old bytes, which the volume is read for only below Reserved::contents: the old size, or the end of the pages the file held blocks for where that comes first. A page in a hole reads zeros, never the block its reserve just took. A write refused part way is cut back to what it put, and the room its reserve took past that is given back (Volume::unreserve; below).
• A sync writes the dirty pages back (writeback_begin, owner read, Volume::write_back, writeback_end), then runs the volume's own sync.
• A truncation changes the volume first and the object after, so a refused one drops no dirty page; so does a truncating open. Nothing writes back between the two.
• Unlink and rename-over drop the target's object. Its holders then read Gone, and the SDK falls back to the server, which answers Stale as before.
• When the pager has no room, it drops a file nobody holds, oldest first. Failing that it withdraws another file's clean pages, then this file's pages outside the requested range, then writes back. A create refused because the pager serves as many objects as it was made for also drops the oldest file nobody holds, and is refused NoSpace only when every one is held open.
• The block cache keeps only the volumes' structures. visit is deleted. Data passes it by through BlockDev::read_data/write_data, which meet a block the cache holds in the cache. A data write outside a flush is counted, so the next flush still asks the disk to keep it: the old unflushed flag would have lost one that landed during a flush. The limits shrink to 16 MiB clean and 8 MiB dirty. toyos-fat32's write splits into reserve and write_back, and its data I/O uses the data calls.
• Pager sizes, as fileserver: a volume's cache keeps an eighth of physical memory clean #1099 sized the cache: together an eighth of physical memory, LOG and BOOT 16 MiB each (512 and 0 dirty), DATA the rest with 32 MiB dirty, and 65,536 objects for DATA, 1,024 for the others.

SDK:

• toyos::paged wraps the calls with typed refusals.
• Dir::open carries the holder, which the SDK takes as one only once a read of nothing through it answers as a holder's does; anything else is refused Malformed, and a holder this process may not read, or whose object ended, is read through the server.
• A served descriptor keeps its holder. A read at its local offset, or a pread, goes through the holder within the one descriptor lookup a read makes, without taking the capability's turn, so a reader waiting on a page holds up no other thread's request. Where the holder cannot answer, the server does.

Charging (the orchestrator's ruling R6): the pager's core is committed at once and kept for its life, resident by kind: the dirty ceiling and up to 512 pages of page-in headroom. kernel::paged::Core counts it; a frame leaving while the core would hold less is held aside for the next supply, never given back and promised again, so the allocator cannot take the core under pressure. Clean pages above the core are promised one at a time as they are supplied. The index and object table are charged with the pager, and a capacity the job cannot be charged for is refused. The capacity itself is the server's; the eighth of memory is the interim bound, recorded with its exit.

Records:

• The swap track's amendment: a pager's core is resident by kind, the file servers get a resident row for their cores alone, and the prerequisites now say 7a is built.
• Ownership: 7a was delivered here, and 7b stays with the process-memory track. Recorded in process-memory-in-4-kib-pages-still-copies-each-programs-text.md and storage-is-layers-and-a-role-is-a-filesystem.md.
• a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md is updated: its exit is now the server's eviction over its objects, and it stays open.
• New issues:
  • a-pager-holds-a-fixed-capacity-until-the-memory-service-reclaims-its-clean-pages.md (the capacity no other job can take back, and its exit);
  • a-read-and-a-write-of-a-paged-file-can-tear-across-pages.md (the XSH 2.9.7 gap);
  • an-overdue-paged-read-is-no-wait-cause-in-the-roster.md.

Rebased onto #1108 and #1120:

• Wakeups are kernel events, and a one-handle timed wait is SYS_HANDLE_WAIT #1108 took object kinds 17–18 and syscalls 148–151, and clock: the TSC keeps CPUID 15H's rate or one the HPET refines; a step the RTC confirms is slewed, a second demotes it #1104 took 152 (SYS_CLOCK_NOW), so Pager and Paged are 19 and 20 and the paged calls 153–158 (SYSCALL_PROFILE_BINS 160; the profile's compile guard names SYS_PAGED_NOTICES). That renumbering is the only change from the rebase onto clock: the TSC keeps CPUID 15H's rate or one the HPET refines; a step the RTC confirms is slewed, a second demotes it #1104, so the metal boots at 0992ed8 still apply. The cache's flush is blockring: the arena holds only what is on the wire, and a flush keeps many writes there #1113's parallel one; its runs count into the same written mark data writes do.
• fileserver: a write refused after allocating gives back the blocks it took #1120 made a refused DATA write give back the blocks it allocated. Here the allocation is reserve and the bytes come after, so its rule now crosses that split: reserve answers a Reserved (the size and the pages the file held blocks for), and Volume::unreserve(node, reserved, landed) returns a refused write to that size, or to where its bytes landed, and gives back the blocks past both. The default cuts back to the size, which is enough without holes (FAT). DATA keeps blocks only up to what the file held, because a write into a hole below the old end takes a block that a cut to the old size would keep. Files::write and Writes::write call it. It records no entry, because only the file's own calls write its entry and none of them runs inside a write.

A new osbench row, reopen_read_4k (open, read 4 KiB, close), is its own first commit, so the base measures it the same way.

Checks

At 82237ea (rebased onto #1104) (evidence): cargo run -- --ci host exit 0, 108 steps green.

At 0992ed8 (review r4) (evidence):

• cargo run -- --ci host: exit 0, 107 steps green.
• cargo test -p fileserver: 107 passed.
• N1: data::tests::a_write_in_part_into_a_hole_reads_zeros_around_it and pages::tests::a_write_in_part_into_a_hole_supplies_zeros_around_it both run on an image whose every block holds 0xAB. Each writes one byte at 6000 into a hole that a truncate up left, and the page then reads zeros around that byte: through a holder and again after write-back. Each fails when its path reads below the old size again (controls D and E, rc=101: "page 1 reads 0xab", "a holder reads another file's bytes").
• The paged path's refused write into a hole (683c56c): pages::tests::a_write_refused_in_a_hole_gives_back_the_block_it_took. The room is unchanged and the hole reads zeros. It fails when Files::write cuts back with truncate again (control C).
• Not taken, review r4 S1: a write refused before any byte landed keeps the time and the dirty mark its reserve set. POSIX says nothing about a failed write's times, and putting the old time back would add a field to every format's Reserved.
• Owed: a testcases boot on the T14 and the AMD at the r4 head (request).

At 8153b4b (rebased, r3) (evidence):

• cargo run -- --ci host: exit 0, 107 steps green.
• Host tests: cargo test -p fileserver 104 passed. fileserver: a write refused after allocating gives back the blocks it took #1120's a_write_refused_after_allocating_leaves_no_old_bytes_past_the_end is back, now through Writes::write, beside a_write_refused_in_a_hole_leaves_the_hole_reading_zeros.
• Negative controls for the port, each applied as a patch, run and restored: with DATA's unreserve deleted, so the default cut to the size runs, both tests fail (the hole's block is kept); with Writes::write giving nothing back, both fail.
• QEMU x86-64, testcases shared boot through --debug: a_pager_s_end_answers_its_waiting_holders_gone, a_served_file_reads_through_its_paged_object, served_file_offset, std_fs and file_mtime, each exit 0, and pagerprobe: ok in the boot log.
• r3's B1 (an owner read racing a release) is closed by construction: the read takes a reference in the same hold that resolves the id, and a Ref guard gives it back. Like a holder, it can then never meet an empty slot. No test drives two pager threads, because today's servers are single-threaded.

Metal A/B at r2 (base 28a8aac, head dd88215 on main 49559dd; T14 and AMD, base, head, base, head; every boot passed; p50 and per-boot results). Judged as at r1: a row counts as moved only where both heads lie outside both bases.

row, p50	T14 base → head	AMD base → head
fs.disk.read_4k	9.48, 9.48 → 0.98, 1.01 µs	9.02, 10.46 → 1.20, 1.13 µs
fs.disk.pread_4k	9.47, 9.45 → 0.96, 0.98 µs	9.00, 10.52 → 1.23, 1.16 µs
fs.disk.read_4k_at	9.89, 9.95 → 1.30, 1.33 µs	9.31, 10.91 → 1.74, 1.67 µs
fs.disk.reopen_read_4k	31.9, 31.3 → 16.6, 16.9 µs	29.2, 34.8 → 18.4, 14.9 µs
fs.disk.open_close	22.3, 21.8 → 15.8, 16.2 µs	19.8, 23.4 → 17.4, 14.1 µs
fs.disk.seq_read.4096	435, 431 → 3640, 3649 MB/s	463, 395 → 5448, 5409 MB/s
fs.disk.readdir.1000	910, 906 → 900, 891 µs	691, 677 → 693, 682 µs
fs.mem.read_4k	850, 851 → 811, 817 ns	861, 861 → 902, 892 ns
fs.mem.read_4k_at	1227, 1225 → 1189, 1225 ns	1392, 1402 → 1463, 1463 ns
fs.mem.seq_read.4096	4764, 5040 → 4626, 4603 MB/s	5351, 5350 → 5173, 5135 MB/s

• The r1 wins hold. A cached served read falls from about 9.5 µs to about 1 µs. seq_read.4096 rises from 430 to 3,640 MB/s on the T14 and from 463 to 5,450 MB/s on the AMD. open_close and reopen_read_4k fall too, since a retained object keeps its node.
• Both r1 regressions are gone. The T14's fs.mem.read_4k, +12% at r1, is now 4% below base. AMD fs.disk.readdir.1000, +28% at r1, now lies within its bases (691/677 → 693/682 µs).
• A small fs.mem difference remains, measured and unexplained. These reads never reach a server. fs.mem.seq_read.4096 reads 3–9% below base on both benches. On the AMD, fs.mem.read_4k and read_4k_at read +4–5%, while on the T14 they read the same as base or lower. This is far inside the bug guard and may be code layout. r2 already removed the second descriptor lookup.
• The AMD's base-3 boot is slow throughout, and so is its head-2 boot. Both read 10–17% above the other boot of their kind on rows the change does not touch (stat, fstat, the direct rows). The fs.mem rows show no such spread. The AMD's fs.disk deltas above are therefore ranges and not points.
• The head testcases boot passed on both benches. a_served_file_reads_through_its_paged_object, served_file_offset and a_pager_s_end_answers_its_waiting_holders_gone passed; so did the T14's proctreecase boot, including app_view (exit 0).

Earlier, at dd88215 on 49559dd (evidence):

• There were eleven host controls, all red, plus two guest controls: a removal collecting its frames in a list panics the kernel at 2,097,152 bytes, and a holder's read resolving by type kills the helper.
• pagerprobe covered the >512 MiB object, the double release and the pipe end as a holder.
• These still apply: the rebase changed none of that code, and r3 touches only the owner read.
• The linkcase test passed at r1.
• Metal testcases at 8153b4b (readbacks): one boot each on the T14 and the AMD, both built from 8153b4b. a_served_file_reads_through_its_paged_object, served_file_offset and a_pager_s_end_answers_its_waiting_holders_gone passed on both, judged offline from the readbacks with --metal-readback (rc=0, 3 passed). The benchcase A/B was not rerun on the new main.

Unresolved and risks

• Not built: the roster's wait cause (filed). A restarted file server serving fresh opens is covered by the existing restart paths, not by a new test.
• A file larger than the pager's capacity is read off the disk again on every pass, because eviction drops the read file's own pages outside the range. The sequential-read issue stays open on this.
• The DATA index is large on a large machine: an eighth of 32 GiB is a million pages, about 60 MiB of index charged to the DATA server's job (an estimate). It is refused, not aborted, where the job cannot hold it.
• The holder read and the server's write are visible at once but not atomic across pages (filed).
• A pager's end keeps its index until the last stale holder closes (review S5, not taken).
• R4's undo of a refused open's count is in main.rs, which no host test reaches; Files::closed, which it calls, is tested.
• The notices scan still walks the whole request list and object table per call (review suggestion, not taken).
• A pager's end and a whole-object withdraw or release free frame by frame under the pager's lock (review r3 S3, not taken). The work is bounded and charged to its cause. Batching it belongs with 7b's revisit of the pager end.
• Two pager threads are not exercised by any test. The owner read's reference makes the race unrepresentable, not tested.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1123 at ae980c4 (memory stage 7a)

I read both commits, the whole diff against the merge base 49559dd, and every changed file in full. I also read the callers and boundaries: pmm/frames promise and free, watch::wait_until, handle-table limits, the spawn commit, the fileserver open/close/sync paths, and the image configs. I checked the work against design rev 3 (R1–R7, S4), round 2 of the review, and memory design §7. I ran nothing; every point below comes from reading the code and the evidence.

Architectural assessment

The split of responsibilities is right. The kernel owns the frames, the index, the one size, the waits and revocation; the server owns policy, write-back and eviction. Pager, owner and holder are separate KObjectRef kinds. Every map, lend and spawn-image path takes handles.get::<SharedMemObject>, so each one refuses the paged kinds by type, and the exhaustive ops arms are updated. The settled points hold:

• waits end only on supply, failure, shrink, the pager's end or a kill;
• supply copies;
• a write grows the size at its unpin, under the lock;
• write pins;
• the table index.

Lock order also holds:

• every user copy runs under pins with no lock held (read, supply, write, notices, writeback);
• the only things done under the pager lock are pmm and watch.post, and nothing takes the pager lock under either;
• wait_until arms before it checks ready, so the wake that follows gone = true cannot be lost;
• Waiting's drop unlinks the request under the lock, and the Arc in each holder keeps the pager alive while a waiter is linked.

R2 holds by inspection. sys_paged_read is the only way a paged read can wait, and its guard KObjectRef::Paged(_) if data.pager.is_pager() refuses every pager process. A host test of that one guard would need machinery built only for the test, so I don't think one is needed. The guest test asked for in B5 below would check it from outside at no extra cost.

There is one problem with the representation. Each file costs a PagedOwner handle in the server's own handle table. That ties the number of files the server keeps, or has open, to a 4096-slot per-process limit (B3).

Size of the change: about +3.25k production lines (kernel ≈1.3k, fileserver ≈0.7k, SDK/ABI ≈0.5k) and ≈0.86k test lines, counting inline test modules. Deleted: visit, WindowOut, data_clean_limit and CLEAN_SHARE. #1099's re-read test moved to pages/tests.rs.

Blockers

B1. A pager can panic the kernel with a supply offset.

• Where: kernel/src/object/paged.rs:503,514, kernel/pure/paged.rs:297, kernel/src/syscall/paged.rs:117-122.
• Problem: the only check on offset is its alignment. For offset = 2^64 − 4096 with one page, ends[k] = page_at + 4096 overflows, and [profile.toyos] builds the kernel with overflow-checks = true. Pure supply's (page + 1) * PAGE overflows the same way.
• Consequence: a userland bug panics the kernel.
• Fix: refuse offset.checked_add(len) past u64::MAX − PAGE (or past the largest page index) at sys_paged_op. Add a unit test at the boundary.

B2. The object-id counter wraps into a kernel panic.

• Where: kernel/pure/paged.rs:177 (checked_add(1).expect("a pager's object ids wrapped")).
• Problem: every paged_create uses a new id, so every open of a file whose object was dropped (unlink, rename-over, eviction) spends one. At 1,000 such creates per second the counter runs out in about 50 days of one server's life.
• Consequence: a kernel panic driven by an ordinary workload.
• Fix: you need no counter. The entry slot is already unique among live objects, because release removes every page before the slot is reused. Key the table by the slot.

B3. The file server runs out of handles after about 4,000 distinct files.

• Where: system/fileserver/src/pages.rs:176-195,115-117,301-321; sdk/abi/src/handle.rs:37 (MAX_SLOTS = 4096); descriptions.rs:24 (MAX_IDS = 1024 per client).
• Problem: opened makes one PagedOwner handle per identity, and only make_room drops one. make_room runs only on NoRoom, which is page pressure. Files with few or no pages never cause it: 4,096 two-page files are 8k of 24k pages. Once the server's table is full, pager.create is refused, opened fails and the open answers Io. A build or a grep -r over a source tree reaches this.
• A second, separate cap: four clients with 1,024 open distinct files each would also exhaust the table, a limit main does not have.
• Fix: bound the retained objects by count. A refused create should make room by dropping a droppable file and retrying, with a host test. The cap on concurrently open files needs either a decision from the orchestrator or a recorded owner and exit. Examples: address owner calls by (pager, object index) instead of one handle per object, or give a pager a larger table.

B4. tests/images/linkcase.toml:42-45 has no pager = true.

• Problem: the image is in the base and in build.rs's config list, but its fileserver row was not updated. pager(role) panics without the role (system/fileserver/src/main.rs:288), so DATA, LOG and BOOT each panic and restart in that image. a_link_ends_under_a_held_stream_and_comes_back, which asserts must_be_clean(), goes red.
• Also: the PR's "all 15 configs" is wrong; there are 16 fileserver rows.
• Fix: add the key, and run that QEMU test.
• Suggestion: derive pager from a row's roles so this mistake cannot happen again.

B5. The pager's core is not held as resident (R6, swap-track amendment).

• Where: give_back (kernel/src/object/paged.rs:223-234).
• Problem: it calls pmm::free_drawn, which uncommits the frame (kernel/pure/frames.rs:771-778). It then calls pmm::promise(short) again in a separate allocator hold, and that call can be refused under pressure or to cover another job's floor. Once refused, the pool stays short. Dirty supplies then need promise(1) per page; when that fails the server reads NoRoom, withdraws, and the re-promise fails again.
• Consequence: the "dirty ceiling and headroom a read's supply can always draw" leak to other jobs exactly when memory is short. The amendment and R6 call that core resident by kind.
• Fix: keep the commitment when a removal would take drawn + pool below the core. A pmm call that frees a frame and re-promises it in one allocator hold, as Frames::discard does for a page that stays committed, would do it. Add a host test that a full allocator does not take the core away.

B6. The 96 MiB interim's justification is out of date, and replacing #1099's sizing goes back to the orchestrator.

• Where: system/fileserver/src/main.rs:281-285, sdk/abi/src/syscall.rs:3131.
• Problem: R6 capped C at "today's CLEAN_LIMIT 64 MiB plus DIRTY_LIMIT 32 MiB, so it causes no RAM regression". That was ruled on 2026-10-29. fileserver: a volume's cache keeps an eighth of physical memory clean #1099, the base of this PR, then made DATA's clean cache an eighth of RAM.
• Consequence: under this PR's own charging, clean pages above the core are promised one page at a time, and capacity beyond the core costs only about 60 B of charged table per page. So the 96 MiB cap does not buy the "no RAM regression" the ruling was about. It does reintroduce fileserver: a volume's cache keeps an eighth of physical memory clean #1099's cliff for files between 96 MiB and RAM/8 on every larger machine, the T14 and the AMD among them. PAGER_MAX_PAGES also fixes this policy in the kernel ABI.
• Fix: either size DATA as fileserver: a volume's cache keeps an eighth of physical memory clean #1099 did, with the kernel bounding capacity by the charged table rather than a constant, or get the orchestrator to re-rule and accept the regression explicitly.
• Record: issues/a-pager-holds-a-fixed-capacity-…md:13-14 says 96 MiB "is the clean and dirty limits its block cache held before". That is false since fileserver: a volume's cache keeps an eighth of physical memory clean #1099.

B7. The SDK wraps an unchecked handle as a Paged holder.

• Where: sdk/toyos/src/files/owner.rs:427, sdk/toyos/src/paged.rs:179-181.
• Problem: opened.holder.map(Paged::received) wraps whatever handle the server's reply carried as Paged. The wrapper alone says nothing about what the object is.
• Fix: check the handle once, where it is received. A zero-length paged_read answers 0 for a holder and NotSupported for anything else; alternatively, have fstat give holders a type of their own. Refuse anything else as Malformed, as opened already does for extra handles.

B8. New private page-size copies.

• Where: kernel/pure/paged.rs:28 (kernel/pure/vm.rs already has PAGE), sdk/toyos/src/paged.rs:20 (toyos_abi::syscall::PAGE_SIZE exists) and system/fileserver/src/volume.rs:194.
• Fix: use the shared declarations.

B9. Evidence gaps for material claims.

• The metal A/B and the head testcases boot have not run. That was already the plan; they are needed before landing.

• app_view's declared target is the proctreecase metal row. It exercises exactly the changed paths:

  • holders on read-only package opens;
  • writes in the app's folder through Files::write;
  • refused truncate and create;
  • gbae's save.

  ab/request.txt does not include it. Add one proctreecase boot per bench to the same metal session. --debug cannot run it, so as it stands nothing verifies it.

• Owner death: the design's §9 guest test is missing: a parked holder must answer Gone when its pager ends, and a restarted pager must serve fresh opens. The kernel path, from on_zero_handles through the wake to Waiting's unlink and PagerObject::drop's assert, is what host tests cannot reach. A defect there hangs readers or panics the kernel only when a server dies. My reading finds it correct, but inspection is not evidence for the main failure path of a new kernel object.

  • Cheapest sufficient check: a small member declared in testcases.toml with pager = true and started through test-runner's launcher. A launch takes its row's role. The member makes a pager, an object and a holder, and gives the holder to a non-pager helper that parks on it. It then asserts its own holder read is PermissionDenied (R2's negative check) and exits. The helper must answer Gone.

• The linkcase QEMU test after the B4 fix.

Suggestions

• S1 (kernel/src/object/paged.rs:354,407-421): began is when the read started, not when the wait started. A long cold read, say a large read from LOG's stick, is reported overdue even when each supply was prompt, and stays counted until the whole read ends. Reset the deadline at each wait, and drop Overdue when that wait ends.
• S2 (kernel/pure/paged.rs:654): if the withdrawal-walk control regresses, the test hangs instead of failing, and CI hangs until its job times out. Assert the number of probes, for example with a counting closure or a returned visit count.
• S3 (system/fileserver/src/pages.rs:166-171,423-436): a refused shrink leaves the object shrunk. Dirty pages are discarded and holders read the new size while the client hears an error. The same happens if the truncating open's volume.open fails after truncating. The server is single-threaded and write-back happens only in Files, so doing the volume first and the object second gives the same safety and a refusal that changes nothing.
• S4 (kernel/pure/paged.rs): slots_held, holds_slot() and dirty_slots() count against the dirty ceiling, but "slot" also names the table's slots in the same module. Rename them, for example dirty_held and counts_dirty. Also "its page table" (kernel/src/object/paged.rs:5-6, swap issue) will read as an MMU table once 7b adds real ones; "its index" avoids that.
• S5: on_zero_handles keeps the whole Pages table (≈1.5 MiB for DATA) until the last stale holder closes, which can be never after a restart. Replace it with an empty table at the pager's end.
• S6 (pages.rs:505-520): a Requested notice the server neither fully supplies nor fails, for example when fill covers 0 pages because the volume's size is behind the object's, leaves the reader waiting forever. Fail the requested range's first page if it is still absent after fill.
• S7: Writes (volume.rs:189-212) also serves forgotten files on pager servers, so its doc is too narrow. A trait with a blanket impl could be a plain function.
• S8 (for the PR body): "nothing it does allocates" is true for supply, read, write and write-back. withdraw, fail, resize, release and the pager's end collect frames into transient Vecs under the lock. That is covered by the call-scope rule in mm/charge.rs, but the claim should say so.
• S9: the evidence branch's history (efe0c13e) still contains the AMD bench's SSH alias that 7ccd7087 masked. Whether to purge it is the orchestrator's call.

Answers to the brief's checks

• Locking and memory ordering: sound; see the assessment.
• Kernel copies never wait holding a lock: confirmed.
• Revocation on owner death: correct by inspection, but untested in the guest (B9).
• Refusal by kind: holds on every map and lend path, by type.
• R2: enforced. A host test is not required; the B9 member checks it.
• Charging: the core is not retained under pressure (B5). The C bound is B6.
• File-server crash: waiters answer Gone, frames go back, the SDK falls back, and dirty data is lost as accepted. A stale table lingers (S5).
• fileserver: a volume's cache keeps an eighth of physical memory clean #1099 regression: real for files between 96 MiB and RAM/8, and the justification is out of date (B6).
• The 9 controls: 8 fail cleanly. The withdrawal control fails only by hanging (S2).
• app_view: not verified, and required (B9).

SEND BACK

[Japabu]

Review r3 of #1123 at dd88215 (focused re-review)

Scope: I checked r1's B1–B9 and r2's R1–R5 at dd88215. I also read the two fix commits (265581d, dd88215) in full against the files they touch: kernel/src/object/paged.rs, kernel/pure/{paged,chunked}.rs, kernel/src/syscall/paged.rs, sdk/toyos/src/{paged.rs,files/owner.rs,fs.rs}, system/fileserver/src/{pages.rs,main.rs,data.rs}, pagerprobe, build.rs and the issue records. I ran nothing. I did not reopen the decided points: B3 addresses objects by id, and B6 commits at most 1/8 of RAM across pagers.

Architectural assessment

The fixes keep the r1 split of responsibilities. The kernel owns the frames, the index, the waits and revocation. The server owns eviction and write-back.

Addressing objects by id removes the per-file server handle without adding a mechanism. Slot ids remove the counter. Core is a small, host-tested account that keeps the core resident. R3 removes the frame lists rather than bounding them.

One consequence of addressing by id was missed. An owner call no longer holds a reference on its object, so any owner call that drops the pager lock and comes back must resolve its object again. supply and write do this. owner_read does not (B1 below).

Earlier findings at dd88215

• B1 (offset overflow): fixed. within_reach gates every byte range in sys_paged_op, and add/resize check LAST_END. Page ranges are clipped or checked (withdraw saturates, fail clips to page_end, write-back uses checked_add). The boundary has a unit test.
• B2 (id counter): fixed. An id is its slot. The slot is pushed free only in unref_locked, after pages.release.
• B3 (handles per file): fixed as decided. The server holds one pager handle. A refused create calls drop_oldest and retries. The object cap is recorded in the capacity issue.
• B4 (linkcase): fixed, and file_servers_are_pagers gates every config, with a test.
• B5 (core kept): fixed. Core::leave keeps a frame aside while held() < core. spare/draining are sized to the core and pre-allocated, and settle moves a frame only once it is unpinned. The accounting balances on every path I traced: supply Present/refused, pager end with a supply in flight (the second lock's live fails, give_back(gone) frees), and Drop after end(). a_core_is_kept_whatever_the_allocator_refuses covers it.
• B6 (sizing): fixed as decided. main.rs:291-303 splits an eighth of RAM. The kernel bounds capacity by the charge new_core(tables) takes. The issue text no longer claims "no regression".
• B7 (unchecked holder): fixed. Paged::received probes with a zero-length read, and owner.rs:444-453 refuses Other as Malformed and closes the fid.
• B8 (page constants): fixed. pure/paged.rs re-exports vm::PAGE, and the SDK uses syscall::PAGE_SIZE. volume.rs:195 aliases toyos_blockdev::BLOCK, a shared declaration.
• B9 (evidence): run, not yet in the PR. ab2/runs holds the dd88215 images: the supervisor log reads build dd88215… clean. Benchcase base/head/base/head on the T14 and the AMD, head testcases on both, and proctreecase on the T14 all read passed. app_view reads exit=0. a_pager_s_end_answers_its_waiting_holders_gone, a_served_file_reads_through_its_paged_object and served_file_offset pass on both machines. Both r1 regressions are gone, by p50 (base-1/head-2/base-3/head-4):
  • T14 fs.disk.readdir.1000: 910/900/906/891 µs.
  • AMD fs.disk.readdir.1000: 691/693/677/682 µs.
  • T14 fs.mem.read_4k: 850/811/851/817 ns.
• R1 (holder read by kind): fixed. sys_paged_read matches KObjectRef::Paged and answers InvalidArgument otherwise. pagerprobe's pipe check covers it, with a guest control.
• R2 (release in one hold): fixed (release → let_go under one lock). A second release fails live on owned.
• R3 (no frame lists): fixed. Every removal calls give_back per frame. The guest control reproduces the 2 MiB panic.
• R4 (refused-open undo): correct. It is unreachable today, because open pre-checks fids.len() >= MAX_IDS and Ids::insert has no other refusal, so the missing host test is no gap.
• R5 (clipped ranges): done.

Blockers

B1. A pager's owner read that races its own release panics the kernel.

• Where: kernel/src/object/paged.rs:430-436 with :449, :452, :475 and :159-161.
• Problem: owner_read resolves live(id) → slot, then drops the lock. read then calls state.entry(slot) under fresh holds: at the start of every chunk, and again after every copy, which runs unlocked. entry expects the slot to be occupied.
• Since r1, an owner call holds no reference on its object. So a PagedOp::Release from another thread of the same pager, on an object with no holder objects, takes entries[slot] to None inside that window (let_go → unref_locked, refs 1→0).
• Failure: thread A runs a write-back's PagedOp::Read of 2 MiB, and thread B releases the object (an unlink). The next state.entry(slot) panics with "paged: a slot an object names is held", and the kernel goes down.
• Why other paths are safe:
  • The pager-end path is safe only because state.gone || short-circuits first.
  • supply and write are safe because their second hold re-runs live(id).
  • Holder reads are safe because Named holds a reference.
• Today's file server is single-threaded, so the image does not hit this. Still, the kernel cannot rely on a pager's threading, and a userland bug must not crash the kernel.
• Fix: have the owner read hold a reference for its duration, as a holder does: refs += 1 in the same hold as live, released by a guard through unref. This also stops the slot being reused under the read. Alternatively, re-run live(id) in each hold and answer Gone.
• A host test is not needed if the reference is a guard type the read path takes. Stating the invariant in read's contract ("slot is held by the caller for the call") makes the requirement visible.

B2. A module header is false since R3.

• Where: kernel/src/object/paged.rs:33-35.
• Problem: "…so no call on it allocates but the transient lists a removal collects." R3 deleted those lists: every removal now gives each frame back as the index answers it, and the PR body says so. The sentence is false.
• Fix: end it at "no call on it allocates".

B3. The file-server page module's header contradicts the code since r1's S3.

• Where: system/fileserver/src/pages.rs:18-19.
• Problem: "A truncation shrinks the object before the volume frees the blocks its dirty pages would have gone to." Files::truncate (:451-463) and the truncating open in main.rs now do the volume first and the object after, and their own docs say so.
• Fix: delete the sentence. truncate's doc already states the ordering.

B4. An unsafe precondition comment is false since on_fd_held.

• Where: sdk/toyos/src/files/owner.rs:515-516 and :546-547.
• Problem: "only one of the closures runs". With on_fd_held, holding runs, and when it answers None, served runs too, each calling out() to build a &mut [u8] over the same caller buffer.
• This is still sound, because the first slice is dead before the second is made, but the SAFETY argument states something that is not true.
• Fix: say that the closures that run, run one after another, so no two of the slices live at once.

Suggestions

• S1 (PR body, for the orchestrator):

  • "Owed at dd88215" is now done: ab2/runs, all passed. The table can take the r2 head.
  • The risk "open_close will cost more" contradicts the measured fall (T14 22.3/21.8 → 15.8/16.2 µs at r2). Delete it.

• S2: a small fs.mem difference remains at r2. These reads never reach a server:

  • fs.mem.seq_read.4096 reads 3–9% below base on both benches (T14 4764/4626/5040/4603, AMD 5351/5173/5350/5135 MB/s).
  • AMD fs.mem.read_4k reads +4% (861/902/861/892 ns).

  It is far inside the bug guard, and may be layout. A sentence in the PR stating it as measured and unexplained would keep the record honest.

• S3: on_zero_handles and a whole-object withdraw/release free each frame through pmm::free_drawn (one allocator hold per frame) under the pager's lock. For DATA's capacity on a large machine that means up to about a million frames in one hold, which every waiting reader spins behind. The work is bounded and charged to its cause, so this is not a blocker. If the 7b work revisits the pager end, free_drawn takes a slice, so batching a fixed-size array per allocator hold would shorten it without bringing back a growing list.

SEND BACK

[Japabu]

Review r4 of #1123 at 8153b4b (focused)

Scope: r3's B1–B4 at 8153b4b; the #1120 port fe05b82 (Reserved, Volume::unreserve, the hole below the old end); and the rebase resolutions in kernel/src/object/ops.rs, sdk/abi/src/{handle,syscall}.rs and system/fileserver/src/{cache,data}.rs. I read code only and ran nothing. I did not reopen B3, B6 or the A/B judgment.

Assessment

The B1 fix is the right shape. An owner read now holds a reference, as a holder does, so the slot's lifetime has one rule. The port keeps #1120's rule across the reserve/write-back split without a second mechanism.

One part of the split is still not carried across. Main's DATA write read a partly written page's old bytes only where the file had held a block for that page. The split reads every page below the old size. For a page in a hole, that page's block is the one the reserve just allocated, and it still holds stale bytes from the file that last owned it.

r3 findings at 8153b4b

• B1: fixed. owner_read (kernel/src/object/paged.rs:430-433) increments refs in the same hold in which live resolves the id. Ref's drop calls unref. I traced every path:

  • Release meanwhile. let_go drops refs from 2 to 1, invalidates the object (gone), and keeps the entry. The read's next hold sees object.gone() and answers Gone. The guard then frees the slot.
  • Pager end meanwhile. on_zero_handles lets each owned entry go and leaves the referenced one. Ref's later unref releases it with gone set, so frames are freed, not kept aside. This is the same path a Named takes after the end.
  • Slot reuse. A referenced slot is never pushed free, so add cannot hand it to a new object under the read.
  • supply/write against a reused slot. These re-run live(id) and can meet a new object at the same id. write_unpin matches on the frame, and the old frame stays pinned by the writer, so it cannot be the new object's. The call returns without panicking.

  I found no path from userland input or pager concurrency to State::entry's expect.

• B2, B3, B4: fixed. All three comments now say what the code does.

Blockers

N1. A partial write into a hole below the old end reads another file's stale bytes into the file (DATA).

• Where:
  • system/fileserver/src/pages.rs:419-426, the pager path's partial page.
  • system/fileserver/src/volume.rs:222-224, Writes::write.
  • Both read below = old.saturating_sub(start) bytes from the volume.
• Mechanism:
  • DATA's reserve (data.rs:553-562) allocates every page from first to last past the extents.
  • allocate zeroes only the bridge covered..first (lib/bcachefs/src/fs.rs:1052-1054, :1113-1120). For the pages in pages it says the caller "writes every page of pages, zero-filling what it does not cover".
  • A truncate-up records only the size, so pages in [covered, size) are a hole with no block.
• Failure:
  • A process writes one block to home/f, ftruncates it to 3 blocks, then pwrites 1 byte at offset 6000.
  • reserve allocates page 1 from freed blocks. put finds page 1 absent and takes the partial branch with below = min(3·4096 − 4096, 4096) = 4096.
  • volume.read(node, 4096, …) now finds the new block through block_for and returns what a deleted file left there.
  • Those bytes are supplied dirty around the one written byte. Holders read them at 4096..6000 and 6001..8192, and write-back makes them durable.
  • Writes::write does the same on a server without a pager, and for a file whose object went.
  • This is the disclosure class fileserver: a write refused after allocating gives back the blocks it took #1120 fixed, inside the file's size.
• Origin: main at 3add613 read the old page only if page < covered && page * BLOCK < open.size and otherwise zero-filled. 8bc51cf dropped that condition in the split. fe05b82 rewrote these lines but tests only the refused path.
• Fix: bound the read of old bytes by what the file held in blocks, not by its size. For example, Reserved could answer the end of the old contents: min(size, held · BLOCK) for DATA and size for FAT, which has no holes. Below that the page is read; above it the page is zeros.
• Cheapest check:
  • A DATA host test on the all-0xAB image the two new tests use: write a block, truncate up, sync, write one byte in the hole, then read the page back as zeros except that byte.
  • The same through Files in pages/tests.rs, since the pager path is the production one.
  • Today's code is the negative control.
• PR body: "A partial page is merged with what it held below the old end" (body line 35) is the behaviour to change.

N2. Entry.refs's doc is incomplete since B1's fix.

• Where: kernel/src/object/paged.rs:88-90.
• Problem: the doc lists "the pager's own reference, while unreleased, and each holder object's". An owner read in progress now holds one too (Ref, :969-979).
• Why it matters: this field's doc is where a reader works out when a slot can empty, which is the reasoning r3's B1 got wrong.
• Fix: add "and each owner read's, while it runs".

N3. The syscall-profile guard no longer guards the ABI's last number.

• Where: sdk/abi/src/syscall.rs:376. The guard asserts SYS_HANDLE_WAIT (151) < SYSCALL_PROFILE_OTHER. The highest number is SYS_PAGED_NOTICES (157).
• Problem: the bin count is right at 159, but the check that exists to catch a wrong one does not cover 152–157.
• Failure: a later rebase that resolves SYSCALL_PROFILE_BINS to main's 153 still compiles. kernel/src/sched/payload.rs:141 would then fold the paged calls into the "other" bin.
• Origin: pre-rebase 7a had the same omission (it named SYS_WAKE_LATENCY_HOLD while SYS_PAGED_NOTICES was 153).
• Fix: name SYS_PAGED_NOTICES.

fe05b82 otherwise

Correct on the refused path. In DATA's unreserve:

• keep_blocks(max(held, ⌈landed/BLOCK⌉)) keeps three kinds of block: those held before, the bridge pages allocate zeroed, and pages the write reached.
• The size goes to max(old, landed).
• Recording no entry is sound in a single-threaded server. The pager path's write_back_some and make_room run inside a write, but they touch only write_back, node_meta and other files' close.

The pager path's resize(cut) and unreserve use the same max(old, landed), so the object and the volume agree.

The default unreserve (FAT) truncates to max(old, landed) after a reserve that already grew the chain, so it only shrinks.

Rebase resolutions

• ops.rs: I compared every Pager/Paged arm with the pre-rebase branch (dd88215 against 49559dd), with Wakeups are kernel events, and a one-handle timed wait is SYS_HANDLE_WAIT #1108's EventSignal/EventWait beside them. They are identical, and the matches are exhaustive.
• handle.rs: Pager = 19 and Paged = 20 sit in ALL's word order, which the const assert checks. PAGER is bit 18, and ALL is 0x7_ffff.
• syscall.rs: 152–157 are dispatched at kernel/src/syscall/dispatch.rs:669-677, and 159 bins = 157 + 2. N3 is the only defect.
• cache.rs: the written/flushed counters compose with blockring: the arena holds only what is on the wire, and a flush keeps many writes there #1113's parallel flush.
  • write_run and an uncached write_data count each completed disk write, refused ones included.
  • The flush reads the count once its runs are done, and asks the disk to flush when the count has moved.
  • A data write that completes during that disk flush is counted after it, so the next flush covers it.
• data.rs: read_data/write_data replace visit and the cache writes as 7a intended. fileserver: a write refused after allocating gives back the blocks it took #1120's allocation refusal stays in reserve. The copy refusal moved to unreserve. N1 is the condition lost in the split.

Suggestions

• S1. A fully refused DATA write (landed == 0) leaves mtime stamped and the file dirty from reserve. fileserver: a write refused after allocating gives back the blocks it took #1120 on main returned before stamping. POSIX marks the time on a successful write. Reserved could carry the old mtime for unreserve to put back when nothing landed.
• S2 (evidence, for the orchestrator). The testcases boot on the T14 and the AMD at 8153b4b is still owed, as the PR says. The N1 fix touches only the file server's write path, so its host tests suffice for it. Once N1 is fixed, that boot should run on the head that contains the fix.

SEND BACK