[Japabu]

Fixes the two defects memory stage 7a (#1123) recorded, each at its owner.

A holder's read is atomic against its pager's write (XSH 2.9.7)

Problem. A holder's SYS_PAGED_READ copied 16 pinned pages at a time while the file server's write landed page by page in the same frames, so a read beside a multi-page write could return pages from before and after it, or a page torn mid-copy (issues/a-read-and-a-write-of-a-paged-file-can-tear-across-pages.md).

Now.

• Every write of a paged object runs as a write span (kernel::paged::Object). Each Write call and each dirty Supply opens one before its first byte lands and closes it after its last. A Resize moves the object's version.
• A holder's read starts only while no span is open. Its bytes count only if the version hasn't moved since it started. A read that meets a write answers WouldBlock. The SDK already asks the server whenever a holder read fails, and the server's loop orders that read against the write. Holder reads never wait on a write, so a pager never waits on its readers.
• A file server write takes several calls when a page is absent, partly written or at the dirty ceiling. A call that doesn't take all it was given leaves the span open (held). Later calls nest inside it, and the new PagedOp::EndWrite closes it. The server's first call of a write is always a Write. If it takes everything, that is the whole write, at no extra syscall. Otherwise the server ends the held span on every path, refusals included.
• Cost: two lock holds more per owner write call. A holder read pays one version compare per chunk. A read that meets a write costs one round trip.

An fsync commits its own file and what the commit carries

Problem. FSYNC synced the whole volume. It wrote back every dirty page and every open entry (issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md).

Now.

• Volume::carried(node) names the open files the next commit carries whatever it is asked to.
  • On DATA, one whose entry was written since the last commit (a rename, a shrink), or whose reserve took blocks since. The commit's allocator holds those blocks as taken.
  • On FAT, every changed open file, since a cache flush writes every chain the cache holds.
• Files::fsync writes back the named file's pages and every carried file's. Then Volume::commit writes the named entry and each carried one, and commits.
  • A carried file whose write-back is refused refuses the commit, since its entry would name blocks without its bytes.
  • After a stop, no entry names a block that holds an earlier tenant's bytes.
• A write over blocks a commit already named is carried by nothing. It waits for its own fsync or the volume's sync, which is what makes one file's fsync cheaper.
• A directory's fsync commits only the carried files. DATA writes every name change into the tree at once.
• SYNC and the write-back timer still sync everything. An fsync that leaves nothing owed (no dirty page, no entry the volume owes: Files::owes, Volume::owes) answers the write-back as a sync does; one that leaves another file's write keeps it due from that write. Round 1 found the fsync left it due always, so the server kept a 2 s deadline on every wait and ran a sync with nothing left to write.

Found along the way and filed, not fixed:

• A sync still commits the entry of a file whose pages were refused (issues/a-sync-commits-the-entry-of-a-file-whose-pages-were-refused.md).
• A refusal on one carried file is paid by the files beside it (issues/a-refused-carried-file-costs-every-other-files-fsync-or-leaks-its-blocks.md, from review r1). A persistently refused write-back refuses every fsync on the volume. A refused carried entry lets the commit land, and the blocks that file took leak after a stop. Neither discloses another file's bytes.

issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md stays open until the metal A/B records its exit's measurement.

Commits

Rebased onto main 058bea2; PagedOp::EndWrite = 9 is still free there.

• a4b6a1ee7 osbench: persist.write_4k_fsync_beside_1m alone, the A/B base.
• 31206cbf4 atomic holder reads (kernel, ABI PagedOp::EndWrite, SDK, file server, guest test). Closes the tear issue.
• b3897d50a per-file fsync (file server volumes). The fsync issue stays open for its measurement.
• 6ae35a46e review r1: docs and issue records only.
• 029ca221c an fsync that leaves nothing owed answers the write-back (new in r2).

Round 1's A/B, investigated

Round 1 (evidence r1) missed its claim's bound (AMD −28.1% against −30%; T14 unmoved) and flagged regressions. Its details: https://github.com/ToyOSOrg/evidence/tree/4abdb91acc0e9949310bf79ea9b02b054cc37c1a/r2

• Most flags were boot mode, not code. paged: a holder's fstat is its object's, without a request (FS stage 3a) #1144's A/B booted this PR's exact head image again as its base. There, the flagged rows measured as main did: T14 reopen_read_4k 16.9/16.7 us (main 16.7/16.7, round 1 head 19.8/20.0); AMD pread_8 811/922 ns (main 772/781, head 912/912); AMD fsync_clean 32.5/30.0 us (main 30.0/29.8, head 35.3/35.9); AMD write_4k_fsync 2.41/2.48 ms (main 2.48/2.47). AMD's round trips sit in one of two modes per boot (open_close about 14.0 or 17.2 us), and both round-1 head boots were slow. The T14's durable writes are bimodal too (1.1 or 5.9 ms) and can switch within a boot.
• One real change in cost, fixed. The old FSYNC reset the write-back deadline and the new one never did, so every server wait after a write and fsync carried a deadline that each park programs into the CPU's timer. Fixed by 029ca221c.
• Read in the code, kept. A holder read takes the same lock holds as main plus two version compares. A write call takes two more uncontended pager-lock holds (write_begin/write_end): tens of ns on a 10 us call, estimated, not measured. Folding them into the pin and unpin loops of both write and supply would tangle the span bookkeeping for that.
• The claim's measure. Raw beside_1m is dominated by the commit's flush, common to both sides and bimodal on the T14. Round 2 judges per boot d = beside_1m − write_4k_fsync, what another file's unsynced 1 MiB write adds. In round 1, d was 0.23x of main's on both benches. The measure was chosen from that data, so round 2 is its first test.

Checks

Evidence: https://github.com/ToyOSOrg/evidence/tree/4abdb91acc0e9949310bf79ea9b02b054cc37c1a/r2 (r0 and r1 on the same branch hold round 0's controls and round 1's A/B).

• cargo run -- --ci host at tree bfae305ac (029ca22): rc=0, 109 steps green.
• QEMU x86_64 (the testcases boot via --debug) at 029ca22, clean tree. All five members exit 0: a_served_files_read_never_tears_against_its_write (1105 reads, 244 generations met, none torn), a_served_file_reads_through_its_paged_object, served_file_offset, a_pager_s_end_answers_its_waiting_holders_gone, std_fs.
• Why the guest test. The tear is an interleaving of the kernel's copy of a holder read with the server's write into the same frames, on two CPUs. Each side's rule is host-tested, but only a guest runs the two together. Targets: QEMU x86_64 (run), T14 and AMD (staged).
• Negative controls, all built and red: round 0's C1–C7 and G1 (r0); the write-back fix's M1–M4 (r2), which make each owes always false or always true, or drop the dirty-page term. The FSYNC arm's three-line use has no host test (Server has none); the metal rows are its evidence.
• Metal round 2: staged, not run. Rules set before any boot (r2 metal-staged/rules.md): per bench, 3 base and 3 head benchcase boots interleaved, then one head testcases boot. Claim: every head d below every base d, with mean ≤ 0.5x. Regression: all three heads beyond all three bases, and the median worse by more than 10%.

Risks

• Concurrent reads and writes fall back. A holder read that overlaps any write to the same file, anywhere in it, goes to the server, because the version is per object, not per range. That is correct, but a reader tailing a file that is being appended pays a round trip whenever a write lands during its copy.
• A pager that never ends a held span leaves its holders reading through the server. It harms only its own clients and cannot fault the kernel.
• On DATA, an fsync next to a file being appended still writes that file back. Eager allocation means the commit carries it. This is ext4 data=ordered without delalloc.
• FAT is unchanged. FAT carries every changed file, so an fsync there costs what a sync of the changed files costs.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1141 at b4edede

I read the three commits, the full diff, kernel/src/object/paged.rs, kernel/pure/paged.rs, the syscall dispatcher, the SDK holder path (sdk/toyos/src/files/owner.rs, paged.rs), the file server's pages.rs, data.rs, fat.rs, volume.rs and main.rs FSYNC/SYNC, the tests, the issue records and the evidence at 1931b15/r0. I ran nothing.

Architecture

• Atomic holder reads. The kernel is the only party between a holder's copy and the pager's write, so it has to own the write span. It is a version under the lock the object already has: it adds no wait or thread, and nothing new depends on the pager. EndWrite is one new op number, not a new syscall, and a call that takes everything costs nothing extra. Falling back to the server keeps "readers never wait on a write". Holders fall back per object rather than per range, and the PR states this.
• Per-file fsync. "What the commit carries" belongs to each volume, behind Volume::carried/commit. Files::fsync only orders the write-backs before the commit. FAT carries every changed file, which matches its flush.
• Size. Net growth is about +250 production lines and +300 test lines. About half of the pages/tests.rs diff moves the Fake impl.

What I checked and found correct

• No userland action panics the kernel.
  • writing = in-flight calls + held. write_end either sets held or decrements, and end_write decrements only a held count. So the checked_sub cannot fire, and checked_add would need 2³² concurrent calls.
  • write_begin takes a slot reference, so write_end's entry_mut survives a concurrent Release and pager end. The slot cannot be reused under it.
  • A spurious EndWrite gets InvalidArgument, and on a released object it gets live's refusal.
• Atomicity. A version is captured only while writing == 0, and is checked after every chunk's copy under the lock. Reader loads come before the reader's unlock, which comes before the writer's write_begin lock, which comes before the writer's stores. So a read that passes its last check saw no write. This holds on weak memory too, because the lock gives the ordering.
  • An EOF answer while a span is open is the before-state, because the size only grows inside a span (cut = old.max(landed)).
  • The zero-length Paged::received probe answers Done before the version check.
• Spans always close. put.held is false only when the first Write took everything. Every later path in Files::write, the refused one included, reaches end_write. Dirty supplies happen only inside put.
  • A pager that never ends its span only sends its own holders to itself.
• Narrowed fsync on DATA.
  • Every tree change that names blocks sets carried: persist, a shrinking truncate, rename's persist, and reserve's allocation.
  • committed clears carried only for entries that are no longer dirty.
  • A Files file keeps its node held, so its carried state outlives every close.
  • So no commit makes durable an entry that names blocks without its bytes.
• Controls. G1 reds the guest member at the same boot, and C1–C7 each break the claim their test asserts.

Required

R1. sdk/abi/src/syscall.rs:3363-3370: the holder's contract omits WouldBlock.

• paged_read is the kernel/program boundary a holder implements against. Its doc lists Io, Gone, PermissionDenied and InvalidArgument, but not the new answer.
• The sentence is on PagedOp::Write instead (:3265-3266), an owner op that no holder reads.
• Failure: a holder written from the ABI, as in pagerprobe or a future mapping client, treats WouldBlock as an unexpected error rather than "ask the pager".
• Fix: move the sentence to paged_read, and keep only the EndWrite rule on Write.

R2. issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md is deleted, but its exit is not met.

• The exit asks for "a guest measurement of one file's fsync beside the volume's".
• At this head, persist.write_4k_fsync_beside_1m has run nowhere, not even QEMU. The handoff and PR body say the metal A/B is staged, not run.
• The commit message still says "Closes".
• Fix: land with the A/B recorded, or keep the issue open and name the missing measurement.

R3. A false durability claim.

• The commit message and PR say that after a stop "no block is held that no entry names". The doc of an_fsync_writes_first_the_pages_of_every_file_its_commit_carries says the same (system/fileserver/src/pages/tests.rs:599-600).
• DataVolume::commit (system/fileserver/src/data.rs:866-868) logs a carried entry whose persist was refused, then commits anyway.
• Failure: file C's reserve took blocks since the last commit, and its update_metadata is refused. Another file's fsync then makes the allocator's take durable with no entry naming those blocks. After a stop, they leak.
  • This is not a disclosure. The refusal is the right call, because the alternative refuses every fsync behind one bad entry.
• The test asserts only that C reads its own bytes, never the free-block count.
• Fix: delete the clause from the test doc, or assert the free blocks after the stop. Narrow the commit message on rework. The PR body is the orchestrator's.

Suggestions

• S1. kernel/pure/paged.rs:275: the version move in write_end is redundant.
  • A version is captured only while writing == 0, and every later write moves it at write_begin. No read can capture a version between a begin and its end.
  • Dropping the move simplifies the field doc (:220) and the "the end moved the version" assertion (:771).
• S2. The guest member wraps its generations.
  • generation as u8 (a_served_files_read_never_tears_against_its_write.rs:44) wraps after 255, so "each byte the generation's number" holds only mod 256. The run's "215 generations met" counts distinct bytes.
  • This is harmless. Either make GENERATIONS ≤ 255 or say so.
• S3. One file's refused write-back now refuses every fsync on the volume.
  • Where a carried file's write-back is persistently refused, every other file's fsync on the volume fails until that clears.
  • That is safe, but it is a new volume-wide failure mode that the Risks section doesn't name.
  • Where the file is carried only by taken blocks (its entry is unwritten since the last commit), leaving it out of the commit costs a leak, not a disclosure. Worth recording.

SEND BACK