Repository navigation
Conversation
As persist.write_4k_fsync, each iteration first writing 1 MiB over another 1 MiB file and leaving it unsynced: an fsync of one file that also makes another file's write durable pays for it here. It is the A/B base for a per-file fsync (issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md), which asks for one file's fsync measured beside the volume's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A holder's SYS_PAGED_READ copied a pinned chunk of 16 pages at a time while the file server's write landed page by page in the same frames, so a read beside a write of several pages could return pages from before and after it, or a page torn at the copy (memory stage 7a's recorded gap). Every write of an object's now runs as a write span on it (kernel::paged::Object): each write call and each dirty supply begins one before its first byte lands and ends it after its last, and a resize moves the object's version. A holder's read begins only while no span is open and counts its bytes only where the version has not moved since; a read a write met answers WouldBlock, and the SDK, which already asks the server whenever a holder read fails, reads through the server, whose loop orders the read against the write. A server write takes many calls when a page is absent, in part or at the dirty ceiling. A call that does not take all it was given leaves the span open (held) until the new PagedOp::EndWrite, and later calls nest in it, so the file server's whole write is one span. The server's first call of a write is always a write: one that takes everything is the whole write at no extra syscall; otherwise the server ends the held span on every path, the refused one included. Holder reads never wait on a write: the pager is never made to wait on its readers, and a read that met one costs one round trip. Tests: - kernel::paged: a read counts only where no write began or was under way; a short call holds the write until its end. - fileserver pages: a write of many calls (an absent page supplied, the dirty ceiling written back, a last page in part) is read whole or not at all at every call boundary, and ends whether it lands or is refused. - guest a_served_files_read_never_tears_against_its_write: a writer puts 400 whole generations of a 64-page served file while a reader on its own description reads it whole through its holder; red on a torn read. Closes issues/a-read-and-a-write-of-a-paged-file-can-tear-across-pages.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FSYNC synced the whole volume: every dirty page of every file written back and every open file's entry written before the format's commit. An fsync now makes durable the file it names and only what the volume's commit cannot leave out: - Volume::carried names the open files the next commit carries whatever it is asked: on DATA one whose entry was written since the last commit (a rename, a shrink) or whose reserve took blocks since, which the commit's allocator holds as taken; on FAT every changed open file, since a cache flush writes every chain the cache holds. - Files::fsync writes back the named file's pages and those of every carried file, then Volume::commit writes the named entry and each carried one and commits. A carried file whose pages are refused refuses the commit: its entry would name blocks without its bytes. - So after a stop no entry names a block holding an earlier tenant's bytes. A write over blocks a commit already named is carried by nothing, and waits for its own fsync or the volume's sync. A directory's fsync commits the carried files only: DATA writes every name change into the tree at once. SYNC and the write-back timer still sync the whole volume, and a per-file fsync leaves the timer due. Found while doing this: a sync commits the entry of a file whose pages were refused (issues/a-sync-commits-the-entry-of-a-file-whose-pages-were-refused.md). Tests (fileserver pages): an fsync makes its file durable across a stop and leaves another file's overwrite unwritten; a carried file's pages and entry go with another's fsync, so after a stop on a volume of stale blocks it reads its own bytes; a directory's fsync makes a new name durable. issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md stays open until its measurement, osbench's persist.write_4k_fsync_beside_1m base against head on metal, is recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aims narrowed - R1: the holder's answer WouldBlock is documented on paged_read, the boundary a holder implements against; PagedOp::Write keeps only the EndWrite rule. - R3: a carried file's test no longer claims no block is held that no entry names: DataVolume::commit commits past a refused carried entry, whose taken blocks then leak after a stop. Filed with S3, one carried file's refused write-back refusing every fsync on the volume (issues/a-refused-carried-file-costs-every-other-files-fsync-or-leaks-its-blocks.md). - S2: the tear member's generations are its bytes mod 256. R2 is in the fsync commit: its issue stays open until the A/B records the exit's measurement. S1 (the version move at a write's end is redundant) is declined for this round: it changes the kernel the staged A/B images were built from. Comments, ABI docs and issue records only: no code changes, so the staged images at b4edede still measure this head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The volume-wide sync an fsync used to run cleared the write-back the server's loop keeps due. The per-file fsync never did: after it, the server waited every turn with a deadline, which every park and idle entry programs into the CPU's timer, and 2 s later walked the volume in a sync with nothing left to write. The volume now says whether an open file's entry is changed past what its last commit made durable (Volume::owes), and Files adds its dirty pages (Files::owes). An fsync that leaves neither answers the write-back as a sync does. One that leaves another file's write keeps it due from that write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A served file's fstat was a file server round trip. A paged object now
publishes what a holder's fstat needs that can change: the kernel's own
size and write-span version, and the mtime its pager sets.
SYS_PAGED_STAT(holder) answers {size, mtime, version} under the
object's lock, WouldBlock while a write is under way (atomic as a read
is), and the SDK then asks the server. Kind, node, volume and nlink come
from the open's reply, which no change to the file moves.
The owner sets the mtime as a word it already sends: PagedArgs.time on
Write and Resize, paged_create's mtime, and PagedOp::Stamp for a utimens.
The file server hands the object the volume's own value after each
change (reserve, truncate, an open that truncates, a set time, a refused
write's cut back), at the volume's resolution. The version is the
object's data generation; there is no second counter.
FAT's mtime was its entry's last level, not its last write
(issues/a-fat-files-mtime-through-its-handle-is-its-last-level.md),
so it could not be published. A write or resize now records its even
second, which the handle answers until the level stamps it.
Tests:
- kernel::paged: a stat publishes no version while a write is held.
- fileserver pages: the object's size and mtime are the volume's after
an open, a write, a refused write, a truncate and a stamp.
- fileserver fat: a file's mtime is its last write's through its handle,
off the disk a sync later and after a reopen (replaces the test that
pinned the stamp to the flush).
- guest a_held_files_fstat_is_its_servers_stat: the held fstat agrees
with the server's stat of the path after each change.
Closes issues/a-fat-files-mtime-through-its-handle-is-its-last-level.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The orchestrator's ruling on the stage-3 split. The client name cache comes on mapped paged objects, with file slots, since stat by path of an existing file is the row that matters. Built before mappings, its slots would be read by syscall and it would need a names-only pager for SYSTEM and a dirty allowance for BOOT: an interim mechanism the mappings would replace. 3a, a held descriptor's fstat, is this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve the published #1141 and #1144 commits as dependencies. Resolve the syscall-number collision with firmware-window by assigning PAGED_STAT 165 and extending the profile to include both calls. These dependencies remain under their existing review; this merge does not claim approval or land them on main.
Save in-progress implementation before integrating paged-object fixes.
Preserve write-span publication and mtime semantics for anonymous objects.
Remove the kernel pathname and file-offset syscall declarations and SDK fallback. Keep generic object reads, writes and metadata queries. File metadata comes from the server so unlink and link counts stay authoritative; paged reads retain their atomic offset publication. Share runtime cwd through the descriptor owner's SDK ABI, initialized only once from spawn metadata. Derive synthetic read-only root and ancestor directories from the names actually held by the immutable namespace, exposed by a bounded generic namespace inspection operation. Host validation: SDK fd-table check and clippy, 5 files tests, 49 ABI tests, and all 106 libc module tests pass. Combined guest compilation follows the kernel filesystem removal and shared Rust reexport update.
Serve /tmp and diskless DATA roots through the fileserver namespace with one retained anonymous paged object per regular inode. Keep names and open descriptions independent, release unlinked bytes only after the last description and holder, and read sparse holes without allocating frames. Delete kernel VFS, tmpfs, pathname syscalls, file cache and file objects. Executable loading now takes its supplied immutable SharedImage directly. Keep bootstrap cwd transport; SDK routing removal is coordinated separately. Add owner-only advisory pager quota headroom, explicit memory grants, and host plus guest coverage for retention, sparse reads and quota refusal.
Consolidation checkpoint: routes DATA through the on-disk engine and removes the provisional format and its synchronous adapter. Holds inode identities and per-page reservation tokens. Engine token APIs and userland memory changes are integrated in following commits; this checkpoint is not a validated landing. The partition signature test passes with actual bcachefs recognition.
Resolve the volume contract in terms of inode identity and adapt memory reservations to explicit old content bounds. Validation follows combined SDK and engine integration.
Includes generic namespace name inspection, explicit namespace ancestors and authoritative file-server metadata. Preserves existing paged I/O improvements.
Write checksummed or authenticated file blocks through the existing journal and bucket allocator, retaining checksum slices of overwritten extents. Reserve concrete unpublished data slots and conservative metadata capacity before accepting dirty pages; keep pinned slots safe through bucket emptying. Resume on-disk logged truncates, clear truncated tails, and reclaim orphan extents. Keep read epochs alive through file-data I/O. Validation: eight focused host tests pass, including torn/reordered write recovery, encrypted version reservation, capacity refusal followed by reserved writes, and a pinned bucket's generation transition. no_std clippy passes with warnings denied. Metadata reservation bounds remain conservative; broader regression and pressure checks continue in the integration task.
Correct readdir continuation, retain paged descriptions across unlink, release unheld orphans, and deduplicate sync failures. Adapt fsync tests to real bcachefs transactions: dirty lengths are not published before data. The outstanding large-write capacity regression remains under repair.
Use the existing fileserver EXPORT protocol for inherited file descriptors, including readable stdin. Transfer each local offset to the server before export, and use that shared offset thereafter. Ordinary std Command spawns export a fresh connection each time; forwarding an already exported connection is explicitly one-shot. POSIX file-action dup2 aliases carry a process-local group in their fd label so the child adopts the protocol connection once and shares one descriptor object. Independent launch refuses ambiguous reuse of one raw exported connection instead of attaching it twice. SDK host tests (153), clippy with fd-table, and libc module tests (106) pass. Extend served_file_offset with guest checks for repeated input spawns, dup2 reads/seeks/writes, exact offset sharing and open-unlinked metadata. Guest execution follows the combined storage integration build.
Store the final partial sector as inline data with its suffix zeroed, retaining earlier checksummed slices and removing all later extents. This keeps shrinking possible after ordinary data capacity is exhausted and makes logged-truncate recovery independent of obtaining a fresh data block. Enable the format's inline-data feature and account inline sectors in the inode. Refuse overflowing reservation offsets before adding the block size. Add coverage for full-capacity truncate, imported extent slicing, reserved I/O failure, and the maximum-offset boundary. All twelve focused data tests pass, including plaintext/encrypted torn-write recovery. The per-token metadata reservation cost remains an unresolved integration blocker.
All ToyOS filesystem paths now use the process namespace in userland. Remove the platform dispatch reexport of the deleted kernel-versus-served routing predicate.
Box the mounted probe result, use a concrete flush-future type, and clean affected file-server lint findings. Empty direct writes take no reservation, and metadata failure cancels an accepted reservation. Validation: fileserver library clippy with warnings denied passes.
The file stays readable after unlink and description close while a holder remains, and its pages and inode are released after the final holder notice.
Replace kernel pathname probes with ordinary SDK filesystem operations. Root hierarchy checks use controlled namespaces, copyout probes use generic objects, and directory pagination checks cover complete served listings. Remove the retired backing-revocation actuator registration and its oracle. Keep process cwd within the bootstrap path bound and refuse oversized input before normalization. Include a complete-listing probe in the shared boot now that it no longer exhausts a kernel filesystem inventory. Validation: SDK 153 host tests, metal 81 unit tests, SDK fd-table clippy, focused path boundary tests, and host guest-harness compilation passed. Guest compilation and execution require the consolidated sysroot/image.
Served fstat now reads authoritative filesystem metadata, so paged objects no longer need a timestamp, Stamp operation, or holder metadata syscall. Remove that path from the kernel, ABI, SDK and filesystem mirror while preserving size, atomic read versions, write spans and authoritative Volume timestamps. Pager create now takes backing kind after size. Namespace syscall 166 and the remaining paged operation numbers stay stable. Keep size-coherence tests and remove only assertions about the obsolete timestamp mirror. Include three mechanical file-server binary lint fixes uncovered by all-target clippy. Validation: SDK 153 and ABI 49 tests, paged 18 tests, both bare-target kernel clippy checks, SDK fd-table clippy, file-server binary check and all-target clippy passed. File-server tests 99 passed; its existing 64-page capacity test still fails with NoSpace before touching the pager. No guest image built here.
Allow an in-memory directory to replace an empty directory. Update each shared directory description once during rename, retaining removed destination descriptions and moving source aliases together. Migrate the identity guest test from implicit kernel directories to served directory lifetimes. Validation: file-server memory and description host tests pass. Guest execution follows the integrated image build.
Let the file server enforce dirty-page limits; cp no longer forces fsync every megabyte. Keep the final sync before destination replacement to report delayed write failures. Update the memory benchmark label and sparse test-device description for userland storage. Validation: toybox copy host tests pass.
The removed suites use only the deleted BCFS/BTND API. Real bcachefs operations, I/O failures, reservation pressure, and crash recovery are tested in engine/tests and the served DATA tests. Keep all upstream-generated format fixtures.
Capture adjacent siblings as one bounded rewrite, publish their replacement pointers after all bytes complete, and transfer later changes by key range. Keep still-underfull replacements queued until their siblings can compact. Collapse a one-child root through the journal and ignore obsolete higher levels during replay after collapse. Expose the same relocation path for metadata bucket evacuation and share the minimum fill calculation with capacity admission. Existing conservative reclaim reserve assumptions remain separate integration work. Validation: four host tests cover repeated create/delete churn, clean-node relocation, changes between merged writes and pointer publication, and torn/reordered crash recovery through merges and root collapse.
Prepare shared capacity accounting by maintaining non-structural leaf bytes, BTREE bucket count and imported non-root node debt across applied changes. Allocation index and metadata-backpointer positions have finite bounds from device geometry; the other records are charged from their encoded lengths. Report space excluding outstanding metadata promises. This prepares the occupancy-based reservation ledger; admission still uses the earlier conservative per-token bound pending rebalance and bucket evacuation integration. no_std cargo check passes.
Hold exact data sectors while pending pages charge their enforced journal growth bound. Prepay geometry-bounded allocator records and account live non-structural records; include imported underfull nodes until rewritten. Sparse closed metadata buckets can move through the existing node writer. Keep a reservation until transaction application succeeds so failed preparation retains the accepted page's capacity. Authenticated version reservation needs no additional leaf-key growth. The 64 MiB pressure test accepts 1200 pending pages and writes all of them after allocation refusal. Twelve data tests and no_std clippy pass. Worst-case shared reclaim sizing and GC pressure integration remain under active review; this checkpoint does not claim that physical proof complete.
A clean imported non-root parent can have only one child. Deferring that child without scheduling its parent leaves no operation able to provide a sibling. Mark the parent for compaction so reclaim can work upward until the child gains a neighbor. Full-range root spines remain exempt. Validation: cargo test -p bcachefs --lib engine::tests::compact, exit 0, four passed. Log: /private/tmp/toyos-bcachefs-reclaim/unary-parent.log. The imported sole-child case is established by inspection; the existing merge, relocation, captured-delta and crash tests are regression coverage.
Packing unrelated files into a 64 KiB checksum range made a small read verify that whole range. Combine live slices only within one filesystem block; keep an existing larger extent alone. Sort by live size for packing and skip a candidate whose packed allocation cannot release bucket space. A device wrapper verifies that both 4 KiB and 512-byte files read exactly one physical block after plaintext and authenticated evacuation. All 19 data tests, including retention pressure, cancellation and torn writes, pass; no_std clippy passes. Clarify that successful application consumes a token while failed preparation preserves it, without promising that all metadata admission errors are impossible at writeback.
Give signed-image preparation rows raw DATA access through the existing partition grant. disks format data reports its unique target and contents, requires --commit to overwrite them, and verifies then cleanly unmounts the new bcachefs volume. Ordinary images receive no formatting authority. Add an explicitly selected preparation boot and two mounted-DATA boots that check Rust file operations across a real reboot using a fresh shared token. Align DATA evidence with native bcachefs and stage argument-bearing Rust test commands by their executable name.
Replace the separate replay key and accounting maps with one pending owner. Insertion identity selects the newest ordinary value while the earliest original journal sequence retains its pin; accounting deltas remain ordered and versioned. Batch preflight funds incoming and merge storage, then moves owned values without allocating during apply. Replay overlays use the same ordered view. Captured transfers establish the destination pin before retiring their source and leave newer entries intact. Run searches avoid scanning the full pending set for each transfer. This is a representation checkpoint. Runtime transaction staging and reclaim liveness are not changed by it. Validation: engine library suite passed 161 tests; the subsequent test-only allocation fault injector passed all seven pending tests, including separate incoming and merge allocation failures. no_std library clippy passed with warnings denied. The inherited integration test target still references the interim filesystem API being removed by the consolidation branch.
A missing executable now fails in userland before the spawn syscall. Use an existing malformed ELF so both process-tree arms still exercise kernel loader refusal after admission, and retain the host check for both refusal records. Remove the obsolete claim that the kernel opens the temporary executable. The T14 hardware run exposed the stale oracle: the guest passed, while the host correctly refused to credit a loader path that had not run. Independent review approved this replacement; both benches will rerun the affected boot.
Ordinary returned errors and unused reservation drops are in the current consolidation scope. Arbitrary future cancellation is a distinct finding: the fileserver currently runs engine operations to completion, and no new cancellation support is claimed by the pending-key representation.
Move logical apply and replay to one pending-key owner, merging its overlay into every cached tree read. Materialize bounded ranges on reclaim pressure, transfer original pins before source retirement, and run allocator/accounting triggers only at original transaction apply. Fund pending workspace before prepare so allocation refusal preserves DATA reservation retry state. Track latest represented journal sequence separately from the oldest lifetime pin and preserve source bset dependencies through fresh rewrites and clean relocation. Permit finite freeing operations past the ordinary stable-growth estimate while retaining physical and journal admission checks. This is an isolated review checkpoint, not a completed reclaim solution. Full-volume reboot/removal tests still exhaust the live journal reserve at removal 865. Both negative controls for bset provenance build and fail at their intended assertion; the fixes are restored. All 163 other library tests pass, and no-default-features library clippy passes. Keep the expanded all-removal, metadata-only churn and reuse tests as the remaining liveness gate.
New ordinary grants could spend space in an entry opened by reclaim even below the ordinary reserve floor. Reclaim repeatedly admitted more work instead of retiring the oldest pins; full-volume reboot and removal stopped at the recovery share. Refuse those new grants until the floor is restored, while preserving existing grants and both reclaim shares. Release superseded ordinary pins only after replacement records are durable, rebuild run pin indices, and preserve captured dependencies and additive accounting. Materialize positive node growth within the existing bound, coalesce shrinking changes across the same leaf, and retire durable values already represented by the cached node. Publish pin-only progress through an empty existing-format journal checkpoint when a ring slot is available. All 168 library tests pass, including complete removal and capacity reuse from both DATA-full and metadata-full crashed volumes. New plaintext and encrypted low-journal-headroom tests exercise torn/reordered publication, read-write recovery, committed-prefix preservation, drain and fsck. The pre-floor candidate fails the unchanged removal test at865, retained as the negative control. No-default-features library clippy passes. This fixes the demonstrated native-geometry starvation. It does not claim termination for node==block geometry, where every node write is fresh and creates further backpointer work; that geometry decision remains open.
Each link of process_tree's depth chain spawned the next from a copy on /tmp. With /tmp served by the file server, every link kept its own /tmp directory connection, so the chain met the server's 32-client share bound at link 32, before the kernel's depth refusal at 65. The test now reads its own program once as the immutable region the SYSTEM server holds it in, and each link starts the next with CommandExt::boot_image and endows it a copy, so no link asks a file server for anything. The depth-65 refusal and whole-subtree teardown assertions are unchanged; B's spawn of a malformed /tmp executable still covers a served-/tmp spawn refused after admission. toyos::process::Image::as_immutable exposes that region only when the image is one, so a writable image's handle still leaves only for a spawn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A served file at stdout now reaches the child libc's system starts: spawn_served_stdout puts a /tmp file at descriptor 1, runs `echo child` through system and reads the parent's then the child's line back from it. libccase starts the MEMORY file server it needs. A write a gibibyte past a DATA file's end lands as a hole: a 64 MiB volume could not bridge the gap, so the host test passes only because nothing is allocated before the written block. 206_libc_refusals stops expecting symlink on /tmp to be refused: /tmp is a file server's, which makes the link. lseek_past_eof reads its ceiling from toyos::fs::MAX_FILE_BYTES, the kernel's file cache it named being gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deleted, each by its cause being gone: - a-kernel-files-fstat-answers-its-handles-mtime: the kernel holds no file; fstat is the file server's. - a-link-on-a-kernel-mount-to-a-served-path-leads-nowhere and the-kernel-resolves-a-path-by-rules-of-its-own: /tmp and /system are served, so every link resolves in fileserver::resolve, whose tests hold absolute hand-back, climbing by directory and loop refusal. - a-symbolic-link-on-tmp-displaces-its-name-and-lists-nowhere: the MEMORY volume refuses an existing name and lists links (symlinks_keep_their_names_and_never_replace_existing_entries); libc sends every symlink to its server. - a-listing-still-scans-the-whole-mount: tmpfs and the private format are gone; DATA lists one directory's dirents, MEMORY its children, ROOT a prefix range. - a-bcachefs-hole-costs-a-block-write-per-page and one-write-far-past-a-files-end-holds-data-s-server: native DATA writes only the block a write touches, so a gap is a hole (a_write_a_gibibyte_past_the_end_leaves_a_hole). - a-file-servers-refused-connection-reads-as-a-missing-program: no is_served routing falls back to a kernel path. On the T14 at 0507cbe (proctreecase) the chain's link 32 read the server's own refusal, "the file server serves as many connections as it takes". - a-served-file-at-stdio-stops-every-spawn: toyos::fd::copy exports a served file; spawn_served_stdout runs it through libc's system. With the export replaced by a refusal, that case alone fails, at system, errno EOPNOTSUPP. bcachefs-crate-is-not-bcachefs is renamed the-bcachefs-crate-must-implement-bcachefs: the private format is gone, and what remains is upstream fsck of the integrated writer and every accepted geometry finishing its unmount. four-shares-take-a-file-server records that /tmp now spends a share's 32 connections, one per live process that touched it, which stopped process_tree's /tmp chain at link 32. The storage and small-kernel tracks, the /tmp performance record and records naming removed kernel files describe the served tree. Comments that still described the kernel's /tmp, its mounts or its path normalization (toyos::launch, the supervisor, toyos-abi's SpawnArgs and Stat, toyos::fs, toyos-swap, osbench and guest tests) describe what serves them now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main took syscall 165 for SYS_TIMING_CONTROL (#1063). This branch's SYS_NAMESPACE_NAMES stays 166 above it, with 168 profile bins, and the one bound assertion names the highest number. Merged rather than rebased: the branch carries the merge commits that joined #1141, #1144 and the integration's own lines, and a linear rebase would replay their merged-in commits onto bases they were never written for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernel's spawn judged the working directory against its own mounts until #1152 removed them, and since then took any text. A child whose spawn named an empty, relative or over-long directory reached its first file call and panicked in the SDK, which reads the directory back with SYS_GETCWD into MAX_PATH bytes and requires it absolute. SYS_SPAWN now refuses such a directory InvalidArgument (kernel::cwd::spawnable); its meaning stays the file server's and its . and .. the client's. Host tests hold the predicate; dropping its absolute check builds and fails the refusal test. abuse_spawn_argv asks the three directories of the kernel through SYS_SPAWN beside an honest spawn of the same arguments. Two guest tests had kept the spawn ABI #1152 removed. abuse_spawn_argv passed no image, so every refusal it asserted was the missing image's and its "not a program" case expected NotFound; each case now carries a real image, and the honest spawn starts a child. abuse_elf_loader spawned paths with no image, so its refusals held vacuously and its dies and refuses_itself cases could not start a child; a path spawn now reads the file into an image, which is the one route left, so its second "as an image" arm is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…be record The owner's 2026-09-01 ruling took an exact check: a volume copied onto a device of another size does not mount. The private format held it in its superblock check, and the native engine dropped it, refusing only buckets past the device. Mount::spans_its_device answers whether a volume has exactly the buckets mkfs lays out on its device, and DataVolume::probe refuses one that has not. Its test copies a formatted volume onto a larger partition; without the refusal it mounts. probe-mounts-on-a-checksum is rewritten against the native code: what bounds a hostile volume now, that authentication stays the metal track's, and why a stamp over a used volume still does not reformat (the stamp is block 0 and the primary superblock is sector 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu
added a commit
that referenced
this pull request
Nov 3, 2026
…in one hold A claim costs a small transfer at least two holds of the object's lock where the copy under the lock took one, and on the T14 that cost small pipe writes 28-30% and request-reply IPC 12-26%. A transfer of at most `BOUNCE` (4096) bytes is now copied between its caller and a `Bounce` on the stack with no lock held. Between the bounce and the frames the copy is kernel to kernel, in one hold: - a pipe write puts its bytes whole (`Bytes::put`), unless it would want frames, which it takes with the lock let go as a claim does; - a pipe read takes its bytes whole (`Bytes::take`) and copies them out after; - a send's bytes are copied in before any lock, and the message is queued whole as its handles move, in the sender's process hold, as before the claims; - a receive pops the front message and installs its handles in the hold that looks the connection up, where the table has room without growing (`HandleTable::has_room`), and copies its bytes out after. Larger transfers, a transfer that meets a claim in flight, and a receive whose table must grow keep the claim path, which kept their gain (large pipe writes +71-83% on the T14). The bounce is uninitialized and filled through `read_raw`, so a 64-byte write zeroes no 4 KiB. A put or a take owes no wake: a copy asks to be woken only while a claim is held, and none is held across either. The track records the orchestrator's 2026-11-03 ruling: 7b needs the met half of S0-a and, after #1152, the gate extended to sleeping locks; the rest of S0-a is built as part of 7b. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A swap staged its bytes on /tmp and the supervisor read them back, and
read the installed copy again at every start of the service. With /tmp
served, each of those reads was a request to the MEMORY file server
from the supervisor's own loop, and the SDK waits on a file server's
reply without bound, so a MEMORY server that took a request and never
answered held the supervisor.
swap now lends the bytes in a memory region sent with the request. The
supervisor copies them once into a region it creates and alone holds,
holds that copy against the digest, and starts the service from it, at
the swap and at every later start. No file is read and no file server
is asked. The request carries the length beside the digest and the
service; the staging and installed-path helpers, the per-start
re-verification and the clean-up of installed files are gone.
This also closes a-swapped-binary-lives-where-any-process-can-rewrite-it,
by its second exit shape ("bytes the supervisor hands over"): nothing
but the supervisor can write what a swapped service runs, at any start.
std's CommandExt::boot_image becomes CommandExt::image, taking either a
region nobody can write or one of the caller's own it keeps unchanged
(toyos::process::Image::held); the kernel already pages a child from
either. The grants' package digest keeps its own bounded read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#934 took syscalls 166 and 167 for the device set; this branch's SYS_NAMESPACE_NAMES moves above them to 168, with 170 profile bins and the one bound assertion on the top number. Object kinds: this branch removed File and main added DeviceSet, which is 20 here. acpicase.toml takes main's form, which starts no storage at all. libccase.toml keeps main's acpiserver beside this branch's MEMORY file server. The supervisor's boot starts are main's start_service, starting a row's binary as this branch's Binary::Row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reclaim writes a node, then journals the update that reaches it. A rewrite to new nodes also changes allocator, backpointer and accounting keys, which dirty further nodes. Where a btree node is one block, every write is such a rewrite, so the unmount's drain never reached zero: on 32 KiB blocks and nodes it stopped the mount with "journal reclaim's share exhausted". A node of several blocks only delays the cascade until its tail is spent, so no ratio had a termination argument. The unmount now has three phases: - Reclaim writes everything committed before the unmount, oldest pin first, keeping the working set within the cache. - The remaining pending keys are materialized. - `Mount::publish` (mount/publish.rs) writes what is left. The publication plans every changed node once: an append where the bset fits, otherwise new nodes split at fixed bounds. It computes every key those writes derive against the volume as it stands, and adds each node such a key lands in, with its parents, to the plan. The plan only grows (members, append to rewrite, bounds, slots), all drawn from the volume's finite nodes, positions and slots, so growth stops. Derived positions and sizes depend on the plan alone, so the derivation then repeats within two rounds (asserted). Every node is written once, with its bsets naming the clean entry. That entry's roots alone publish the set, and nothing is journaled. A crash before it is durable leaves the old window and roots: the new nodes take only slots free on the medium, and their bsets name a sequence the next mount blacklists. Reclaim's pointer update is factored so that reclaim and the publication share it, and `fresh_node` takes its sequence. mkfs takes a block and node size (`Shape`) so tests can reach every geometry the writer accepts. The blacklist test now reads the crashed mount's bsets before the next mount, which may legitimately write those free slots again. Tests: every accepted geometry (one-block nodes, one or two to a bucket; two-block nodes; the native eight) through two mounts of churn; encrypted one-block nodes with the nonce oracle; one publication of 127 scattered members; a publication forced to regroup; and crashes at every request, unmount included, plain and encrypted. Negative controls, each red: the old drain, members not joining, releases before fills, values not settled, a rewrite over the old slot before the clean entry. Upstream's fsck passes ratio 2 and 8 images, clean and crashed; it cannot check written one-block-node volumes, as recorded in issues/upstream-cannot-check-written-bcachefs-volumes-of-one-block-nodes.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An ordinary transaction that the room left cannot hold was refused only when nothing at all was pending, dirty or unpointed. On a node of one block, reclaim never reaches that state: each write's allocator, backpointer and accounting keys dirty another node. So a create at the edge of a full volume ran reclaim forever, neither granted nor refused. Measured on 64 MiB of 32 KiB blocks and nodes, filled with 240-byte names: 8000 creates in 3 s, then one that never returned. The base revision 0249f7f does the same; the dump shows the refusal's inputs unchanged across 60000 reclaim passes. The refusal now waits only for what could change its answer. That is no other grant outstanding, no emptying awaiting durability, and nothing committed before the request's arrival left unwritten. The ledgers it judges (logical metadata bytes, the non-data buckets) are left as they are by reclaim's own later rewrites: their derived keys are prepaid and their buckets stay non-data. Pins older than the arrival are finite, and reclaim retires the oldest first, so the refusal is reached. The same fill now refuses at 9343 creates in 3.5 s, as on native nodes. The node-of-one-block full-volume test (fill to NoSpace, crash, remount with every node fresh, remove everything, churn, 256 KiB reuse, clean unmount) passes. With the old condition it did not finish in 300 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hecked geometry The writable-geometry fix ends every clean unmount in one publication, and the owner kept one-block nodes writable; upstream cannot check those volumes, which their own record carries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
Review r1:
|
Japabu
added a commit
that referenced
this pull request
Nov 4, 2026
Review r1 of #1152, R1. Members of the terminal publication only join, but a derived position can leave between rounds. An LRU key's position is its bucket's fill, so a later round's fill or release in the same bucket moves it, or removes it. An LRU leaf that joined for such a key could end with no change. Being written and appendable, it was still appended to: an empty later bset, which the reader refuses ("later bset with no keys"). The volume no longer opened after a clean unmount. An append member with no changed position is now neither written nor reached again. Its parent's pointer change then leaves in the next derivation, settling upward. The encoder refuses an empty later bset, as the reader does. Write time checks that an appended bset and a new node have the size their pointers were derived with, and that a new node fits its slot. The fixed point still holds: - Growth is unchanged and monotone: members, append to rewrite, bounds and slots. - Leaf-level derived keys come only from rewrites' fills and releases and from accounting, so they follow from the plan alone. - A level's pointer changes follow from which members below it have something to write, which the level below settles one round earlier. - So once growth stops, positions settle within one round per level, values in one more, and the next round finds no change. The settling bound is now 4 levels + 2. In the same file, R2 and S5: the module header keeps only the contract and the invariants a change must keep, and the publication's slot type is renamed `Placed`, apart from the writer's `Slot`. The new test reproduces R1 on 256 MiB of 4 KiB blocks and 32 KiB nodes, four to a bucket. Data buckets are left a quarter full, so the LRU spans leaves split by fill. One inode is changed, and its leaf and the accounting leaf are to be rewritten, as after an unclean mount. The inode leaf's new node leaves its bucket a quarter full; the accounting leaf, joining, takes the bucket's next slot and moves the key out of the first LRU leaf. The test asserts that a member was left with nothing. With the fix reverted the reopened volume is refused (Corrupt "later bset with no keys"); with only the encoder's assertion kept, the assertion fires at the unmount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu
added a commit
that referenced
this pull request
Nov 4, 2026
R2, the rest: - The admission comments no longer claim later changes are reclaim's own: the refusal waits for outstanding grants because the ledgers change when a transaction applies. - The `Order::LeavesFirst` doc names its use. - mkfs's `Shape` says what it lays out, not what upstream's format takes, which refuses block = node. R3 records the publication's memory beyond `cache_nodes` as issues/the-bcachefs-unmount-publication-holds-its-closure-in-memory.md. Optional review items taken: - One-block nodes filled to the refusal, then unmounted clean with nothing removed, one and two to a bucket. - The full-volume crash, removal and reuse on two one-block nodes to a bucket. - The nonce oracle on an encrypted eight-block-node publication, which appends. The new-node size check (S1) and the `Placed` rename (S5) are in the R1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Replace the private BCFS/BTND DATA format with the native bcachefs engine. Move the RAM-backed filesystem into the userland file server and remove the kernel VFS, tmpfs, pathname syscalls and associated file cache. Rust std and libc resolve files through the capability namespace; the kernel receives executable memory objects instead of opening paths.
The branch also preserves served descriptor ownership across spawn, obtains authoritative file metadata from the server, adds native DATA writes, sparse-bucket evacuation and journal-owned pending runs with reserve-protected reclaim, and provides an explicitly authorized
disks format data --commitpreparation path with a two-boot persistence test.Why
Persistent DATA must use the format the crate claims to implement, and filesystem parsing and pathname policy belong in userland. This consolidates the existing storage paths rather than retaining both implementations.
Evidence
Integration checkpoints
f32a98839,db6cab94a(maine3a421563),7987307ff,e7f2a85b0(#934 merged in) andf14f79e50(geometry fix) (sanitized logs, README per file: https://github.com/ToyOSOrg/evidence/tree/96293651edd3ca736d253677c8946a16b4f6695b/storage-integration):cargo run -- --ci host: 109 steps green at each checkpoint; the latest, exit 0, atf14f79e50.cargo run -- --ci image(x86-64 and AArch64): exit 0 atf32a98839,db6cab94a,7987307ffande7f2a85b0.cargo test -p bcachefs --libatf14f79e50: exit 0, 176 passed; no-default-features clippy exit 0.libc_children(QEMU x86-64) ate7f2a85b0: exit 0.process_tree,lseek_past_eof,fs_truncate_persist,spawn_from_tmp,abuse_elf_loader) build for x86_64-unknown-toyos with no warnings; none is booted by a run above.cargo test -p bcachefs --libat4e829b244(reclaim integrated): exit 0, 169 passed;cargo clippy -p bcachefs --lib --no-default-features -- -D warnings: exit 0.libc_children(QEMU x86-64,tests/images/libccase.toml) at25f19a076plus the then-uncommitted served-stdout case, identical to48316fc0e's: exit 0, every member including the newspawn_served_stdout. Negative control:toyos::fd::copyrefusing a served file builds, and onlyspawn_served_stdoutfails, atsystem, errnoEOPNOTSUPP.fileserver'sa_write_a_gibibyte_past_the_end_leaves_a_hole(in the host run): passes on a 64 MiB volume, which no bridged gap could fit.Earlier evidence, by checkpoint (not rerun; raw logs in local scratch, not yet published):
cargo run -- --ci hostat63783da5d: exit 0, 109 steps.cff9c3ebd: explicit native DATA formatting, write/sync, reboot and read-back;bench_data_formattedandbench_data_survives_rebootgreen. Upstream bcachefs-tools v1.39.7fsck -n -Kexit 0 on both benches' DATA. Plain and encrypted host artifacts passed upstream fsck and independent FUSE reads, including 1,600 retained sparse files.605c7c3cb: 14 Rust/C file, descriptor and pager tests plus launch, badge and app-view checks, 17 green.0249f7f23content): whole-removal, metadata-full and DATA-full reopen, low-journal-headroom crash oracle (plain and encrypted, 8 cut points × 4 variants), the floor-disabled control failing at removal 865, and provenance negative controls. Independent delta review found no further confirmed defect. Integrated here as11632e8b6and4e829b244, with two conflict hunks (a doc comment andcan_free's terms).Regressions this branch had introduced, fixed
SYS_SPAWNnow refuses itInvalidArgument(kernel::cwd::spawnable, host-tested with a negative control;abuse_spawn_argvasks it through the syscall)./tmp. Swap staged binaries on/tmpand the supervisor read them from its loop; the SDK waits on a file server without bound. Nowswaplends the bytes in a memory region, the supervisor verifies its own copy and starts the service from it at every start. No file server is asked, andissues/a-swapped-binary-lives-where-any-process-can-rewrite-it.mdcloses by its "bytes the supervisor hands over" exit shape. std'sCommandExt::boot_imagebecomesCommandExt::image, accepting the caller's own unchanged region too. The record named that choice the owner's; it was made under the orchestrator's direction and needs the owner's confirmation. Swap has no behavioural test before or after.Mount::spans_its_devicerestores it at the DATA probe, with a negative control.issues/probe-mounts-on-a-checksum.mdis rewritten against the native code.abuse_spawn_argvandabuse_elf_loaderspawned without an image, so their refusals held vacuously and their started-child cases could not pass; both carry real images now.Process-depth test
process_tree's chain ran each link from a copy on/tmp. With/tmpserved, every link kept its own/tmpconnection, so on the T14 at0507cbe36the chain met the file server's 32-client share bound at link 32 before the kernel's depth refusal at 65. Each link now starts the next from this program's immutable image throughCommandExt::boot_image, endowing it a copy, so no link asks a file server for anything. The depth-65 and subtree-teardown assertions are unchanged. The cap is not raised; the limit/tmpnow shares is recorded inissues/four-shares-take-a-file-server.md./tmpspawn coverage stays (spawn_from_tmp,fs_truncate_persist, B's malformed-ELF refusal).What is unsure
Draft: not ready to land.
8e117f19eandf14f79e50(from the reclaim branch'saaa71f0bb,825b9ba7a) end every clean unmount in one terminal publication; the owner kept one-block nodes writable (issues/upstream-cannot-check-written-bcachefs-volumes-of-one-block-nodes.md). Its review found a blocker (an empty appended bset after a clean unmount on production-sized volumes); that fix follows from the geometry agent.process_treeon both benches (metal-only; staged), the sharedtestcasesboots (staged) that carry the C corpus and Rust guest binaries (206_libc_refusalschanged:/tmpnow makes a symlink), final runtime-reclaim validation on both benches, regenerated upstream fsck of the integrated writer, and interleaved Linux/ToyOS performance on both benches. That comparison must checkfs.disk.renamespecifically: paged: a holder's read is atomic against its pager's write; an fsync commits its own file #1141's last A/B flagged it +19.8% on the AMD, unexplained./tmp's file-server cost against the kernel tmpfs it replaces is unmeasured (issues/a-tmpfs-metadata-call-costs-toyos-up-to-six-times-linuxs.md).SYS_TIMING_CONTROL) and 166/167 (PCI root bridges declared by acpiserver, device lifecycle and native hot-plug #934's device set);SYS_NAMESPACE_NAMESis 168 with 170 bins. Main was merged in rather than rebased: the branch carries the merge commits that joined paged: a holder's read is atomic against its pager's write; an fsync commits its own file #1141, paged: a holder's fstat is its object's, without a request (FS stage 3a) #1144 and the integration lines.Anything a reader of main must not miss
mainbranch and pinned by this branch's gitlink. It removes std's reexport of the deleted SDKis_servedpredicate. Review the fork delta and SDK change together here, following the workflow used in handles: only an owner ends, moves, duplicates or lends a handle #1080, files: one client of the file servers, in toyos, for std and libc alike #988 and sdk: the transport is toyos::transport, and no crate of its own #972. The unnecessary separate Rust PR was closed without merging.f32a98839's message;bcachefs-crate-is-not-bcachefsis renamedthe-bcachefs-crate-must-implement-bcachefs, whose exit is the integrated writer's upstream fsck and every accepted geometry finishing its unmount.🤖 Generated with Claude Code