From a308f58d6faa2f0c0e8d6deb6c608319315e1fa5 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 22 Oct 2026 10:33:08 +0200 Subject: [PATCH 01/10] mm: vm_reserve, vm_protect, vm_discard, vm_decommit and vm_release The calls a process changes its own memory with, without the charge, which lands with the ledger. `SYS_MMAP` and `SYS_MUNMAP` become `SYS_VM_RESERVE` and `SYS_VM_RELEASE`, and three calls join them: - `vm_reserve(len, prot, at)` places a range anywhere at an alignment, or fixed in place of what the pages held; any protection but `None` commits it. - `vm_protect` commits reserved private pages and changes protection in place: a committed page keeps its frame and contents under `None` (its leaf is held, not present). An object's pages take nothing past the protection they were mapped with, which the inventory now keeps per object range. Every change to a present leaf shoots down, a widening included. - `vm_discard` frees the frames and keeps the ranges: a private page reads zeros at its next touch, as `MADV_DONTNEED` does. - `vm_decommit` frees frames and commitment; the pages are reserved again. All are all-or-nothing over pages every one of which a range holds (`BadAddress` otherwise), and discard and decommit free through the one clear primitive, which ends every futex wait on a frame before it goes. `VmProt` is a value with no writable and executable member; `ReadExec` is `NotSupported` until executable private memory lands with its cache maintenance. Consumers: - std's dlmalloc reserves through `vm_reserve` and trims with `free_part`. - std's thread stacks are a reservation of the size asked, rounded to pages, with one guard page below and the stack committed before the thread starts; the 2 MiB floor is gone, so a smaller size is honoured. - libc's `mprotect` and `madvise(MADV_DONTNEED)` work; `PROT_EXEC` is still `ENOTSUP`. Its allocator trims too. - Guest tests move to the new calls; `mmap_stress` now covers the lifecycle, with a child that faults on each kind of unreachable page, and a 1 TiB reservation. - osbench's notes no longer say ToyOS commits in 2 MiB pages. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...bc-refuses-what-toyos-cannot-yet-answer.md | 9 +- ...b-pages-still-copies-each-programs-text.md | 6 +- ...-capability-end-state-is-twelve-answers.md | 2 +- kernel/pure/vm.rs | 14 +- kernel/src/mm/space.rs | 89 ++++++- kernel/src/syscall/dispatch.rs | 12 +- kernel/src/syscall/vm.rs | 111 +++++---- sdk/abi/src/syscall.rs | 156 +++++++----- sdk/libc/arch/src/memory_refusals.rs | 35 ++- sdk/libc/src/lib.rs | 38 +-- sdk/libc/src/memreq.rs | 49 +++- sdk/libc/src/posix_io.rs | 117 ++++++--- sdk/libc/src/refused.rs | 5 - sdk/std/sys/alloc.rs | 34 ++- sdk/std/sys/thread.rs | 44 ++-- sdk/toyos/src/net/stream.rs | 4 +- sdk/toyos/src/shm.rs | 4 +- tests/corpus/tinycc/206_libc_refusals.c | 9 +- tests/corpus/tinycc/206_libc_refusals.expect | 7 +- tests/guest/kernelprobe/src/arch/aarch64.rs | 4 +- tests/guest/kernelprobe/src/unmap_touch.rs | 6 +- tests/guest/osbench/src/mem.rs | 12 +- tests/guest/osbench/src/sys/linux.rs | 5 +- tests/guest/osbench/src/sys/toyos.rs | 44 ++-- tests/guest/osbench/src/system.rs | 8 +- tests/guest/rust-tests/src/arch/mmap.rs | 24 +- .../rust-tests/src/bin/abuse_handle_table.rs | 11 +- .../rust-tests/src/bin/abuse_kernel_addr.rs | 25 +- .../rust-tests/src/bin/abuse_page_straddle.rs | 11 +- .../src/bin/abuse_readonly_copyout.rs | 27 +-- .../rust-tests/src/bin/abuse_spawn_argv.rs | 11 +- .../rust-tests/src/bin/abuse_thread_table.rs | 9 +- .../rust-tests/src/bin/abuse_tls_alloc.rs | 11 +- .../rust-tests/src/bin/futex_wake_counts.rs | 20 +- tests/guest/rust-tests/src/bin/mmap_prot.rs | 90 +++---- tests/guest/rust-tests/src/bin/mmap_stress.rs | 228 +++++++++++++----- tests/guest/rust-tests/src/bin/std_mmap.rs | 6 +- .../rust-tests/src/bin/tlb_shootdown_waits.rs | 20 +- 38 files changed, 790 insertions(+), 527 deletions(-) diff --git a/issues/libc-refuses-what-toyos-cannot-yet-answer.md b/issues/libc-refuses-what-toyos-cannot-yet-answer.md index bb3e04aa613..a4000d786a8 100644 --- a/issues/libc-refuses-what-toyos-cannot-yet-answer.md +++ b/issues/libc-refuses-what-toyos-cannot-yet-answer.md @@ -21,12 +21,9 @@ These answer failure in their POSIX form and do nothing LLVM's `getDefaultStackSize` (`llvm/lib/Support/ProgramStack.cpp`) ignores the failure and reads the `rlimit` it did not get. - `gethostname`, `ENOSYS`: no host name is published to a process. -- `msync` and `mprotect`, `ENOSYS`: no call answers which pages of a range are - mapped, and a mapping's protection is fixed when it is made. -- `mmap` of a file, `ENODEV`, and of executable memory, `ENOTSUP`: the kernel - maps neither. -- `madvise`'s `MADV_DONTNEED`, `ENOSYS`: no call discards a range's pages and - keeps it mapped, so it cannot read back as zeros as Linux's does. +- `msync`, `ENOSYS`: no mapping is of a file, so there is nothing to write back. +- `mmap` of a file, `ENODEV`, and `mmap` or `mprotect` of executable memory, + `ENOTSUP`: the kernel maps neither. - `fcntl` (`sdk/libc/src/fdreq.rs`): a record lock, `EINVAL`, POSIX's answer for a file that supports no locking; `F_GETFL`, `F_SETFL`, `F_GETOWN` and `F_SETOWN`, `ENOSYS`. diff --git a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md index 75b9b203e7c..ed6e0c0f882 100644 --- a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md +++ b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md @@ -75,7 +75,11 @@ live split of a user leaf. hot paths' costs in `issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md`. 4. **Demand-zero anonymous `mmap`.** Built, over 4 KiB pages only: a fault populates one page, unmapping a range no thread touched sends no IPI, and - `munmap` and a `FIXED` `mmap` take any run of pages, as POSIX does. What + `munmap` and a `FIXED` `mmap` take any run of pages, as POSIX does. + `vm_reserve`, `vm_protect`, `vm_discard`, `vm_decommit` and `vm_release` + are the calls, behind libc's `mmap`, `mprotect`, `madvise` and `munmap` + and std's allocator, which trims, and its thread stacks, each reserved + at the size asked with a guard page below. What remains of the stage's exit — `mmap` past what memory can hold refused at `mmap` — is the strict charge of the allocator track's migration (`issues/toyos-has-its-own-allocator.md`). diff --git a/issues/the-capability-end-state-is-twelve-answers.md b/issues/the-capability-end-state-is-twelve-answers.md index 99b4c3ee73c..8e78cf9645b 100644 --- a/issues/the-capability-end-state-is-twelve-answers.md +++ b/issues/the-capability-end-state-is-twelve-answers.md @@ -142,7 +142,7 @@ table** can still reach: - **its own execution** — `SYS_EXIT`, `SYS_THREAD_EXIT`, `SYS_THREAD_SPAWN`, `SYS_THREAD_JOIN`, `SYS_FUTEX_WAIT`/`WAKE`, `SYS_NANOSLEEP`, `SYS_SET_THREAD_NAME`; -- **its own address space** — `SYS_MMAP`, `SYS_MUNMAP`, `SYS_TLS_ALLOC_BLOCK`, +- **its own address space** — `SYS_VM_*`, `SYS_TLS_ALLOC_BLOCK`, `SYS_STACK_INFO`; - **its own record** — `SYS_GETPID`, `SYS_GET_ENV`, `SYS_ENDOWMENTS`, `SYS_QUERY_MODULES`, `SYS_GETCWD`, `SYS_CHDIR`; diff --git a/kernel/pure/vm.rs b/kernel/pure/vm.rs index 9314754c941..12c7f152a7f 100644 --- a/kernel/pure/vm.rs +++ b/kernel/pure/vm.rs @@ -73,8 +73,8 @@ pub enum Kind { /// Memory of this space's own, its frames recorded in the leaves that map /// them. Uncommitted, it is a reservation: no page of it can be touched. Private { committed: bool }, - /// `object`'s bytes from `offset` on. - Object { object: O, offset: u64 }, + /// `object`'s bytes from `offset` on, never given more than `max` allows. + Object { object: O, offset: u64, max: Prot }, } #[derive(Clone, Debug)] @@ -98,7 +98,9 @@ impl Entry { fn tail(&self, start: u64, at: u64) -> Self { let kind = match &self.kind { Kind::Private { committed } => Kind::Private { committed: *committed }, - Kind::Object { object, offset } => Kind::Object { object: object.clone(), offset: offset + (at - start) }, + Kind::Object { object, offset, max } => { + Kind::Object { object: object.clone(), offset: offset + (at - start), max: *max } + } }; Self { end: self.end, prot: self.prot, kind } } @@ -108,8 +110,8 @@ impl Entry { self.prot == next.prot && match (&self.kind, &next.kind) { (Kind::Private { committed: a }, Kind::Private { committed: b }) => a == b, - (Kind::Object { object: a, offset: x }, Kind::Object { object: b, offset: y }) => { - a.same(b) && x + (self.end - start) == *y + (Kind::Object { object: a, offset: x, max: m }, Kind::Object { object: b, offset: y, max: n }) => { + a.same(b) && x + (self.end - start) == *y && m == n } _ => false, } @@ -334,7 +336,7 @@ mod tests { } fn object(end: u64, object: u32, offset: u64) -> Entry { - Entry { end, prot: Prot::ReadWrite, kind: Kind::Object { object: Obj(object), offset } } + Entry { end, prot: Prot::ReadWrite, kind: Kind::Object { object: Obj(object), offset, max: Prot::ReadWrite } } } fn starts(v: &Inventory) -> Vec<(u64, u64)> { diff --git a/kernel/src/mm/space.rs b/kernel/src/mm/space.rs index 00f766a9d66..eb10025d0db 100644 --- a/kernel/src/mm/space.rs +++ b/kernel/src/mm/space.rs @@ -269,7 +269,7 @@ impl AddressSpace { } self.room_at(len, at)?; let start = self.locate(len, at)?; - self.inventory.insert(start, Entry { end: start + len, prot, kind: Kind::Object { object, offset } }); + self.inventory.insert(start, Entry { end: start + len, prot, kind: Kind::Object { object, offset, max: prot } }); self.tables.build(start, start + len); self.usage.made.fetch_add(1, Relaxed); Ok(UserAddr::new(start)) @@ -293,6 +293,91 @@ impl AddressSpace { drop(removed); } + /// Give every page of `[start, start + len)` protection `prot`: a reserved + /// private page is committed unless `prot` is `None`, and a committed one + /// keeps its frame and commitment whatever `prot` is. An object's page + /// takes nothing past what it was mapped with. All or nothing: every page + /// is held (`BadAddress`) and allowed `prot` (`PermissionDenied`) first. + /// Every CPU stops using a present leaf's old protection before this + /// returns, a widening included. + pub fn protect(&mut self, start: UserAddr, len: u64, prot: Prot) -> Result<(), SyscallError> { + let (start, end) = self.held(start, len)?; + self.room(start, end, 0)?; + let allowed = self.inventory.meeting(start, end).all(|(_, entry)| match entry.kind { + Kind::Object { max, .. } => prot.within(max), + Kind::Private { .. } => true, + }); + if !allowed { + return Err(SyscallError::PermissionDenied); + } + let mut commits = false; + self.inventory.update(start, end, |_, entry| { + entry.prot = prot; + if let Kind::Private { committed } = &mut entry.kind { + if !*committed && prot != Prot::None { + *committed = true; + commits = true; + } + } + }); + if commits { + self.tables.build(start, end); + } + let leaf = leaf_prot(prot); + let mut present = false; + self.tables.edit(start, end, |_, old| match (old, leaf) { + (Leaf::Mapped(m), Some(p)) if m.prot == p => None, + (Leaf::Mapped(m), Some(p)) => { + present = true; + Some(Leaf::Mapped(Mapping { prot: p, ..m })) + } + (Leaf::Mapped(m), None) => { + present = true; + Some(if m.owned { Leaf::Held(m.phys) } else { Leaf::Empty }) + } + (Leaf::Held(phys), Some(p)) => Some(Leaf::Mapped(Mapping { phys, prot: p, owned: true, cache: CachePolicy::Normal })), + (Leaf::Held(_), None) | (Leaf::Empty, _) => None, + }); + if present { + self.tables.shoot_down(start, end, &Unlinked::none()); + } + Ok(()) + } + + /// Free every frame behind `[start, start + len)` and keep its ranges as + /// they are: a private page reads zeros at its next touch, an object's + /// page is mapped again from the object. + pub fn discard(&mut self, start: UserAddr, len: u64) -> Result<(), SyscallError> { + let (start, end) = self.held(start, len)?; + self.clear(start, end); + Ok(()) + } + + /// Free every frame and the commitment of `[start, start + len)`, which + /// is all private memory: it is reserved again. + pub fn decommit(&mut self, start: UserAddr, len: u64) -> Result<(), SyscallError> { + let (start, end) = self.held(start, len)?; + self.room(start, end, 0)?; + if !self.inventory.meeting(start, end).all(|(_, entry)| matches!(entry.kind, Kind::Private { .. })) { + return Err(SyscallError::InvalidArgument); + } + self.inventory.update(start, end, |_, entry| { + entry.prot = Prot::None; + entry.kind = Kind::Private { committed: false }; + }); + self.clear(start, end); + Ok(()) + } + + /// `[start, start + len)` as pages every one of which a range holds. + fn held(&self, start: UserAddr, len: u64) -> Result<(u64, u64), SyscallError> { + let end = pages(start.raw(), len).ok_or(SyscallError::InvalidArgument)?; + if !self.inventory.holds(start.raw(), end) || !self.inventory.covered(start.raw(), end) { + return Err(SyscallError::BadAddress); + } + Ok((start.raw(), end)) + } + /// [`release`](Self::release) of every page of `[start, start + len)` /// that still maps `object` where a mapping of it from its start at /// `start` would: a page the process unmapped or replaced since is left as @@ -468,7 +553,7 @@ fn page_leaf(start: u64, entry: &Entry>, page: u64) -> Opti }; Leaf::Mapped(Mapping { phys: frame.into_raw(), prot, owned: true, cache: CachePolicy::Normal }) } - Kind::Object { object, offset } => { + Kind::Object { object, offset, .. } => { Leaf::Mapped(Mapping { phys: object.page(offset + (page - start)), prot, owned: false, cache: object.cache() }) } }) diff --git a/kernel/src/syscall/dispatch.rs b/kernel/src/syscall/dispatch.rs index 595c881be7e..14816405016 100644 --- a/kernel/src/syscall/dispatch.rs +++ b/kernel/src/syscall/dispatch.rs @@ -50,7 +50,10 @@ use super::proc::{ spawn_place, sys_endowments, sys_exit, sys_nanosleep, sys_process_stats, sys_job_create, sys_job_own, sys_process_quit, sys_process_wait, sys_rt_enter, sys_spawn, sys_thread_exit, sys_thread_join, sys_thread_spawn, }; -use super::vm::{shared_image, sys_dlopen, sys_dlsym, sys_mmap, sys_munmap, sys_query_modules, sys_tls_alloc_block}; +use super::vm::{ + shared_image, sys_dlopen, sys_dlsym, sys_query_modules, sys_tls_alloc_block, sys_vm_decommit, sys_vm_discard, + sys_vm_protect, sys_vm_release, sys_vm_reserve, +}; /// `sysret-ss-probe`, run once a boot on the first syscall: a park that /// switches away from it and back. @@ -349,8 +352,11 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> Some(addr) => process::futex_wake(addr, a2), None => bad_addr, }, - SYS_MMAP => sys_mmap(a1, a2, MmapProt(a3), MmapFlags(a4)), - SYS_MUNMAP => sys_munmap(a1, a2), + SYS_VM_RESERVE => sys_vm_reserve(a1, a2, a3, a4), + SYS_VM_RELEASE => sys_vm_release(a1, a2), + SYS_VM_PROTECT => sys_vm_protect(a1, a2, a3), + SYS_VM_DISCARD => sys_vm_discard(a1, a2), + SYS_VM_DECOMMIT => sys_vm_decommit(a1, a2), SYS_PREAD => { let Some(mut buf) = ctx.user_bytes_mut(UserAddr::new(a2), a3) else { return bad_addr }; with_object(RawHandle(a1 as u32), Rights::READ, |o| ops::pread(o, a4, &mut buf)) diff --git a/kernel/src/syscall/vm.rs b/kernel/src/syscall/vm.rs index 1a30aa84f32..a2d0838fa02 100644 --- a/kernel/src/syscall/vm.rs +++ b/kernel/src/syscall/vm.rs @@ -1,5 +1,5 @@ -//! The caller's own address space: `mmap`/`munmap` reserve, commit and take -//! away private memory, `dlopen` maps a library image, `SYS_TLS_ALLOC_BLOCK` +//! The caller's own address space: `vm_*` reserve, commit, protect, discard +//! and take away private memory, `dlopen` maps a library image, `SYS_TLS_ALLOC_BLOCK` //! maps a thread's per-module TLS block, `SYS_QUERY_MODULES` reports what is //! loaded. Exhausting address space is an error return, never an `.expect`. //! @@ -21,56 +21,71 @@ fn whole_pages(size: u64) -> Option { size.checked_next_multiple_of(crate::mm::PAGE_SIZE).filter(|&bytes| bytes != 0) } -/// Private memory honouring `prot`, populated as it is touched: `NONE` -/// reserves the range and nothing more. `MmapFlags::FIXED` with a non-null -/// `req_addr` places it exactly there, in place of whatever those pages held. -pub(super) fn sys_mmap(req_addr: u64, size: u64, prot: MmapProt, flags: MmapFlags) -> u64 { - // First, so the answer is the bit and not whichever other refusal came - // first: a bit this kernel does not define is a request it cannot serve. - if prot.0 & !MmapProt::KNOWN.0 != 0 || flags.0 & !MmapFlags::KNOWN.0 != 0 { - return SyscallError::InvalidArgument.to_u64(); - } - let Some(len) = whole_pages(size) else { - return SyscallError::InvalidArgument.to_u64(); - }; - // Anonymous memory is never executable: `MmapProt` has no bit for it. - let prot = if prot.contains(MmapProt::WRITE) { - Prot::ReadWrite - } else if prot == MmapProt::NONE { - Prot::None - } else { - Prot::Read - }; - let at = match (flags.contains(MmapFlags::FIXED), req_addr) { - (true, addr) if addr != 0 => match UserAddr::checked(addr) { - Some(addr) => At::Fixed(addr), - None => return SyscallError::InvalidArgument.to_u64(), - }, - _ => At::ANYWHERE, - }; - match process::current_address_space().lock().reserve(len, at, prot) { - Ok(at) => at.raw(), - Err(e) => e.to_u64(), +/// A protection a program may give its own private memory. Executable memory +/// is not yet one: `ReadExec` is `NotSupported`. +fn private_prot(word: u64) -> Result { + match VmProt::from_u64(word).ok_or(SyscallError::InvalidArgument)? { + VmProt::None => Ok(Prot::None), + VmProt::Read => Ok(Prot::Read), + VmProt::ReadWrite => Ok(Prot::ReadWrite), + VmProt::ReadExec => Err(SyscallError::NotSupported), } } -/// Take every page of `[addr, addr + size)` away, as POSIX `munmap` does: any -/// run of pages, across mappings and the holes between them. `addr` is on a -/// page and `size` is not zero, or nothing is taken. -pub(super) fn sys_munmap(addr: u64, size: u64) -> u64 { - let (Some(start), Some(len)) = (UserAddr::checked(addr), whole_pages(size)) else { - return SyscallError::InvalidArgument.to_u64(); - }; - if !addr.is_multiple_of(crate::mm::PAGE_SIZE) { - return SyscallError::InvalidArgument.to_u64(); - } - let space = process::current_address_space(); - let mut space = space.lock(); - if let Err(e) = space.room(start.raw(), start.raw() + len, 0) { - return e.to_u64(); +/// `[addr, addr + len)` as a page-aligned start and whole pages. +fn range(addr: u64, len: u64) -> Result<(UserAddr, u64), SyscallError> { + match (UserAddr::checked(addr), whole_pages(len)) { + (Some(start), Some(len)) if addr.is_multiple_of(crate::mm::PAGE_SIZE) => Ok((start, len)), + _ => Err(SyscallError::InvalidArgument), } - space.release(start, len); - 0 +} + +fn answer(done: Result<(), SyscallError>) -> u64 { + done.map_or_else(SyscallError::to_u64, |()| 0) +} + +/// Private memory, committed unless `prot` is `None` and populated as it is +/// touched, placed anywhere at a multiple of `place` or, for `VM_AT_FIXED`, +/// exactly at `place` in place of whatever those pages held. +pub(super) fn sys_vm_reserve(len: u64, prot: u64, place: u64, kind: u64) -> u64 { + let reserved = (|| { + let prot = private_prot(prot)?; + let len = whole_pages(len).ok_or(SyscallError::InvalidArgument)?; + let at = match kind { + VM_AT_ANYWHERE => At::Anywhere(place), + VM_AT_FIXED => At::Fixed(range(place, len)?.0), + _ => return Err(SyscallError::InvalidArgument), + }; + process::current_address_space().lock().reserve(len, at, prot) + })(); + reserved.map_or_else(SyscallError::to_u64, UserAddr::raw) +} + +/// Take every page of `[addr, addr + len)` away, as POSIX `munmap` does. +pub(super) fn sys_vm_release(addr: u64, len: u64) -> u64 { + answer(range(addr, len).and_then(|(start, len)| { + let space = process::current_address_space(); + let mut space = space.lock(); + space.room(start.raw(), start.raw() + len, 0)?; + space.release(start, len); + Ok(()) + })) +} + +pub(super) fn sys_vm_protect(addr: u64, len: u64, prot: u64) -> u64 { + answer((|| { + let prot = private_prot(prot)?; + let (start, len) = range(addr, len)?; + process::current_address_space().lock().protect(start, len, prot) + })()) +} + +pub(super) fn sys_vm_discard(addr: u64, len: u64) -> u64 { + answer(range(addr, len).and_then(|(start, len)| process::current_address_space().lock().discard(start, len))) +} + +pub(super) fn sys_vm_decommit(addr: u64, len: u64) -> u64 { + answer(range(addr, len).and_then(|(start, len)| process::current_address_space().lock().decommit(start, len))) } /// The first `len` bytes of the shared memory object `handle` names, as a diff --git a/sdk/abi/src/syscall.rs b/sdk/abi/src/syscall.rs index 230555e86b7..72d46365c44 100644 --- a/sdk/abi/src/syscall.rs +++ b/sdk/abi/src/syscall.rs @@ -94,8 +94,8 @@ pub const SYS_FUTEX_WAKE: u64 = 59; pub const SYS_FTRUNCATE: u64 = 60; pub const SYS_STACK_INFO: u64 = 61; pub const SYS_CPU_COUNT: u64 = 62; -pub const SYS_MMAP: u64 = 63; -pub const SYS_MUNMAP: u64 = 64; +pub const SYS_VM_RESERVE: u64 = 63; +pub const SYS_VM_RELEASE: u64 = 64; pub const SYS_READ_NONBLOCK: u64 = 66; pub const SYS_WRITE_NONBLOCK: u64 = 67; pub const SYS_EXIT: u64 = 72; @@ -314,6 +314,9 @@ pub const SYS_PREAD: u64 = 138; /// Write a kernel file at an offset, whether or not its handle appends, and /// leaving the handle's offset where it is. See [`pwrite`]. pub const SYS_PWRITE: u64 = 139; +pub const SYS_VM_PROTECT: u64 = 140; +pub const SYS_VM_DISCARD: u64 = 141; +pub const SYS_VM_DECOMMIT: u64 = 142; /// Bins in the per-process syscall profile — one for every number this ABI /// issues, and one at the end for every number it does not. @@ -321,13 +324,13 @@ pub const SYS_PWRITE: u64 = 139; /// **The profile's parts sum to its total, and that is the whole requirement.** /// A bin array narrower than the ABI reaches drops calls out of the line while /// the total goes on counting them. -pub const SYSCALL_PROFILE_BINS: usize = 141; +pub const SYSCALL_PROFILE_BINS: usize = 144; /// Where a number this ABI does not issue is counted. Merging is a degradation /// a reader can see in the line; dropping is one nobody can. pub const SYSCALL_PROFILE_OTHER: usize = SYSCALL_PROFILE_BINS - 1; -const _: () = assert!(SYS_PWRITE < SYSCALL_PROFILE_OTHER as u64); +const _: () = assert!(SYS_VM_DECOMMIT < SYSCALL_PROFILE_OTHER as u64); pub const WNOHANG: u64 = 1; /// Every bit `SYS_PROCESS_WAIT`'s flag word defines; a wait carrying another @@ -697,49 +700,49 @@ impl core::ops::BitOrAssign for OpenFlags { fn bitor_assign(&mut self, rhs: Self) { self.0 |= rhs.0; } } -/// Memory protection flags for [`mmap`]. -/// -/// What a mapping is created with is what it stays. `NONE` reserves the -/// address range and commits nothing: any access to it faults, which is the -/// guard page a libc asks for. Anything without `WRITE` is mapped read-only, -/// and a store to it is a protection violation that kills the process. +/// What pages of a process's own memory may be used for. No value is both +/// writable and executable. `None` pages fault on any access. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct MmapProt(pub u64); - -impl MmapProt { - pub const NONE: Self = Self(0); - pub const READ: Self = Self(1); - pub const WRITE: Self = Self(2); - /// Every bit defined above; [`mmap`] refuses another `InvalidArgument`. - pub const KNOWN: Self = Self(Self::READ.0 | Self::WRITE.0); - - pub const fn contains(self, flag: Self) -> bool { self.0 & flag.0 != 0 } -} - -impl core::ops::BitOr for MmapProt { - type Output = Self; - fn bitor(self, rhs: Self) -> Self { Self(self.0 | rhs.0) } +#[repr(u64)] +pub enum VmProt { + None = 0, + Read = 1, + ReadWrite = 2, + ReadExec = 3, +} + +impl VmProt { + /// The protection a syscall argument names, `None` for a word that names none. + pub const fn from_u64(word: u64) -> Option { + Some(match word { + 0 => Self::None, + 1 => Self::Read, + 2 => Self::ReadWrite, + 3 => Self::ReadExec, + _ => return None, + }) + } } -/// Memory mapping flags for [`mmap`]. +/// Where [`vm_reserve`] places a range. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct MmapFlags(pub u64); - -impl MmapFlags { - pub const ANONYMOUS: Self = Self(1); - pub const PRIVATE: Self = Self(2); - pub const FIXED: Self = Self(4); - /// Every bit defined above; [`mmap`] refuses another `InvalidArgument`. - pub const KNOWN: Self = Self(Self::ANONYMOUS.0 | Self::PRIVATE.0 | Self::FIXED.0); - - pub const fn contains(self, flag: Self) -> bool { self.0 & flag.0 != 0 } +pub enum VmAt { + /// Wherever the space has room, at a multiple of `align`: a power of two, + /// and a page at least. + Anywhere { align: usize }, + /// Exactly there, on a page, in place of whatever those pages held. + Fixed(*mut u8), } -impl core::ops::BitOr for MmapFlags { - type Output = Self; - fn bitor(self, rhs: Self) -> Self { Self(self.0 | rhs.0) } +impl VmAt { + /// Anywhere, on a page. + pub const ANYWHERE: Self = Self::Anywhere { align: PAGE_SIZE }; } +/// `VmAt`'s two kinds as the syscall's fourth argument. +pub const VM_AT_ANYWHERE: u64 = 0; +pub const VM_AT_FIXED: u64 = 1; + /// Result of [`pipe`]: the read and write ends. #[derive(Debug, Clone, Copy)] pub struct PipeEnds { @@ -2346,39 +2349,70 @@ pub fn cpu_count() -> u32 { syscall(SYS_CPU_COUNT, 0, 0, 0, 0) as u32 } -/// Map private memory, zeroed and populated a page at a time as it is -/// touched. Returns its address, or null on failure. -/// -/// `size` is rounded up to whole [`PAGE_SIZE`]s. If `addr` is null, or -/// `MmapFlags::FIXED` is not given, the kernel chooses the address. With -/// `FIXED` and a non-null `addr` on a page, the mapping is placed at exactly -/// that address and replaces whatever those pages held, as POSIX's -/// `MAP_FIXED` does. +/// `len` bytes of private memory, rounded up to whole [`PAGE_SIZE`]s, placed +/// as `at` says. Any `prot` but `None` commits the pages: they read as zeros +/// and are populated as they are touched. `None` reserves them, and +/// [`vm_protect`] commits them later. /// /// # Safety -/// Caller is responsible for managing the returned memory region, and for -/// anything a `FIXED` mapping replaces. -pub unsafe fn mmap(addr: *mut u8, size: usize, prot: MmapProt, flags: MmapFlags) -> *mut u8 { - let result = syscall(SYS_MMAP, addr as u64, size as u64, prot.0, flags.0); - if SyscallError::from_u64(result).is_some() { core::ptr::null_mut() } else { - core::ptr::with_exposed_provenance_mut(result as usize) - } +/// A `Fixed` range replaces whatever those pages held, so nothing may still +/// reach that. +pub unsafe fn vm_reserve(len: usize, prot: VmProt, at: VmAt) -> Result<*mut u8, SyscallError> { + let (place, kind) = match at { + VmAt::Anywhere { align } => (align as u64, VM_AT_ANYWHERE), + VmAt::Fixed(addr) => (addr as u64, VM_AT_FIXED), + }; + check(syscall(SYS_VM_RESERVE, len as u64, prot as u64, place, kind)) + .map(|addr| core::ptr::with_exposed_provenance_mut(addr as usize)) } /// Ranges one address space holds, each a run of pages with one kind and /// protection: a call that could pass it is `ResourceExhausted`. pub const MAX_RANGES: usize = 32_768; -/// Take away every page of `[addr, addr + size)`, as POSIX's `munmap` does: -/// any run of whole pages, across mappings and the holes between them. `addr` -/// off a page, or a `size` of zero, is `InvalidArgument` and takes nothing; -/// so does a space holding nearly [`MAX_RANGES`], `ResourceExhausted`, since -/// cutting a range in two adds one. +/// Take away every page of `[addr, addr + len)`, as POSIX's `munmap` does: any +/// run of whole pages, across ranges and the holes between them. `addr` off a +/// page, or a `len` of zero, is `InvalidArgument` and takes nothing; so does a +/// space holding nearly [`MAX_RANGES`], `ResourceExhausted`, since cutting a +/// range in two adds one. /// /// # Safety /// Nothing reaches those pages after this answers `Ok`. -pub unsafe fn munmap(addr: *mut u8, size: usize) -> Result<(), SyscallError> { - check_unit(syscall(SYS_MUNMAP, addr as u64, size as u64, 0, 0)) +pub unsafe fn vm_release(addr: *mut u8, len: usize) -> Result<(), SyscallError> { + check_unit(syscall(SYS_VM_RELEASE, addr as u64, len as u64, 0, 0)) +} + +/// Give every page of `[addr, addr + len)` the protection `prot`, as POSIX's +/// `mprotect` does. A reserved private page becomes committed unless `prot` is +/// `None`; a committed one keeps its contents and its commitment whatever +/// `prot` is. A page of a shared object takes no protection past what it was +/// mapped with. Every page must be held (`BadAddress` otherwise), and either +/// every page changes or none does. +/// +/// # Safety +/// No live reference into the range relies on an access `prot` takes away. +pub unsafe fn vm_protect(addr: *mut u8, len: usize, prot: VmProt) -> Result<(), SyscallError> { + check_unit(syscall(SYS_VM_PROTECT, addr as u64, len as u64, prot as u64, 0)) +} + +/// Free the frames behind every page of `[addr, addr + len)` and keep the +/// pages committed: the next touch of a private page reads zeros, as Linux's +/// `MADV_DONTNEED` makes it. Every page must be held. +/// +/// # Safety +/// Nothing relies on the private pages' contents. +pub unsafe fn vm_discard(addr: *mut u8, len: usize) -> Result<(), SyscallError> { + check_unit(syscall(SYS_VM_DISCARD, addr as u64, len as u64, 0, 0)) +} + +/// Free the frames and the commitment of every page of `[addr, addr + len)`, +/// which must all be private memory: they are reserved again, faulting on any +/// access until [`vm_protect`] commits them. +/// +/// # Safety +/// Nothing reaches those pages meanwhile. +pub unsafe fn vm_decommit(addr: *mut u8, len: usize) -> Result<(), SyscallError> { + check_unit(syscall(SYS_VM_DECOMMIT, addr as u64, len as u64, 0, 0)) } /// Non-blocking read. Returns bytes read, or `Err(WouldBlock)` if no data available. diff --git a/sdk/libc/arch/src/memory_refusals.rs b/sdk/libc/arch/src/memory_refusals.rs index 29e061701b4..5016fe95a73 100644 --- a/sdk/libc/arch/src/memory_refusals.rs +++ b/sdk/libc/arch/src/memory_refusals.rs @@ -55,19 +55,34 @@ fn posix_madvise_takes_its_five_and_only_them() { } #[test] -fn madvise_takes_its_hints_and_refuses_its_discard() { - use memreq::AdviceRefusal::{Discard, Invalid}; +fn madvise_takes_its_hints_and_discards_for_dontneed() { + use memreq::Advice::{Discard, Hint}; let page = 0x4000_0000; for (addr, advice, want) in [ - (page, 0, None), - (page, 1, None), - (page, 2, None), - (page, 3, None), + (page, 0, Some(Hint)), + (page, 1, Some(Hint)), + (page, 2, Some(Hint)), + (page, 3, Some(Hint)), (page, 4, Some(Discard)), - (page, 5, Some(Invalid)), - (page, -1, Some(Invalid)), - (page + 1, 3, Some(Invalid)), + (page, 5, None), + (page, -1, None), + (page + 1, 3, None), ] { - assert_eq!(memreq::madvise_refusal(addr, advice), want, "madvise({addr:#x}, {advice})"); + assert_eq!(memreq::madvise(addr, advice), want, "madvise({addr:#x}, {advice})"); + } +} + +#[test] +fn a_protection_word_names_one_access_or_is_refused() { + use memreq::Access; + for (prot, want) in [ + (0, Some(Access::None)), + (PROT_READ, Some(Access::Read)), + (PROT_WRITE, Some(Access::ReadWrite)), + (PROT_READ | PROT_WRITE, Some(Access::ReadWrite)), + (PROT_READ | PROT_EXEC, Some(Access::Exec)), + (0x8, None), + ] { + assert_eq!(memreq::access(prot), want, "prot {prot:#x}"); } } diff --git a/sdk/libc/src/lib.rs b/sdk/libc/src/lib.rs index 8db880ef797..be1e48bc27c 100644 --- a/sdk/libc/src/lib.rs +++ b/sdk/libc/src/lib.rs @@ -143,36 +143,36 @@ mod runtime { toyos_abi::syscall::exit(134) } - struct MmapAllocator; + /// dlmalloc's memory: the process's own private pages, committed whole and + /// populated as they are touched, and given back in part when it trims. + struct VmAllocator; - unsafe impl dlmalloc::Allocator for MmapAllocator { + unsafe impl dlmalloc::Allocator for VmAllocator { fn alloc(&self, size: usize) -> (*mut u8, usize, u32) { - use toyos_abi::syscall::{MmapProt, MmapFlags}; - let ptr = unsafe { - toyos_abi::syscall::mmap( - core::ptr::null_mut(), - size, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS, - ) - }; - if ptr.is_null() { (core::ptr::null_mut(), 0, 0) } else { (ptr, size, 0) } + use toyos_abi::syscall::{self, VmAt, VmProt}; + // SAFETY: placed anywhere, so it replaces nothing. + match unsafe { syscall::vm_reserve(size, VmProt::ReadWrite, VmAt::ANYWHERE) } { + Ok(ptr) => (ptr, size, 0), + Err(_) => (core::ptr::null_mut(), 0, 0), + } } fn remap(&self, _ptr: *mut u8, _old: usize, _new: usize, _can_move: bool) -> *mut u8 { core::ptr::null_mut() } - fn free_part(&self, _ptr: *mut u8, _old: usize, _new: usize) -> bool { - false + fn free_part(&self, ptr: *mut u8, old: usize, new: usize) -> bool { + // SAFETY: dlmalloc gives up the tail of a segment it no longer uses. + unsafe { toyos_abi::syscall::vm_release(ptr.wrapping_add(new), old - new).is_ok() } } fn free(&self, ptr: *mut u8, size: usize) -> bool { - unsafe { toyos_abi::syscall::munmap(ptr, size).is_ok() } + // SAFETY: dlmalloc gives up a whole segment it no longer uses. + unsafe { toyos_abi::syscall::vm_release(ptr, size).is_ok() } } fn can_release_part(&self, _flags: u32) -> bool { - false + true } fn allocates_zeros(&self) -> bool { @@ -180,13 +180,13 @@ mod runtime { } fn page_size(&self) -> usize { - 0x1000 + toyos_abi::syscall::PAGE_SIZE } } /// A contended allocation sleeps on the lock's futex rather than spinning. - static DLMALLOC: toyos::sync::Lock> = - toyos::sync::Lock::new(dlmalloc::Dlmalloc::new_with_allocator(MmapAllocator)); + static DLMALLOC: toyos::sync::Lock> = + toyos::sync::Lock::new(dlmalloc::Dlmalloc::new_with_allocator(VmAllocator)); pub(crate) struct LibcAllocator; diff --git a/sdk/libc/src/memreq.rs b/sdk/libc/src/memreq.rs index c3e2bf2b981..4a50f4b4f78 100644 --- a/sdk/libc/src/memreq.rs +++ b/sdk/libc/src/memreq.rs @@ -62,21 +62,48 @@ pub(crate) fn is_advice(advice: i32) -> bool { /// file or as zeros. const MADV_DONTNEED: i32 = 4; -/// Why `madvise` refuses `advice` at `addr`. +/// What Linux's `madvise` does with one of its advice values. #[derive(Debug, PartialEq, Eq)] -pub(crate) enum AdviceRefusal { - /// An address off a page, or advice other than the five `posix_madvise` - /// shares with it: `EINVAL`. - Invalid, - /// `MADV_DONTNEED`'s discard, which nothing here can do: `ENOSYS`. +pub(crate) enum Advice { + /// A hint, which its man page, `madvise(2)`, lets a kernel ignore. + Hint, + /// `MADV_DONTNEED`: the range's pages are discarded. Discard, } -/// Why Linux's `madvise` of `advice` at `addr` is refused, if it is. Its four -/// hints are taken, as its man page, `madvise(2)`, lets a kernel ignore each. -pub(crate) fn madvise_refusal(addr: usize, advice: i32) -> Option { +/// What `madvise` of `advice` at `addr` does; `None` (`EINVAL`) for an +/// address off a page, or advice other than the five `posix_madvise` shares +/// with it. +pub(crate) fn madvise(addr: usize, advice: i32) -> Option { if !addr.is_multiple_of(PAGE) || !is_advice(advice) { - return Some(AdviceRefusal::Invalid); + return None; + } + Some(if advice == MADV_DONTNEED { Advice::Discard } else { Advice::Hint }) +} + +/// What a `PROT_*` word asks of pages. +#[derive(Debug, PartialEq, Eq)] +pub(crate) enum Access { + None, + Read, + /// Writing, which on every machine ToyOS runs on brings reading with it. + ReadWrite, + /// Executing, which no call maps yet: `ENOTSUP`. + Exec, +} + +/// What `prot` asks; `None` (`EINVAL`) for a bit `PROT_*` does not define. +pub(crate) fn access(prot: i32) -> Option { + if prot & !(PROT_READ | PROT_WRITE | PROT_EXEC) != 0 { + return None; } - (advice == MADV_DONTNEED).then_some(AdviceRefusal::Discard) + Some(if prot & PROT_EXEC != 0 { + Access::Exec + } else if prot & PROT_WRITE != 0 { + Access::ReadWrite + } else if prot & PROT_READ != 0 { + Access::Read + } else { + Access::None + }) } diff --git a/sdk/libc/src/posix_io.rs b/sdk/libc/src/posix_io.rs index d487be8f4f1..968454c46aa 100644 --- a/sdk/libc/src/posix_io.rs +++ b/sdk/libc/src/posix_io.rs @@ -850,44 +850,68 @@ pub struct Stat { const NANOS_PER_SEC: u64 = 1_000_000_000; -// mmap/munmap (real implementations using toyos-abi) +// mmap, munmap, mprotect, madvise: the process's own private memory. + +const MAP_FAILED: *mut u8 = usize::MAX as *mut u8; + +/// The protection `prot` names, or the errno it is refused with. +fn vm_prot(prot: i32) -> Result { + use crate::memreq::Access; + match crate::memreq::access(prot) { + None => Err(EINVAL), + // POSIX's ENOTSUP, which is EOPNOTSUPP here. + Some(Access::Exec) => Err(crate::errcode::EOPNOTSUPP), + Some(Access::None) => Ok(syscall::VmProt::None), + Some(Access::Read) => Ok(syscall::VmProt::Read), + Some(Access::ReadWrite) => Ok(syscall::VmProt::ReadWrite), + } +} + +/// POSIX's errno for a refusal of the pages of a range: `ENOMEM` for pages +/// the process does not hold, `EACCES` for a protection a shared mapping was +/// not made with. +fn range_errno(e: syscall::SyscallError) -> i32 { + match e { + syscall::SyscallError::PermissionDenied => crate::errcode::EACCES, + syscall::SyscallError::InvalidArgument => EINVAL, + _ => crate::errcode::ENOMEM, + } +} #[no_mangle] pub unsafe extern "C" fn mmap( addr: *mut u8, len: usize, prot: i32, flags: i32, _fd: i32, _offset: i64, ) -> *mut u8 { use crate::memreq::{self, MapRefusal}; - use toyos_abi::syscall::{MmapProt, MmapFlags}; - const MAP_FAILED: *mut u8 = usize::MAX as *mut u8; if let Some(refusal) = memreq::mmap_refusal(addr.addr(), len, prot, flags) { crate::errno::set(match refusal { MapRefusal::Invalid => EINVAL, MapRefusal::File => crate::errcode::ENODEV, - // POSIX's ENOTSUP, which is EOPNOTSUPP here. MapRefusal::Exec => crate::errcode::EOPNOTSUPP, }); return MAP_FAILED; } - - let mut mp = MmapProt::NONE; - if prot & memreq::PROT_READ != 0 { mp = mp | MmapProt::READ; } - if prot & memreq::PROT_WRITE != 0 { mp = mp | MmapProt::WRITE; } - - let mut mf = MmapFlags::PRIVATE | MmapFlags::ANONYMOUS; - if flags & memreq::MAP_FIXED != 0 { mf = mf | MmapFlags::FIXED; } - - // Asked for in whole pages; more pages than a `usize` counts is the null - // the kernel answers a size it cannot map. - let ptr = memreq::whole_pages(len) - .map_or(core::ptr::null_mut(), |len| unsafe { syscall::mmap(addr, len, mp, mf) }); - if ptr.is_null() { - // The kernel's refusal does not say which it is: ENOMEM is POSIX's - // for a place or size the address space does not allow. - crate::errno::set(crate::errcode::ENOMEM); - MAP_FAILED - } else { - ptr + let prot = match vm_prot(prot) { + Ok(prot) => prot, + Err(errno) => { + crate::errno::set(errno); + return MAP_FAILED; + } + }; + // POSIX takes a null address with `MAP_FIXED` as the address zero, which + // no process holds; anything else without it is only a hint. + let at = if flags & memreq::MAP_FIXED != 0 { syscall::VmAt::Fixed(addr) } else { syscall::VmAt::ANYWHERE }; + // SAFETY: a `MAP_FIXED` caller gives up what those pages held, as POSIX says. + match unsafe { syscall::vm_reserve(len, prot, at) } { + Ok(ptr) => ptr, + Err(e) => { + crate::errno::set(match e { + syscall::SyscallError::InvalidArgument => EINVAL, + _ => crate::errcode::ENOMEM, + }); + MAP_FAILED + } } } @@ -897,17 +921,42 @@ pub unsafe extern "C" fn posix_madvise(_addr: *mut u8, _len: usize, advice: i32) if crate::memreq::is_advice(advice) { 0 } else { EINVAL } } -/// Linux's `madvise`, which POSIX does not define (`memreq::madvise_refusal`). +/// Linux's `madvise`, which POSIX does not define: its hints are taken and +/// ignored, and `MADV_DONTNEED` discards the range, which then reads zeros. #[no_mangle] -pub unsafe extern "C" fn madvise(addr: *mut u8, _len: usize, advice: i32) -> i32 { - use crate::memreq::AdviceRefusal; - match crate::memreq::madvise_refusal(addr.addr(), advice) { - None => 0, - Some(refusal) => { - crate::errno::set(match refusal { - AdviceRefusal::Invalid => EINVAL, - AdviceRefusal::Discard => crate::errcode::ENOSYS, - }); +pub unsafe extern "C" fn madvise(addr: *mut u8, len: usize, advice: i32) -> i32 { + use crate::memreq::Advice; + let done = match crate::memreq::madvise(addr.addr(), advice) { + None => Err(EINVAL), + Some(Advice::Hint) => Ok(()), + Some(Advice::Discard) => match crate::memreq::whole_pages(len) { + None => Err(EINVAL), + // SAFETY: the caller's, as Linux's `madvise` leaves it. + Some(len) => unsafe { syscall::vm_discard(addr, len) }.map_err(range_errno), + }, + }; + match done { + Ok(()) => 0, + Err(errno) => { + crate::errno::set(errno); + -1 + } + } +} + +/// POSIX's `mprotect`: every whole page `[addr, addr + len)` reaches into +/// takes `prot`, or none does. +#[no_mangle] +pub unsafe extern "C" fn mprotect(addr: *mut u8, len: usize, prot: i32) -> i32 { + let done = vm_prot(prot).and_then(|prot| match crate::memreq::whole_pages(len) { + None => Err(EINVAL), + // SAFETY: the caller's, as POSIX's `mprotect` leaves it. + Some(len) => unsafe { syscall::vm_protect(addr, len, prot) }.map_err(range_errno), + }); + match done { + Ok(()) => 0, + Err(errno) => { + crate::errno::set(errno); -1 } } @@ -923,7 +972,7 @@ pub unsafe extern "C" fn munmap(addr: *mut u8, len: usize) -> i32 { let unmapped = match crate::memreq::whole_pages(len) { None => Err(EINVAL), // SAFETY: the caller's, as C's `munmap` leaves it. - Some(len) => unsafe { syscall::munmap(addr, len) }.map_err(|e| match e { + Some(len) => unsafe { syscall::vm_release(addr, len) }.map_err(|e| match e { syscall::SyscallError::ResourceExhausted => crate::errcode::ENOMEM, _ => EINVAL, }), diff --git a/sdk/libc/src/refused.rs b/sdk/libc/src/refused.rs index c2708f9e889..3ec65d6dc44 100644 --- a/sdk/libc/src/refused.rs +++ b/sdk/libc/src/refused.rs @@ -117,8 +117,3 @@ pub unsafe extern "C" fn msync(_addr: *mut u8, _len: usize, _flags: i32) -> i32 refuse() } -/// A mapping's protection is fixed when it is made. -#[no_mangle] -pub unsafe extern "C" fn mprotect(_addr: *mut u8, _len: usize, _prot: i32) -> i32 { - refuse() -} diff --git a/sdk/std/sys/alloc.rs b/sdk/std/sys/alloc.rs index 571b6ee288a..cc8fd1d6a0b 100644 --- a/sdk/std/sys/alloc.rs +++ b/sdk/std/sys/alloc.rs @@ -1,50 +1,48 @@ use core::cell::SyncUnsafeCell; -use toyos_abi::syscall::{self, MmapFlags, MmapProt}; +use toyos_abi::syscall::{self, VmAt, VmProt}; use crate::alloc::Layout; use crate::ptr; use crate::sys::sync::Mutex; -/// dlmalloc backing allocator that provides memory via mmap/munmap syscalls. +/// dlmalloc's memory: the process's own private pages, committed whole and +/// populated as they are touched, and given back in part when it trims. struct ToyOsAllocator; unsafe impl dlmalloc::Allocator for ToyOsAllocator { fn alloc(&self, size: usize) -> (*mut u8, usize, u32) { - let ptr = unsafe { - syscall::mmap( - ptr::null_mut(), - size, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS, - ) - }; - if ptr.is_null() { (ptr::null_mut(), 0, 0) } else { (ptr, size, 0) } + // SAFETY: placed anywhere, so it replaces nothing. + match unsafe { syscall::vm_reserve(size, VmProt::ReadWrite, VmAt::ANYWHERE) } { + Ok(ptr) => (ptr, size, 0), + Err(_) => (ptr::null_mut(), 0, 0), + } } fn remap(&self, _ptr: *mut u8, _oldsize: usize, _newsize: usize, _can_move: bool) -> *mut u8 { - // No mremap equivalent ptr::null_mut() } - fn free_part(&self, _ptr: *mut u8, _oldsize: usize, _newsize: usize) -> bool { - false + fn free_part(&self, ptr: *mut u8, oldsize: usize, newsize: usize) -> bool { + // SAFETY: dlmalloc gives up the tail of a segment it no longer uses. + unsafe { syscall::vm_release(ptr.wrapping_add(newsize), oldsize - newsize).is_ok() } } fn free(&self, ptr: *mut u8, size: usize) -> bool { - unsafe { syscall::munmap(ptr, size).is_ok() } + // SAFETY: dlmalloc gives up a whole segment it no longer uses. + unsafe { syscall::vm_release(ptr, size).is_ok() } } fn can_release_part(&self, _flags: u32) -> bool { - false + true } fn allocates_zeros(&self) -> bool { - true // mmap returns zeroed pages + true } fn page_size(&self) -> usize { - 0x1000 + syscall::PAGE_SIZE } } diff --git a/sdk/std/sys/thread.rs b/sdk/std/sys/thread.rs index 7e810716e19..6b536a7ebf9 100644 --- a/sdk/std/sys/thread.rs +++ b/sdk/std/sys/thread.rs @@ -1,6 +1,5 @@ use toyos_abi::syscall; -use crate::alloc::Layout; use crate::cell::UnsafeCell; use crate::ffi::CStr; use crate::io; @@ -25,8 +24,9 @@ unsafe impl Sync for Thread {} struct Block { tid: AtomicU32, state: AtomicU32, - stack: *mut u8, - layout: Layout, + /// The stack's reservation: [`GUARD`] reserved, then the stack committed. + reservation: *mut u8, + reserved: usize, /// Taken by the thread as it starts. init: UnsafeCell>>, /// The next block on [`REAPED`]. @@ -43,6 +43,11 @@ static REAPED: AtomicPtr = AtomicPtr::new(ptr::null_mut()); pub const DEFAULT_MIN_STACK_SIZE: usize = 2 * 1024 * 1024; +/// Reserved pages below a thread's stack, so that running off it faults: one +/// page is enough, since every frame larger than a page probes each page it +/// spans (the target's inline stack probes). +const GUARD: usize = syscall::PAGE_SIZE; + /// Wait for `block`'s thread to be gone, then free its stack and `block`. A /// join the kernel refuses frees nothing, since the stack may still be in use. /// @@ -52,7 +57,7 @@ unsafe fn reap(block: *mut Block) { let b = unsafe { &*block }; if syscall::thread_join(b.tid.load(Ordering::Relaxed) as u64) == 0 { unsafe { - crate::alloc::dealloc(b.stack, b.layout); + let _ = syscall::vm_release(b.reservation, b.reserved); drop(Box::from_raw(block)); } } @@ -68,21 +73,30 @@ impl Thread { ended = next; } - let stack_size = stack.max(DEFAULT_MIN_STACK_SIZE); - let layout = Layout::from_size_align(stack_size, 4096).unwrap(); - let stack_base = unsafe { crate::alloc::alloc(layout) }; - if stack_base.is_null() { - return Err(io::const_error!( - io::ErrorKind::OutOfMemory, - "thread stack allocation failed" - )); + // The size asked for, in whole pages, committed before the thread + // starts so that no touch of it can find no memory. + let Some(stack_size) = stack.max(1).checked_next_multiple_of(syscall::PAGE_SIZE) else { + return Err(io::const_error!(io::ErrorKind::InvalidInput, "thread stack size too large")); + }; + let reserved = stack_size.checked_add(GUARD).ok_or(io::const_error!( + io::ErrorKind::InvalidInput, + "thread stack size too large" + ))?; + // SAFETY: placed anywhere, so it replaces nothing, and the stack is + // inside the reservation. + let reservation = unsafe { syscall::vm_reserve(reserved, syscall::VmProt::None, syscall::VmAt::ANYWHERE) } + .map_err(|_| io::const_error!(io::ErrorKind::OutOfMemory, "no room for a thread stack"))?; + let stack_base = reservation.wrapping_add(GUARD); + if unsafe { syscall::vm_protect(stack_base, stack_size, syscall::VmProt::ReadWrite) }.is_err() { + let _ = unsafe { syscall::vm_release(reservation, reserved) }; + return Err(io::const_error!(io::ErrorKind::OutOfMemory, "thread stack commit failed")); } let stack_top = stack_base.expose_provenance() as u64 + stack_size as u64; let block = Box::into_raw(Box::new(Block { tid: AtomicU32::new(0), state: AtomicU32::new(JOINABLE), - stack: stack_base, - layout, + reservation, + reserved, init: UnsafeCell::new(Some(init)), next: ptr::null_mut(), })); @@ -100,7 +114,7 @@ impl Thread { if syscall::SyscallError::from_u64(tid).is_some() { unsafe { drop(Box::from_raw(block)); - crate::alloc::dealloc(stack_base, layout); + let _ = syscall::vm_release(reservation, reserved); } return Err(io::const_error!(io::ErrorKind::Uncategorized, "thread spawn failed")); } diff --git a/sdk/toyos/src/net/stream.rs b/sdk/toyos/src/net/stream.rs index 3fb27cfd580..8d6e2c2d28f 100644 --- a/sdk/toyos/src/net/stream.rs +++ b/sdk/toyos/src/net/stream.rs @@ -306,7 +306,7 @@ fn free(slot: usize, end: End) { if let Some((_, base)) = end.region { // SAFETY: the halves that read the mapping are gone, and nothing else // reaches it. - let _ = unsafe { syscall::munmap(base as *mut u8, REGION_BYTES) }; + let _ = unsafe { syscall::vm_release(base as *mut u8, REGION_BYTES) }; } let region = end.region.map(|(region, _)| region); for handle in [Some(end.connection), Some(end.receive_bell), Some(end.send_bell), region].into_iter().flatten() { @@ -973,7 +973,7 @@ fn take_up(connection: OwnedHandle, bells: [RawHandle; BELLS], region: RawHandle (connection, receive, send, region) => { // SAFETY: nothing has read the mapping through this end yet, and // nothing else reaches it. - let _ = unsafe { syscall::munmap(base, REGION_BYTES) }; + let _ = unsafe { syscall::vm_release(base, REGION_BYTES) }; let connection = connection.map(|(descriptor, own)| [descriptor, own]).into_iter().flatten(); connection.chain(receive).chain(send).chain(region).for_each(syscall::close); Err(NetError::Io) diff --git a/sdk/toyos/src/shm.rs b/sdk/toyos/src/shm.rs index af21ab3246f..c0292747f91 100644 --- a/sdk/toyos/src/shm.rs +++ b/sdk/toyos/src/shm.rs @@ -89,7 +89,7 @@ impl SharedMemory { } /// Give up ownership: the handle and its mapping stay, the mapping for - /// the caller to take away with `munmap` of [`len`](Self::len) bytes + /// the caller to take away with `vm_release` of [`len`](Self::len) bytes /// before it closes the handle. pub fn into_raw(self) -> (RawHandle, *mut u8) { let this = core::mem::ManuallyDrop::new(self); @@ -127,7 +127,7 @@ impl Drop for SharedMemory { fn drop(&mut self) { // SAFETY: the mapping `map` made, which nothing reaches once this drops; // the handle closes after it. - let _ = unsafe { syscall::munmap(self.ptr, self.mapped) }; + let _ = unsafe { syscall::vm_release(self.ptr, self.mapped) }; } } diff --git a/tests/corpus/tinycc/206_libc_refusals.c b/tests/corpus/tinycc/206_libc_refusals.c index 0dafe653b85..557a4a4bb50 100644 --- a/tests/corpus/tinycc/206_libc_refusals.c +++ b/tests/corpus/tinycc/206_libc_refusals.c @@ -83,7 +83,8 @@ int main(void) { said("getrlimit", getrlimit(RLIMIT_STACK, &limit)); said("setrlimit", setrlimit(RLIMIT_CORE, &limit)); said("msync", msync(page, 4096, MS_SYNC)); - said("mprotect", mprotect(page, 4096, PROT_READ)); + said("mprotect executable", mprotect(page, 4096, PROT_READ | PROT_EXEC)); + said("mprotect off a page", mprotect(page + 1, 4096, PROT_READ)); said("utimes", utimes(FILE_PATH, NULL)); int status = 0; said("wait", wait(&status)); @@ -179,8 +180,8 @@ int main(void) { said("sysconf 12345", sysconf(12345)); /* posix_madvise takes every advice and answers an error number; madvise - takes its hints and refuses MADV_DONTNEED's discard, which would have - zeroed the page. */ + takes its hints, and MADV_DONTNEED discards the page, which then reads + zeros. */ page[0] = 'x'; printf("posix_madvise WILLNEED: %d\n", posix_madvise(page, 4096, POSIX_MADV_WILLNEED)); printf("posix_madvise 99: %s\n", errno_name(posix_madvise(page, 4096, 99))); @@ -188,7 +189,7 @@ int main(void) { said("madvise DONTNEED", madvise(page, 4096, MADV_DONTNEED)); said("madvise off a page", madvise(page + 1, 4096, MADV_WILLNEED)); said("madvise 99", madvise(page, 4096, 99)); - printf("page after madvise: %c\n", page[0]); + printf("page after madvise: %s\n", page[0] == 0 ? "zeros" : "its old bytes"); /* Nothing refused was done: the file has its one byte, its mode through its name and its descriptor, and no second name; the page is still diff --git a/tests/corpus/tinycc/206_libc_refusals.expect b/tests/corpus/tinycc/206_libc_refusals.expect index 8047297dfcf..12c39fdecba 100644 --- a/tests/corpus/tinycc/206_libc_refusals.expect +++ b/tests/corpus/tinycc/206_libc_refusals.expect @@ -13,7 +13,8 @@ fstatvfs: -1, ENOSYS getrlimit: -1, ENOSYS setrlimit: -1, ENOSYS msync: -1, ENOSYS -mprotect: -1, ENOSYS +mprotect executable: -1, ENOTSUP +mprotect off a page: -1, EINVAL utimes: -1, ENOTSUP wait: -1, ECHILD wait4: -1, ECHILD @@ -60,10 +61,10 @@ sysconf 12345: -1, EINVAL posix_madvise WILLNEED: 0 posix_madvise 99: EINVAL madvise WILLNEED: 0, no errno -madvise DONTNEED: -1, ENOSYS +madvise DONTNEED: 0, no errno madvise off a page: -1, EINVAL madvise 99: -1, EINVAL -page after madvise: x +page after madvise: zeros file: 1 bytes; mode as it was through fstat, as it was through stat; second name: none page after mprotect: y limit after getrlimit: 0 0 diff --git a/tests/guest/kernelprobe/src/arch/aarch64.rs b/tests/guest/kernelprobe/src/arch/aarch64.rs index a07a1cc54ba..719bfd7963f 100644 --- a/tests/guest/kernelprobe/src/arch/aarch64.rs +++ b/tests/guest/kernelprobe/src/arch/aarch64.rs @@ -238,7 +238,7 @@ pub mod first_entry { } } -/// Reads the word at `page`, unmaps the `len` bytes there with `SYS_MUNMAP` +/// Reads the word at `page`, unmaps the `len` bytes there with `SYS_VM_RELEASE` /// and reads the word again, in one block, so nothing but the unmap itself /// can switch the CPU between the reads. Answers the first read, the unmap's /// answer and the second read. @@ -259,7 +259,7 @@ pub unsafe fn read_unmap_read(page: *const u64, len: usize) -> (u64, u64, u64) { page = in(reg) page, first = out(reg) first, second = lateout(reg) second, - inlateout("x0") toyos_abi::syscall::SYS_MUNMAP => answer, + inlateout("x0") toyos_abi::syscall::SYS_VM_RELEASE => answer, in("x1") page, in("x2") len, in("x3") 0u64, diff --git a/tests/guest/kernelprobe/src/unmap_touch.rs b/tests/guest/kernelprobe/src/unmap_touch.rs index 6fffa672340..4cd65af08e9 100644 --- a/tests/guest/kernelprobe/src/unmap_touch.rs +++ b/tests/guest/kernelprobe/src/unmap_touch.rs @@ -11,7 +11,7 @@ use std::process::Command; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering::{Acquire, Release}}; use toyos_abi::end::{End, Fault}; -use toyos_abi::syscall::{mmap, munmap, MmapFlags, MmapProt}; +use toyos_abi::syscall::{self, VmAt, VmProt}; const TRIALS: usize = 4; const PAGE: usize = 2 * 1024 * 1024; @@ -60,7 +60,7 @@ fn judge() { /// A fresh page, written. fn written() -> *mut u8 { // SAFETY: a fresh anonymous mapping this function alone names. - let page = unsafe { mmap(core::ptr::null_mut(), PAGE, MmapProt::READ | MmapProt::WRITE, MmapFlags::ANONYMOUS | MmapFlags::PRIVATE) }; + let page = unsafe { syscall::vm_reserve(PAGE, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!page.is_null(), "unmap_touch: mmap refused"); // SAFETY: inside the mapping just made. unsafe { page.write_volatile(0x5A) }; @@ -87,7 +87,7 @@ fn seen() { } println!("{UNMAPPING}"); // SAFETY: the mapping made above, whole; its reader's next read is the probe. - unsafe { munmap(page, PAGE) }.expect("unmap_touch: munmap refused"); + unsafe { syscall::vm_release(page, PAGE) }.expect("unmap_touch: munmap refused"); UNMAPPED.store(true, Release); // The reader's fault ends this process; this thread waits for it. loop { diff --git a/tests/guest/osbench/src/mem.rs b/tests/guest/osbench/src/mem.rs index 04ace337fcd..03afc6f420e 100644 --- a/tests/guest/osbench/src/mem.rs +++ b/tests/guest/osbench/src/mem.rs @@ -2,13 +2,11 @@ //! and giving back part of a mapping, the allocator under many threads, and a //! large `realloc`. //! -//! **Where the cost lands differs by design.** ToyOS commits, zeroes and maps -//! an anonymous range whole in 2 MiB pages inside `mmap`, so a first touch -//! takes no fault there; Linux faults each page on first touch, in 4 KiB -//! pages where huge pages are refused and in 2 MiB pages where they are asked -//! for and given. `map_touch_unmap` is the like-for-like total. The -//! allocator is each system's own: ToyOS std's dlmalloc under one lock, -//! Linux's static musl malloc. +//! Both systems fault each anonymous page in on its first touch: ToyOS in +//! 4 KiB pages always, Linux in 4 KiB pages where huge pages are refused and +//! in 2 MiB pages where they are asked for and given. `map_touch_unmap` is the +//! whole cost. The allocator is each system's own: ToyOS std's dlmalloc +//! under one lock, Linux's static musl malloc. use std::alloc::{alloc, dealloc, realloc, Layout}; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; diff --git a/tests/guest/osbench/src/sys/linux.rs b/tests/guest/osbench/src/sys/linux.rs index 739ee9c7371..b673c355a27 100644 --- a/tests/guest/osbench/src/sys/linux.rs +++ b/tests/guest/osbench/src/sys/linux.rs @@ -66,9 +66,8 @@ pub unsafe fn unmap(ptr: *mut u8, len: usize) { assert_eq!(unsafe { libc::munmap(ptr.cast(), len) }, 0); } -/// An anonymous mapping with transparent huge pages asked for, which -/// ToyOS's anonymous mappings always are; [`huge_backing`] says what the -/// kernel gave. +/// An anonymous mapping with transparent huge pages asked for; +/// [`huge_backing`] says what the kernel gave. pub fn map_anon_huge(len: usize) -> *mut u8 { let ptr = map_anon(len); // SAFETY: an advice changes no contents. diff --git a/tests/guest/osbench/src/sys/toyos.rs b/tests/guest/osbench/src/sys/toyos.rs index 4c1accefb30..cfb9a56a7aa 100644 --- a/tests/guest/osbench/src/sys/toyos.rs +++ b/tests/guest/osbench/src/sys/toyos.rs @@ -1,6 +1,6 @@ use std::sync::atomic::AtomicU32; -use toyos_abi::syscall::{self, MmapFlags, MmapProt}; +use toyos_abi::syscall::{self, VmAt, VmProt}; /// ToyOS has no call that says which CPU runs a thread, so CPUs are named as /// the architecture's own instructions name them. @@ -29,23 +29,21 @@ pub fn futex_wake(word: &AtomicU32, count: u32) -> u32 { u32::try_from(woken).expect("a wake count fits the count asked for") } -/// ToyOS commits, zeroes and maps an anonymous range whole, in 2 MiB pages, -/// before `mmap` answers. +/// ToyOS commits an anonymous range whole and populates it a 4 KiB page at a +/// time as it is touched. pub fn map_anon(len: usize) -> *mut u8 { // SAFETY: a fresh anonymous mapping aliases nothing. - let ptr = - unsafe { syscall::mmap(std::ptr::null_mut(), len, MmapProt::READ | MmapProt::WRITE, MmapFlags::ANONYMOUS) }; - assert!(!ptr.is_null(), "mmap of {len} bytes was refused"); - ptr + unsafe { syscall::vm_reserve(len, VmProt::ReadWrite, VmAt::ANYWHERE) } + .unwrap_or_else(|e| panic!("vm_reserve of {len} bytes was refused: {e:?}")) } -/// Every ToyOS anonymous mapping is in 2 MiB pages. +/// ToyOS maps no user memory in large pages yet: the same 4 KiB mapping. pub fn map_anon_huge(len: usize) -> *mut u8 { map_anon(len) } pub fn huge_backing() -> String { - "all".into() + "0KiB".into() } /// # Safety @@ -53,7 +51,7 @@ pub fn huge_backing() -> String { /// reaches it afterwards. pub unsafe fn unmap(ptr: *mut u8, len: usize) { // SAFETY: as the caller promises. - unsafe { syscall::munmap(ptr, len) }.expect("munmap of a whole mapping"); + unsafe { syscall::vm_release(ptr, len) }.expect("vm_release of a whole mapping"); } /// Nothing a program can read: the kernel's frequency request and the @@ -73,9 +71,8 @@ pub fn log_time_ns(monotonic: u64) -> u64 { monotonic + toyos_abi::clock::page().stamp_at_boot } -/// ToyOS raises every thread stack to std's floor (`sys/thread.rs`'s -/// `DEFAULT_MIN_STACK_SIZE`), whatever the builder asks. -pub const STACK_FLOOR: &str = "2MiB"; +/// ToyOS gives a thread the stack its builder asks for, in whole pages. +pub const STACK_FLOOR: &str = "none"; /// What [`current_cpu`] names a CPU by, for the fields that print one. pub const CPU_NAMING: &str = "apic"; @@ -342,30 +339,21 @@ pub mod diag { } } -/// ToyOS has no call that changes a mapping's protection: `toyos_abi` -/// declares none. -pub fn protect(_ptr: *mut u8, _len: usize, _writable: bool) -> Result<(), super::Refused> { - Err(super::Refused::Absent("toyos_abi has no mprotect: a mapping keeps the protection mmap gave it".into())) +pub fn protect(ptr: *mut u8, len: usize, writable: bool) -> Result<(), super::Refused> { + let prot = if writable { VmProt::ReadWrite } else { VmProt::Read }; + // SAFETY: the caller's mapping; a protection change moves no memory. + unsafe { syscall::vm_protect(ptr, len, prot) }.map_err(|e| super::Refused::Failed(format!("vm_protect: {e:?}"))) } -/// Give back part of a mapping, which ToyOS refuses: `munmap` takes a whole -/// mapping (`toyos_abi::syscall::munmap`). -/// /// # Safety /// `ptr` and `len` lie inside one mapping from [`map_anon`] that nothing /// reaches in that range afterwards. pub unsafe fn unmap_part(ptr: *mut u8, len: usize) -> Result<(), super::Refused> { // SAFETY: as the caller promises. - unsafe { syscall::munmap(ptr, len) }.map_err(|e| match e { - syscall::SyscallError::InvalidArgument => { - super::Refused::Absent("munmap takes a whole mapping and answers InvalidArgument for a part".into()) - } - e => super::Refused::Failed(format!("munmap of a part answered {e:?}")), - }) + unsafe { syscall::vm_release(ptr, len) }.map_err(|e| super::Refused::Failed(format!("vm_release: {e:?}"))) } -/// A mapping whose first touch of each 4 KiB page is what is measured: every -/// ToyOS anonymous mapping is in 2 MiB pages, committed inside `mmap`. +/// A mapping that faults in 4 KiB pages, which every ToyOS anonymous mapping does. pub fn map_anon_4k(len: usize) -> *mut u8 { map_anon(len) } diff --git a/tests/guest/osbench/src/system.rs b/tests/guest/osbench/src/system.rs index 2c064ffd78d..a49c83fcfe9 100644 --- a/tests/guest/osbench/src/system.rs +++ b/tests/guest/osbench/src/system.rs @@ -142,9 +142,7 @@ pub fn pipe_throughput(size: usize) { /// Anonymous memory made usable and given back, ns per MiB of a 32 MiB /// mapping: the `mmap`, a write to every 4 KiB page, and the `munmap`, each -/// timed alone and summed. Where the cost lands differs by design — Linux faults -/// each page on first touch, ToyOS commits and zeroes the range in `mmap` — -/// so only `total` compares like for like. One unread round, then ten. +/// timed alone and summed. Both systems fault each page in on its first touch. One unread round, then ten. pub fn anon_memory() { const LEN: usize = 32 * MIB; const MIBS: u64 = (LEN / MIB) as u64; @@ -197,8 +195,8 @@ pub fn map_unmap_small() { /// stores a pattern, `copy` stores the source xor a constant — a loop the /// compiler cannot turn into a `memcpy` call, so both systems run the same /// instructions — and `memcpy` is `copy_from_slice`, each system's own. -/// Linux is asked for huge pages, which every ToyOS anonymous mapping is; the -/// line says how much Linux gave. One unread round, then seven, per kind. +/// Both are asked for huge pages; the line says how much each gave, and +/// ToyOS gives none yet. One unread round, then seven, per kind. pub fn bandwidth() { const LEN: usize = 64 * MIB; let mut src = Mapping::huge(LEN); diff --git a/tests/guest/rust-tests/src/arch/mmap.rs b/tests/guest/rust-tests/src/arch/mmap.rs index dc4aa7cb12e..c6fc675a1a2 100644 --- a/tests/guest/rust-tests/src/arch/mmap.rs +++ b/tests/guest/rust-tests/src/arch/mmap.rs @@ -1,30 +1,30 @@ -//! `SYS_MMAP` past `toyos_abi::syscall::mmap`, which answers every refusal with -//! a null: here the kernel's error comes back by name. +//! `SYS_VM_RESERVE` with words `toyos_abi::syscall::vm_reserve`'s types +//! cannot spell, so a test can ask with an undefined one. #[cfg(target_arch = "x86_64")] pub use x86_64::*; #[cfg(target_arch = "x86_64")] mod x86_64 { - use toyos_abi::syscall::{MmapFlags, MmapProt, SyscallError, SYS_MMAP}; + use toyos_abi::syscall::{SyscallError, SYS_VM_RESERVE}; - /// The mapping's address, or the error the kernel refused it with. + /// The range's address, or the error the kernel refused it with. /// /// # Safety - /// A `FIXED` mapping replaces whatever this process has at `addr`. - pub unsafe fn raw(addr: u64, len: u64, prot: MmapProt, flags: MmapFlags) -> Result { + /// A fixed range replaces whatever this process has at `place`. + pub unsafe fn raw(len: u64, prot: u64, place: u64, kind: u64) -> Result { let ret: u64; - // SAFETY: the caller's for what a `FIXED` mapping replaces; no argument + // SAFETY: the caller's for what a fixed range replaces; no argument // is a pointer this call dereferences, and the `syscall` is the ABI's // (`toyos_abi::syscall`), which clobbers rax, rcx and r11. unsafe { core::arch::asm!( "syscall", - in("rdi") SYS_MMAP, - in("rsi") addr, - in("rdx") len, - in("r8") prot.0, - in("r9") flags.0, + in("rdi") SYS_VM_RESERVE, + in("rsi") len, + in("rdx") prot, + in("r8") place, + in("r9") kind, lateout("rax") ret, out("rcx") _, out("r11") _, diff --git a/tests/guest/rust-tests/src/bin/abuse_handle_table.rs b/tests/guest/rust-tests/src/bin/abuse_handle_table.rs index 5ea1bd48bce..f299182b640 100644 --- a/tests/guest/rust-tests/src/bin/abuse_handle_table.rs +++ b/tests/guest/rust-tests/src/bin/abuse_handle_table.rs @@ -32,7 +32,7 @@ mod image; use toyos_abi::syscall::{ - self, EndowEntry, MmapFlags, MmapProt, SpawnArgs, SyscallError, MAX_SPAWN_ENDOWMENTS, QUIT_LABEL, SELF_LABEL, + self, EndowEntry, VmAt, VmProt, SpawnArgs, SyscallError, MAX_SPAWN_ENDOWMENTS, QUIT_LABEL, SELF_LABEL, }; use toyos_abi::RawHandle; use toyos_abi::end::End; @@ -68,12 +68,7 @@ fn main() { } own(); let region = unsafe { - syscall::mmap( - core::ptr::null_mut(), - REGION, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(REGION, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!region.is_null(), "mmap failed"); @@ -157,7 +152,7 @@ fn main() { let reused = syscall::dup(stdout()).expect("dup must work again after closing handles"); syscall::close(reused); - unsafe { syscall::munmap(region, REGION) }.expect("munmap"); + unsafe { syscall::vm_release(region, REGION) }.expect("munmap"); println!("handle table capped at {MAX_HANDLES} on every insert path (refused at {n})"); } diff --git a/tests/guest/rust-tests/src/bin/abuse_kernel_addr.rs b/tests/guest/rust-tests/src/bin/abuse_kernel_addr.rs index 039d08e717e..d44c40a9e95 100644 --- a/tests/guest/rust-tests/src/bin/abuse_kernel_addr.rs +++ b/tests/guest/rust-tests/src/bin/abuse_kernel_addr.rs @@ -18,7 +18,7 @@ //! (`SYS_DEBUG` actions 10 and 11): where they are, and whether they still say //! what it put there. -use toyos_abi::syscall::{self, DlopenArgs, MmapFlags, MmapProt, SyscallError}; +use toyos_abi::syscall::{self, DlopenArgs, VmAt, VmProt, SyscallError}; use toyos_abi::syscall::debug_action::{CANARY_ADDR, CANARY_CHANGED}; @@ -26,7 +26,7 @@ use toyos_abi::syscall::debug_action::{CANARY_ADDR, CANARY_CHANGED}; /// an `init_array` — so a successful call has something to report. const LIB: &[u8] = b"/system/lib/libtls_dlopen_lib.so"; -const PAGE_2M: u64 = 2 * 1024 * 1024; +const PAGE: u64 = syscall::PAGE_SIZE as u64; /// `dl_open` builds its own `init_info` on the stack, so the typed wrapper /// cannot express the argument under test: `SYS_DLOPEN` of [`LIB`], read once @@ -70,21 +70,12 @@ fn main() { assert!(canary_intact(), "the canary changed while other kernel addresses were tried"); // 3. A user address off the value's alignment is still refused. - let region = unsafe { - syscall::mmap( - core::ptr::null_mut(), - 2 * PAGE_2M as usize, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) - }; - assert!(!region.is_null(), "mmap failed"); + // SAFETY: placed anywhere, so it replaces nothing. + let region = unsafe { syscall::vm_reserve(2 * PAGE as usize, VmProt::ReadWrite, VmAt::ANYWHERE) }.expect("reserve two pages"); let base = region as u64; - let boundary = (base + PAGE_2M) & !(PAGE_2M - 1); - for &addr in &[base + 4] { - let ret = dlopen_raw(LIB, addr); - assert_eq!(err(ret), Some(SyscallError::BadAddress), "dlopen took {addr:#x}"); - } + let boundary = base + PAGE; + let ret = dlopen_raw(LIB, base + 4); + assert_eq!(err(ret), Some(SyscallError::BadAddress), "dlopen took {:#x}", base + 4); // 3b. A refused dlopen must register nothing: the library loads and maps // before the copy-out, so a `BadAddress` used to leave a module in the @@ -154,7 +145,7 @@ fn main() { let ret = unsafe { syscall::futex_wake(word, 1) }; assert!(err(ret).is_none(), "futex_wake refused a word of this process's own: {ret:#x}"); - unsafe { syscall::munmap(region, 2 * PAGE_2M as usize) }.expect("munmap"); + unsafe { syscall::vm_release(region, 2 * PAGE as usize) }.expect("release"); assert!(canary_intact(), "the canary changed during the run"); println!("dlopen and futex refuse an address they may not reach, and kernel memory is untouched"); } diff --git a/tests/guest/rust-tests/src/bin/abuse_page_straddle.rs b/tests/guest/rust-tests/src/bin/abuse_page_straddle.rs index 45c7fa2e5bb..8fe78a0e175 100644 --- a/tests/guest/rust-tests/src/bin/abuse_page_straddle.rs +++ b/tests/guest/rust-tests/src/bin/abuse_page_straddle.rs @@ -17,7 +17,7 @@ mod image; use toyos_abi::RawHandle; use toyos_abi::syscall::{ - self, MmapFlags, MmapProt, OpenFlags, ProcessStats, SpawnArgs, Stat, SyscallError, SYS_FSTAT, + self, VmAt, VmProt, OpenFlags, ProcessStats, SpawnArgs, Stat, SyscallError, SYS_FSTAT, SYS_PROCESS_STATS, }; @@ -58,12 +58,7 @@ fn err(ret: u64) -> Option { fn main() { let region = unsafe { - syscall::mmap( - core::ptr::null_mut(), - 2 * PAGE as usize, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(2 * PAGE as usize, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!region.is_null(), "mmap failed"); let base = region as u64; @@ -128,6 +123,6 @@ fn main() { let stats = unsafe { (straddling as *const ProcessStats).read_unaligned() }; assert!(stats.wall_ns > 0 && stats.pid != 0, "process_stats wrote a snapshot with no wall time or pid in it"); - unsafe { syscall::munmap(region, 2 * PAGE as usize) }.expect("munmap"); + unsafe { syscall::vm_release(region, 2 * PAGE as usize) }.expect("munmap"); println!("a typed syscall argument across a page boundary is read and written whole"); } diff --git a/tests/guest/rust-tests/src/bin/abuse_readonly_copyout.rs b/tests/guest/rust-tests/src/bin/abuse_readonly_copyout.rs index f43f7900fc1..e266fd9022b 100644 --- a/tests/guest/rust-tests/src/bin/abuse_readonly_copyout.rs +++ b/tests/guest/rust-tests/src/bin/abuse_readonly_copyout.rs @@ -22,7 +22,7 @@ use std::os::toyos::process::ChildExt; use std::process::Command; use toyos_abi::clock::{ClockPage, CLOCK_MAGIC, CLOCK_PAGE}; -use toyos_abi::syscall::{self, MmapFlags, MmapProt, OpenFlags, ProcessStats, SeekFrom, SyscallError}; +use toyos_abi::syscall::{self, OpenFlags, VmAt, VmProt, ProcessStats, SeekFrom, SyscallError}; use toyos_abi::RawHandle; const SELF_PATH: &str = "/system/bin/test_rs_abuse_readonly_copyout"; @@ -92,17 +92,10 @@ fn refused(what: &str, addr: u64, stats: RawHandle) -> Result<(), String> { } fn refused_mmap(stats: RawHandle) -> Result<(), String> { - let ro = unsafe { - syscall::mmap( - core::ptr::null_mut(), - PAGE_4K as usize, - MmapProt::READ, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) - }; - assert!(!ro.is_null(), "mmap a read-only page"); + // SAFETY: placed anywhere, so it replaces nothing. + let ro = unsafe { syscall::vm_reserve(PAGE_4K as usize, VmProt::Read, VmAt::ANYWHERE) }.expect("reserve a read-only page"); let verdict = refused("a read-only mmap", ro as u64, stats); - unsafe { syscall::munmap(ro, PAGE_4K as usize) }.expect("munmap"); + unsafe { syscall::vm_release(ro, PAGE_4K as usize) }.expect("release"); verdict } @@ -111,13 +104,11 @@ fn refused_mmap(stats: RawHandle) -> Result<(), String> { fn refused_across_pages() -> Result<(), String> { let flags = OpenFlags::READ | OpenFlags::WRITE | OpenFlags::CREATE | OpenFlags::TRUNCATE; let probe = syscall::open(PROBE_PATH, flags).expect("create the probe file"); - let rw = MmapProt::READ | MmapProt::WRITE; - let anon = MmapFlags::ANONYMOUS | MmapFlags::PRIVATE; - let pages = unsafe { syscall::mmap(core::ptr::null_mut(), 2 * PAGE_4K as usize, rw, anon) }; - assert!(!pages.is_null(), "mmap two pages"); + // SAFETY: placed anywhere, so it replaces nothing. + let pages = unsafe { syscall::vm_reserve(2 * PAGE_4K as usize, VmProt::ReadWrite, VmAt::ANYWHERE) }.expect("reserve two pages"); let page = pages as u64 + PAGE_4K; - let ro = unsafe { syscall::mmap(page as *mut u8, PAGE_4K as usize, MmapProt::READ, anon | MmapFlags::FIXED) }; - assert_eq!(ro as u64, page, "map the second page read-only in place"); + // SAFETY: nothing holds a reference into the page. + unsafe { syscall::vm_protect(page as *mut u8, PAGE_4K as usize, VmProt::Read) }.expect("make the second page read-only"); let at = page - (STRADDLE / 2) as u64; let before = snapshot(at); // The writable half is offered its own bytes and the read-only half their @@ -134,7 +125,7 @@ fn refused_across_pages() -> Result<(), String> { } else { Ok(()) }; - unsafe { syscall::munmap(pages, 2 * PAGE_4K as usize) }.expect("munmap"); + unsafe { syscall::vm_release(pages, 2 * PAGE_4K as usize) }.expect("release"); syscall::close(probe); syscall::delete(PROBE_PATH).expect("delete the probe file"); verdict diff --git a/tests/guest/rust-tests/src/bin/abuse_spawn_argv.rs b/tests/guest/rust-tests/src/bin/abuse_spawn_argv.rs index bfcef6cd8df..14409be1606 100644 --- a/tests/guest/rust-tests/src/bin/abuse_spawn_argv.rs +++ b/tests/guest/rust-tests/src/bin/abuse_spawn_argv.rs @@ -7,7 +7,7 @@ use std::process::Command; -use toyos_abi::syscall::{self, MmapFlags, MmapProt, SpawnArgs, SyscallError}; +use toyos_abi::syscall::{self, VmAt, VmProt, SpawnArgs, SyscallError}; /// Where the child starts: `SpawnArgs` names a working directory or the spawn is refused. const CWD: &str = "/"; @@ -20,12 +20,7 @@ const BLOB: usize = 2 * 1024 * 1024; fn main() { let region = unsafe { - syscall::mmap( - core::ptr::null_mut(), - REGION, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(REGION, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!region.is_null(), "mmap failed"); for i in 0..BLOB / 2 { @@ -98,6 +93,6 @@ fn main() { .expect("spawn /system/bin/echo"); assert!(status.success(), "/system/bin/echo exited {status:?}"); - unsafe { syscall::munmap(region, REGION) }.expect("munmap"); + unsafe { syscall::vm_release(region, REGION) }.expect("munmap"); println!("oversized spawn path/argv/env rejected, spawn still usable"); } diff --git a/tests/guest/rust-tests/src/bin/abuse_thread_table.rs b/tests/guest/rust-tests/src/bin/abuse_thread_table.rs index 912ce62d2c1..d61c81d4ec2 100644 --- a/tests/guest/rust-tests/src/bin/abuse_thread_table.rs +++ b/tests/guest/rust-tests/src/bin/abuse_thread_table.rs @@ -6,7 +6,7 @@ use std::sync::atomic::{AtomicU32, Ordering}; use std::time::{Duration, Instant}; -use toyos_abi::syscall::{self, MmapFlags, MmapProt, SyscallError, MAX_THREADS}; +use toyos_abi::syscall::{self, VmAt, VmProt, SyscallError, MAX_THREADS}; const PAGE_2M: usize = 2 * 1024 * 1024; @@ -42,12 +42,7 @@ impl Stacks { fn next(&mut self) -> (u64, u64) { if self.chunk == 0 || self.used == PAGE_2M { let chunk = unsafe { - syscall::mmap( - core::ptr::null_mut(), - PAGE_2M, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(PAGE_2M, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!chunk.is_null(), "no memory for thread stacks"); self.chunk = chunk as usize; diff --git a/tests/guest/rust-tests/src/bin/abuse_tls_alloc.rs b/tests/guest/rust-tests/src/bin/abuse_tls_alloc.rs index e0b48467149..5db11a2c80d 100644 --- a/tests/guest/rust-tests/src/bin/abuse_tls_alloc.rs +++ b/tests/guest/rust-tests/src/bin/abuse_tls_alloc.rs @@ -8,7 +8,7 @@ use std::sync::atomic::{AtomicU64, Ordering}; -use toyos_abi::syscall::{self, MmapFlags, MmapProt, SyscallError}; +use toyos_abi::syscall::{self, VmAt, VmProt, SyscallError}; const PAGE_2M: usize = 2 * 1024 * 1024; @@ -41,12 +41,7 @@ fn alloc_block_preserving_dtv(module_id: u64) -> Result { fn map_2m() -> *mut u8 { let p = unsafe { - syscall::mmap( - core::ptr::null_mut(), - PAGE_2M, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(PAGE_2M, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!p.is_null(), "mmap failed"); p @@ -127,7 +122,7 @@ fn main() { } assert_eq!(unsafe { block.read_volatile() }, MINE, "TLS block at {first:#x} was reissued to another mapping"); for p in claimed { - unsafe { syscall::munmap(p, PAGE_2M) }.expect("munmap"); + unsafe { syscall::vm_release(p, PAGE_2M) }.expect("munmap"); } // 3. A thread's mappings come back when it exits. Each run gets a fresh diff --git a/tests/guest/rust-tests/src/bin/futex_wake_counts.rs b/tests/guest/rust-tests/src/bin/futex_wake_counts.rs index d2376b76949..68e5fd35f12 100644 --- a/tests/guest/rust-tests/src/bin/futex_wake_counts.rs +++ b/tests/guest/rust-tests/src/bin/futex_wake_counts.rs @@ -48,7 +48,7 @@ use std::time::Duration; use toyos::endow::{Endowments, SYSCAP_LABEL}; use toyos::syscap::SysCap; -use toyos_abi::syscall::{self, MmapFlags, MmapProt}; +use toyos_abi::syscall::{self, VmAt, VmProt}; #[path = "../roster.rs"] mod roster; @@ -406,12 +406,7 @@ fn orphaned_by_unmap(cap: &SysCap) { let regions: Vec = (0..STALE_FRAMES) .map(|i| { let ptr = unsafe { - syscall::mmap( - std::ptr::null_mut(), - PAGE_2M, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(PAGE_2M, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!ptr.is_null(), "mmap #{i} for a futex word failed"); // The word is the first in the region, so its offset in the frame @@ -437,7 +432,7 @@ fn orphaned_by_unmap(cap: &SysCap) { // The unmap. Every frame behind these regions goes back to the PMM here, // and `mmap` hands the lowest free one to the next asker. for &addr in ®ions { - unsafe { syscall::munmap(addr as *mut u8, PAGE_2M) }.expect("munmap a futex region"); + unsafe { syscall::vm_release(addr as *mut u8, PAGE_2M) }.expect("munmap a futex region"); } go.write_all(b"go\n").expect("tell the sweeper to go"); @@ -486,12 +481,7 @@ fn sweep() { let mut regions = Vec::new(); for i in 0..SWEEP_FRAMES { let ptr = unsafe { - syscall::mmap( - std::ptr::null_mut(), - PAGE_2M, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(PAGE_2M, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!ptr.is_null(), "sweeper mmap #{i} failed"); let word = ptr as *const u32; @@ -502,7 +492,7 @@ fn sweep() { regions.push(ptr); } for ptr in regions { - unsafe { syscall::munmap(ptr, PAGE_2M) }.expect("sweeper munmap"); + unsafe { syscall::vm_release(ptr, PAGE_2M) }.expect("sweeper munmap"); } assert!( stolen.is_empty(), diff --git a/tests/guest/rust-tests/src/bin/mmap_prot.rs b/tests/guest/rust-tests/src/bin/mmap_prot.rs index 07ac47fa99f..908f4ac8df4 100644 --- a/tests/guest/rust-tests/src/bin/mmap_prot.rs +++ b/tests/guest/rust-tests/src/bin/mmap_prot.rs @@ -1,6 +1,6 @@ //! Memory protection, which the kernel used to discard in both directions. //! -//! `sys_mmap`'s third argument was named `_prot`. Every mapping came back +//! `sys_mmap`'s third argument was once named `_prot`. Every mapping came back //! readable and writable whatever the caller asked for, so //! `sdk/libc`'s translation of POSIX `PROT_NONE` produced a writable //! guard page and the stack-overflow detection a C program builds on it @@ -25,7 +25,7 @@ use std::process::{Command, Stdio}; use std::sync::mpsc; use std::thread; -use toyos_abi::syscall::{mmap, munmap, MmapFlags, MmapProt, SyscallError}; +use toyos_abi::syscall::{self, SyscallError, VmAt, VmProt, PAGE_SIZE, VM_AT_ANYWHERE, VM_AT_FIXED}; #[path = "../arch/mmap.rs"] mod mmap; @@ -41,9 +41,9 @@ const RET: u8 = 0xC3; fn main() { match std::env::args().nth(1).as_deref() { - Some("write-none") => return touch(MmapProt::NONE, Access::Write), - Some("read-none") => return touch(MmapProt::NONE, Access::Read), - Some("write-ro") => return touch(MmapProt::READ, Access::Write), + Some("write-none") => return touch(VmProt::None, Access::Write), + Some("read-none") => return touch(VmProt::None, Access::Read), + Some("write-ro") => return touch(VmProt::Read, Access::Write), Some("exec-heap") => return exec_heap(), Some("exec-stack") => return exec_stack(), Some("write-text") => return write_text(), @@ -68,22 +68,15 @@ fn main() { println!("all mmap protection tests passed"); } -fn map(prot: MmapProt) -> *mut u8 { - let p = unsafe { - mmap(core::ptr::null_mut(), SIZE, prot, MmapFlags::ANONYMOUS | MmapFlags::PRIVATE) - }; - assert!( - !p.is_null() && (p as u64) < u64::MAX - 255, - "mmap(prot={:#x}) refused: {p:p}", - prot.0 - ); - p +fn map(prot: VmProt) -> *mut u8 { + // SAFETY: placed anywhere, so it replaces nothing. + unsafe { syscall::vm_reserve(SIZE, prot, VmAt::ANYWHERE) }.unwrap_or_else(|e| panic!("vm_reserve({prot:?}) refused: {e:?}")) } /// The positive control for every refusal below: the ordinary mapping every /// allocator in the system asks for still works. fn readwrite_is_readable_and_writable() { - let p = map(MmapProt::READ | MmapProt::WRITE); + let p = map(VmProt::ReadWrite); unsafe { p.add(OFFSET).write_volatile(0x5A); assert_eq!(p.add(OFFSET).read_volatile(), 0x5A, "a RW mapping lost a byte"); @@ -94,7 +87,7 @@ fn readwrite_is_readable_and_writable() { /// Read-only means readable, not merely unwritable. A kernel that refused the /// mapping outright would satisfy the write test and fail this one. fn readonly_is_readable() { - let p = map(MmapProt::READ); + let p = map(VmProt::Read); assert_eq!(unsafe { p.add(OFFSET).read_volatile() }, 0, "a fresh mapping is not zeroed"); println!(" PASS: PROT_READ is readable"); } @@ -102,8 +95,8 @@ fn readonly_is_readable() { /// `PROT_NONE` is a request for address space that faults, not a bad argument. /// The range has to be reserved — two of them in a row must not overlap. fn none_is_a_mapping_and_not_an_error() { - let a = map(MmapProt::NONE); - let b = map(MmapProt::NONE); + let a = map(VmProt::None); + let b = map(VmProt::None); let (lo, hi) = if a < b { (a, b) } else { (b, a) }; assert!( (hi as usize) - (lo as usize) >= SIZE, @@ -112,36 +105,34 @@ fn none_is_a_mapping_and_not_an_error() { println!(" PASS: PROT_NONE reserves address space and returns it"); } -/// A caller that sets a bit `MmapProt` or `MmapFlags` does not define is asking -/// for something it will not get — an executable mapping, say — so the request -/// is refused rather than served as the bits below it. The differential is the -/// bit and nothing else: the same call without it is the mapping this returns. +/// A protection word `VmProt` does not define, or a placement kind `VmAt` +/// does not, asks for something the caller will not get — an executable +/// mapping, say — so the request is refused rather than served as something +/// near it. The differential is the word and nothing else: the same call with +/// a defined one is the range this returns. fn an_undefined_bit_in_either_word_is_refused() { - const PROT: MmapProt = MmapProt(MmapProt::READ.0 | MmapProt::WRITE.0); - const FLAGS: MmapFlags = MmapFlags(MmapFlags::ANONYMOUS.0 | MmapFlags::PRIVATE.0); - // The lowest bit each word does not define. - const UNDEFINED_PROT: u64 = 1 << MmapProt::KNOWN.0.trailing_ones(); - const UNDEFINED_FLAG: u64 = 1 << MmapFlags::KNOWN.0.trailing_ones(); - - for (prot, flags, what) in [ - (MmapProt(PROT.0 | UNDEFINED_PROT), FLAGS, "a prot"), - (PROT, MmapFlags(FLAGS.0 | UNDEFINED_FLAG), "a flags"), - ] { - // SAFETY: not `FIXED`, so nothing of this process's is replaced. - let refused = unsafe { mmap::raw(0, SIZE as u64, prot, flags) }; + const UNDEFINED_PROT: u64 = 4; + const UNDEFINED_KIND: u64 = 2; + const _: () = assert!(UNDEFINED_KIND != VM_AT_ANYWHERE && UNDEFINED_KIND != VM_AT_FIXED); + assert_eq!(VmProt::from_u64(UNDEFINED_PROT), None); + + let rw = VmProt::ReadWrite as u64; + for (prot, kind, what) in [(UNDEFINED_PROT, VM_AT_ANYWHERE, "a protection"), (rw, UNDEFINED_KIND, "a placement")] { + // SAFETY: not fixed, so nothing of this process's is replaced. + let refused = unsafe { mmap::raw(SIZE as u64, prot, PAGE_SIZE as u64, kind) }; assert_eq!( refused, Err(SyscallError::InvalidArgument), - "{what} bit this ABI does not define was served: {refused:#x?}", + "{what} word this ABI does not define was served: {refused:#x?}", ); } - // The control: the same request without either bit is the mapping the + // The control: the same request with defined words is the range the // refusals above must not have been about. - // SAFETY: not `FIXED`. - let served = unsafe { mmap::raw(0, SIZE as u64, PROT, FLAGS) }.expect("the defined request was refused"); - unsafe { munmap(served as *mut u8, SIZE) }.expect("unmap the served request"); - println!(" PASS: an undefined mmap prot or flags bit is InvalidArgument, and without it the same call maps"); + // SAFETY: not fixed. + let served = unsafe { mmap::raw(SIZE as u64, rw, PAGE_SIZE as u64, VM_AT_ANYWHERE) }.expect("the defined request was refused"); + unsafe { syscall::vm_release(served as *mut u8, SIZE) }.expect("release the served request"); + println!(" PASS: an undefined protection or placement word is InvalidArgument, and a defined one maps"); } /// Lengths past every placement window. @@ -162,11 +153,11 @@ fn a_length_no_window_holds_is_refused_and_the_space_still_answers() { let mut rounds = 0u64; loop { let stopping = stop_rx.try_recv().is_ok(); - let p = map(MmapProt::READ | MmapProt::WRITE); + let p = map(VmProt::ReadWrite); unsafe { p.add(OFFSET).write_volatile(0x3C); assert_eq!(p.add(OFFSET).read_volatile(), 0x3C, "the sibling's mapping lost a byte"); - munmap(p, SIZE).expect("the sibling unmaps what it mapped"); + syscall::vm_release(p, SIZE).expect("the sibling releases what it reserved"); } rounds += 1; if rounds == 1 { @@ -181,10 +172,10 @@ fn a_length_no_window_holds_is_refused_and_the_space_still_answers() { started_rx.recv().expect("the sibling never finished its first round"); let mut answers = Vec::with_capacity(2 * UNPLACEABLE.len()); - for prot in [MmapProt::NONE, MmapProt::READ | MmapProt::WRITE] { + for prot in [VmProt::None, VmProt::ReadWrite] { for size in UNPLACEABLE { - // SAFETY: not `FIXED`. - answers.push((prot, size, unsafe { mmap::raw(0, size, prot, MmapFlags::ANONYMOUS | MmapFlags::PRIVATE) })); + // SAFETY: not fixed. + answers.push((prot, size, unsafe { mmap::raw(size, prot as u64, PAGE_SIZE as u64, VM_AT_ANYWHERE) })); } } @@ -197,8 +188,7 @@ fn a_length_no_window_holds_is_refused_and_the_space_still_answers() { assert_eq!( ret, Err(SyscallError::InvalidArgument), - "mmap(len={size:#x}, prot={:#x}) answered {ret:#x?}", - prot.0, + "vm_reserve(len={size:#x}, {prot:?}) answered {ret:#x?}", ); } println!( @@ -269,7 +259,7 @@ enum Access { Write, } -fn touch(prot: MmapProt, access: Access) { +fn touch(prot: VmProt, access: Access) { let p = map(prot); println!("armed at {p:p}"); match access { @@ -299,7 +289,7 @@ unsafe fn call(code: *const u8) { /// The heap: what a `PROT_READ | PROT_WRITE` mapping is, and where every /// allocation a program makes lives. fn exec_heap() { - let p = map(MmapProt::READ | MmapProt::WRITE); + let p = map(VmProt::ReadWrite); unsafe { p.add(OFFSET).write_volatile(RET) }; println!("armed at {:p}", unsafe { p.add(OFFSET) }); unsafe { call(p.add(OFFSET)) }; diff --git a/tests/guest/rust-tests/src/bin/mmap_stress.rs b/tests/guest/rust-tests/src/bin/mmap_stress.rs index 59ac6db5793..646b1d19ff5 100644 --- a/tests/guest/rust-tests/src/bin/mmap_stress.rs +++ b/tests/guest/rust-tests/src/bin/mmap_stress.rs @@ -1,19 +1,30 @@ -//! `mmap` and `munmap` as POSIX states them, over 4 KiB pages: no two -//! mappings overlap, `munmap` takes away any run of pages — part of a mapping, -//! across two, or a hole — and `MAP_FIXED` replaces whatever the pages it names -//! held while leaving the rest of each mapping as it was; and the ranges one +//! A process's own memory over 4 KiB pages, through the `vm_*` calls: no two +//! ranges overlap; `vm_release` takes away any run of pages — part of a +//! range, across two, or a hole — and a fixed reserve replaces whatever the +//! pages it names held; `vm_protect` commits a reservation and changes +//! protection in place, contents kept; `vm_discard` keeps the pages and reads +//! back zeros; `vm_decommit` makes them a reservation again. The ranges one //! space holds are bounded. +//! +//! A page left unreachable must fault, which ends the process, so each such +//! case runs in a child that makes the page itself and then loads from it. use std::collections::HashSet; +use std::process::Command; -use toyos_abi::syscall::{mmap, munmap, MmapFlags, MmapProt, SyscallError, MAX_RANGES}; +use toyos_abi::syscall::{self, SyscallError, VmAt, VmProt, MAX_RANGES}; const PAGE: usize = 4096; -const RW: MmapProt = MmapProt(MmapProt::READ.0 | MmapProt::WRITE.0); -const ANON: MmapFlags = MmapFlags(MmapFlags::ANONYMOUS.0 | MmapFlags::PRIVATE.0); -fn map(at: *mut u8, len: usize, flags: MmapFlags) -> *mut u8 { - unsafe { mmap(at, len, RW, flags) } +fn reserve(len: usize, prot: VmProt) -> *mut u8 { + // SAFETY: placed anywhere, so it replaces nothing. + unsafe { syscall::vm_reserve(len, prot, VmAt::ANYWHERE) }.unwrap_or_else(|e| panic!("reserve {len} bytes {prot:?}: {e:?}")) +} + +fn fixed(at: *mut u8, len: usize) -> Result<*mut u8, SyscallError> { + // SAFETY: every fixed range here lies over pages this test owns and + // reaches through raw pointers only. + unsafe { syscall::vm_reserve(len, VmProt::ReadWrite, VmAt::Fixed(at)) } } fn byte(p: *mut u8, page: usize) -> u8 { @@ -24,82 +35,181 @@ fn set(p: *mut u8, page: usize, value: u8) { unsafe { p.add(page * PAGE).write_volatile(value) } } +/// The unreachable pages a child makes and loads from. +const UNREACHABLE: [&str; 3] = ["reserved", "protected-none", "decommitted"]; + fn main() { - // No mapping may overlap another. - let mut regions: Vec<(*mut u8, usize)> = Vec::new(); + if let Some(case) = std::env::args().nth(1) { + return load_from(&case); + } + placement_and_release(); + protection(); + discard_and_decommit(); + ranges_are_bounded(); + for case in UNREACHABLE { + faults(case); + } + println!("vm_reserve, vm_protect, vm_discard, vm_decommit and vm_release do what each says"); +} + +fn placement_and_release() { + // No range may overlap another. + let mut ranges: Vec<(*mut u8, usize)> = Vec::new(); let mut seen: HashSet = HashSet::new(); for i in 0..64 { let pages = 1 + i % 5; - let ptr = map(core::ptr::null_mut(), pages * PAGE, ANON); - assert!(!ptr.is_null(), "mmap #{i} of {pages} pages failed"); - assert!(ptr as usize % PAGE == 0, "mmap #{i} returned {ptr:p}, off a page"); + let ptr = reserve(pages * PAGE, VmProt::ReadWrite); + assert!(ptr as usize % PAGE == 0, "reserve #{i} returned {ptr:p}, off a page"); for page in 0..pages { - assert!(seen.insert(ptr as usize + page * PAGE), "mmap #{i}: page {page} overlaps an earlier mapping"); + assert!(seen.insert(ptr as usize + page * PAGE), "reserve #{i}: page {page} overlaps an earlier range"); } set(ptr, 0, i as u8); set(ptr, pages - 1, i as u8); - regions.push((ptr, pages)); + ranges.push((ptr, pages)); } - for (i, &(ptr, pages)) in regions.iter().enumerate() { - assert_eq!((byte(ptr, 0), byte(ptr, pages - 1)), (i as u8, i as u8), "mapping #{i} lost a byte"); + for (i, &(ptr, pages)) in ranges.iter().enumerate() { + assert_eq!((byte(ptr, 0), byte(ptr, pages - 1)), (i as u8, i as u8), "range #{i} lost a byte"); } - for (ptr, pages) in regions { - unsafe { munmap(ptr, pages * PAGE) }.expect("munmap a whole mapping"); + for (ptr, pages) in ranges { + unsafe { syscall::vm_release(ptr, pages * PAGE) }.expect("release a whole range"); } - // Part of a mapping goes, and the rest stays as it was. - let four = map(core::ptr::null_mut(), 4 * PAGE, ANON); - assert!(!four.is_null(), "mmap four pages"); + // A reservation costs no memory, so a tebibyte of one is placed. + let tib = reserve(1 << 40, VmProt::None); + unsafe { syscall::vm_release(tib, 1 << 40) }.expect("release the tebibyte"); + + // Part of a range goes, and the rest stays as it was. + let four = reserve(4 * PAGE, VmProt::ReadWrite); (0..4).for_each(|page| set(four, page, 0x10 + page as u8)); - unsafe { munmap(four.add(PAGE), 2 * PAGE) }.expect("munmap the middle two pages"); - assert_eq!((byte(four, 0), byte(four, 3)), (0x10, 0x13), "unmapping the middle took a neighbour"); - // The hole is free: a FIXED mapping there is fresh zeroes. - let refilled = map(unsafe { four.add(PAGE) }, 2 * PAGE, ANON | MmapFlags::FIXED); - assert_eq!(refilled, unsafe { four.add(PAGE) }, "a FIXED mapping over the hole"); + unsafe { syscall::vm_release(four.add(PAGE), 2 * PAGE) }.expect("release the middle two pages"); + assert_eq!((byte(four, 0), byte(four, 3)), (0x10, 0x13), "releasing the middle took a neighbour"); + // The hole is free: a fixed range there is fresh zeroes. + assert_eq!(fixed(unsafe { four.add(PAGE) }, 2 * PAGE), Ok(unsafe { four.add(PAGE) }), "a fixed range over the hole"); assert_eq!((byte(four, 1), byte(four, 2)), (0, 0), "the hole held its old bytes"); - // FIXED over part of a mapping replaces those pages alone. + // Fixed over part of a range replaces those pages alone. set(four, 1, 0x21); - let over = map(unsafe { four.add(PAGE) }, PAGE, ANON | MmapFlags::FIXED); - assert_eq!(over, unsafe { four.add(PAGE) }, "FIXED over one page of a mapping"); - assert_eq!((byte(four, 0), byte(four, 1), byte(four, 3)), (0x10, 0, 0x13), "FIXED replaced more than its page"); - - // A range across mappings and holes goes whole, and a hole alone is no refusal. - unsafe { munmap(four, 4 * PAGE) }.expect("munmap across what is left"); - unsafe { munmap(four, 4 * PAGE) }.expect("munmap of a range that holds nothing"); - - // What munmap and FIXED refuse: an address off a page, and no bytes. - let one = map(core::ptr::null_mut(), PAGE, ANON); - assert!(!one.is_null(), "mmap one page"); - assert_eq!(unsafe { munmap(one.add(1), PAGE) }, Err(SyscallError::InvalidArgument), "munmap off a page"); - assert_eq!(unsafe { munmap(one, 0) }, Err(SyscallError::InvalidArgument), "munmap of no bytes"); - assert!(map(unsafe { one.add(1) }, PAGE, ANON | MmapFlags::FIXED).is_null(), "FIXED off a page"); + assert_eq!(fixed(unsafe { four.add(PAGE) }, PAGE), Ok(unsafe { four.add(PAGE) }), "fixed over one page of a range"); + assert_eq!((byte(four, 0), byte(four, 1), byte(four, 3)), (0x10, 0, 0x13), "fixed replaced more than its page"); + + // A run across ranges and holes goes whole, and a hole alone is no refusal. + unsafe { syscall::vm_release(four, 4 * PAGE) }.expect("release across what is left"); + unsafe { syscall::vm_release(four, 4 * PAGE) }.expect("release of a run that holds nothing"); + + // What release and fixed refuse: an address off a page, and no bytes. + let one = reserve(PAGE, VmProt::ReadWrite); + assert_eq!(unsafe { syscall::vm_release(one.add(1), PAGE) }, Err(SyscallError::InvalidArgument), "release off a page"); + assert_eq!(unsafe { syscall::vm_release(one, 0) }, Err(SyscallError::InvalidArgument), "release of no bytes"); + assert_eq!(fixed(unsafe { one.add(1) }, PAGE), Err(SyscallError::InvalidArgument), "fixed off a page"); set(one, 0, 0x5A); - assert_eq!(byte(one, 0), 0x5A, "a refused call touched the mapping"); + assert_eq!(byte(one, 0), 0x5A, "a refused call touched the range"); + unsafe { syscall::vm_release(one, PAGE) }.expect("release one page"); + println!(" PASS: ranges are placed apart, and any run of pages is released or replaced"); +} - // A range given back is placed again. - unsafe { munmap(one, PAGE) }.expect("munmap one page"); - let again = map(core::ptr::null_mut(), PAGE, ANON); - assert!(!again.is_null(), "mmap after an munmap"); +fn protection() { + // A reservation is committed by protecting it, a page of it if asked. + let three = reserve(3 * PAGE, VmProt::None); + unsafe { syscall::vm_protect(three.add(PAGE), PAGE, VmProt::ReadWrite) }.expect("commit the middle page"); + set(three, 1, 0x31); + assert_eq!(byte(three, 1), 0x31, "a committed page lost its byte"); - // A space holds a bounded number of ranges: pages alternately reserved - // and readable never join, so each is one range more until the next is - // refused, and every one is still given back at the bound. + // Read-only keeps the bytes, and so does None and back. + unsafe { syscall::vm_protect(three.add(PAGE), PAGE, VmProt::Read) }.expect("make the page read-only"); + assert_eq!(byte(three, 1), 0x31, "read-only lost the page's byte"); + unsafe { syscall::vm_protect(three.add(PAGE), PAGE, VmProt::None) }.expect("make the page unreachable"); + unsafe { syscall::vm_protect(three.add(PAGE), PAGE, VmProt::ReadWrite) }.expect("make the page writable again"); + assert_eq!(byte(three, 1), 0x31, "None and back lost the page's byte"); + + // What protect refuses, all or nothing: a page no range holds, and + // executable memory, which no call maps yet. + unsafe { syscall::vm_release(three.add(2 * PAGE), PAGE) }.expect("release the last page"); + assert_eq!(unsafe { syscall::vm_protect(three, 3 * PAGE, VmProt::Read) }, Err(SyscallError::BadAddress), "protect over a hole"); + assert_eq!(unsafe { syscall::vm_protect(three, PAGE, VmProt::ReadExec) }, Err(SyscallError::NotSupported), "protect executable"); + set(three, 1, 0x32); + assert_eq!(byte(three, 1), 0x32, "a refused protect changed the committed page"); + unsafe { syscall::vm_release(three, 2 * PAGE) }.expect("release the rest"); + println!(" PASS: protect commits reservations and changes protection with contents kept, all or nothing"); +} + +fn discard_and_decommit() { + let two = reserve(2 * PAGE, VmProt::ReadWrite); + set(two, 0, 0x41); + set(two, 1, 0x42); + unsafe { syscall::vm_discard(two, PAGE) }.expect("discard the first page"); + assert_eq!((byte(two, 0), byte(two, 1)), (0, 0x42), "discard took the wrong page or kept its bytes"); + set(two, 0, 0x43); + assert_eq!(byte(two, 0), 0x43, "a discarded page is not writable again"); + + unsafe { syscall::vm_decommit(two.add(PAGE), PAGE) }.expect("decommit the second page"); + unsafe { syscall::vm_protect(two.add(PAGE), PAGE, VmProt::ReadWrite) }.expect("commit it again"); + assert_eq!(byte(two, 1), 0, "a decommitted page came back with its old bytes"); + assert_eq!(byte(two, 0), 0x43, "decommitting a neighbour took a byte"); + + assert_eq!(unsafe { syscall::vm_discard(two, 3 * PAGE) }, Err(SyscallError::BadAddress), "discard past the range"); + assert_eq!(byte(two, 0), 0x43, "a refused discard took a byte"); + unsafe { syscall::vm_release(two, 2 * PAGE) }.expect("release"); + println!(" PASS: discard reads back zeros in place, and decommit makes pages a reservation again"); +} + +/// Pages alternately reserved and readable never join, so each is one range +/// more until the next is refused; every one is still given back at the +/// bound, where releasing a whole range cuts none. +fn ranges_are_bounded() { let mut held = Vec::new(); loop { - let prot = if held.len() % 2 == 0 { MmapProt::NONE } else { MmapProt::READ }; - let p = unsafe { mmap(core::ptr::null_mut(), PAGE, prot, ANON) }; - if p.is_null() { - break; + let prot = if held.len() % 2 == 0 { VmProt::None } else { VmProt::Read }; + // SAFETY: placed anywhere, so it replaces nothing. + match unsafe { syscall::vm_reserve(PAGE, prot, VmAt::ANYWHERE) } { + Ok(p) => held.push(p), + Err(e) => { + assert_eq!(e, SyscallError::ResourceExhausted, "a reserve at {} ranges", held.len()); + break; + } } - held.push(p); assert!(held.len() <= MAX_RANGES, "{} ranges placed, past the bound of {MAX_RANGES}", held.len()); } assert!(held.len() > MAX_RANGES - 64, "refused at {} ranges, well short of {MAX_RANGES}", held.len()); for p in held { - unsafe { munmap(p, PAGE) }.expect("munmap a whole range at the bound"); + unsafe { syscall::vm_release(p, PAGE) }.expect("release a whole range at the bound"); } - assert!(!map(core::ptr::null_mut(), PAGE, ANON).is_null(), "mmap after the ranges went back"); + let again = reserve(PAGE, VmProt::ReadWrite); + unsafe { syscall::vm_release(again, PAGE) }.expect("release"); + println!(" PASS: a space holds at most {MAX_RANGES} ranges, and gives each back at the bound"); +} - println!("mmap and munmap take and give back any run of pages, up to a bound of ranges"); +/// A child that makes the page `case` names and loads from it must die of it. +fn faults(case: &str) { + let out = Command::new("/system/bin/test_rs_mmap_stress") + .arg(case) + .output() + .unwrap_or_else(|e| panic!("spawn the {case} child: {e}")); + let said = String::from_utf8_lossy(&out.stdout); + assert!(said.contains("armed"), "the {case} child never reached the load:\n{said}"); + assert!(!said.contains("SURVIVED"), "a {case} page was readable:\n{said}"); + assert!(!out.status.success(), "a load from a {case} page did not end the child"); + println!(" PASS: a load from a {case} page ends the process"); +} + +/// Make a page, written once where the case has one, unreachable as `case` +/// says, then load from it. +fn load_from(case: &str) { + let page = match case { + "reserved" => reserve(PAGE, VmProt::None), + "protected-none" | "decommitted" => { + let page = reserve(PAGE, VmProt::ReadWrite); + set(page, 0, 0x77); + let done = if case == "decommitted" { + unsafe { syscall::vm_decommit(page, PAGE) } + } else { + unsafe { syscall::vm_protect(page, PAGE, VmProt::None) } + }; + done.unwrap_or_else(|e| panic!("{case}: {e:?}")); + page + } + other => panic!("no case {other}"), + }; + println!("armed at {page:p}"); + let v = byte(page, 0); + println!("SURVIVED, read {v:#x}"); } diff --git a/tests/guest/rust-tests/src/bin/std_mmap.rs b/tests/guest/rust-tests/src/bin/std_mmap.rs index b1a0fa68c43..280afb29f44 100644 --- a/tests/guest/rust-tests/src/bin/std_mmap.rs +++ b/tests/guest/rust-tests/src/bin/std_mmap.rs @@ -1,8 +1,8 @@ -use toyos_abi::syscall::{mmap, munmap, MmapProt, MmapFlags}; +use toyos_abi::syscall::{self, VmAt, VmProt}; fn main() { let size: usize = 2 * 1024 * 1024; - let ptr = unsafe { mmap(core::ptr::null_mut(), size, MmapProt::READ | MmapProt::WRITE, MmapFlags::ANONYMOUS | MmapFlags::PRIVATE) }; + let ptr = unsafe { syscall::vm_reserve(size, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!ptr.is_null(), "mmap failed"); for i in 0..size { @@ -14,7 +14,7 @@ fn main() { assert_eq!(val, 0xAB, "mismatch at offset {i}: expected 0xAB, got {val:#x}"); } - unsafe { munmap(ptr, size) }.expect("munmap failed"); + unsafe { syscall::vm_release(ptr, size) }.expect("munmap failed"); println!("all mmap tests passed"); } diff --git a/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs b/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs index 71b04e28cd2..42653bad344 100644 --- a/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs +++ b/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs @@ -20,7 +20,7 @@ //! the whole TLB within milliseconds — so what is measured is the property that //! closes the window: the free happens after the flush. -use toyos_abi::syscall::{self, MmapFlags, MmapProt, SYS_DEBUG}; +use toyos_abi::syscall::{self, VmAt, VmProt, SYS_DEBUG}; /// Long enough to read off a clock through two syscalls, short enough that a /// sweep of them is not a boot's worth of stalled CPU. The delay spins with @@ -92,12 +92,7 @@ fn timed(f: impl FnOnce()) -> u64 { fn map(size: usize) -> *mut u8 { let p = unsafe { - syscall::mmap( - core::ptr::null_mut(), - size, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE, - ) + syscall::vm_reserve(size, VmProt::ReadWrite, VmAt::ANYWHERE).unwrap_or(core::ptr::null_mut()) }; assert!(!p.is_null(), "mmap failed"); p @@ -121,7 +116,7 @@ fn main() { let region = map(PAGE_2M); armed(cpus); let elapsed = timed(|| { - unsafe { syscall::munmap(region, PAGE_2M) }.expect("munmap"); + unsafe { syscall::vm_release(region, PAGE_2M) }.expect("munmap"); }); if elapsed >= FLOOR_NANOS { judged = true; @@ -145,12 +140,7 @@ fn main() { armed(cpus); let elapsed = timed(|| { let p = unsafe { - syscall::mmap( - placed, - PAGE_2M, - MmapProt::READ | MmapProt::WRITE, - MmapFlags::ANONYMOUS | MmapFlags::PRIVATE | MmapFlags::FIXED, - ) + syscall::vm_reserve(PAGE_2M, VmProt::ReadWrite, VmAt::Fixed(placed)).unwrap_or(core::ptr::null_mut()) }; assert_eq!(p, placed, "MAP_FIXED did not honour the address"); }); @@ -159,7 +149,7 @@ fn main() { "a fixed mmap returned in {elapsed}ns with every other CPU answering {DELAY_NANOS}ns \ late — it replaced the mapping without waiting for the flush", ); - unsafe { syscall::munmap(placed, PAGE_2M) }.expect("munmap the fixed mapping"); + unsafe { syscall::vm_release(placed, PAGE_2M) }.expect("munmap the fixed mapping"); disarm(); From a5cbff6e3a7380201ad3acd8619536c2cbfd1b14 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 22 Oct 2026 14:36:54 +0200 Subject: [PATCH 02/10] mm: answer review round 1 of the lifecycle stage - `mmap_stress` asks `vm_protect` to make the clock page, the one object every process maps read-only, writable, and is answered `PermissionDenied`: the bound an object range keeps is now tested where it is applied. With the bound ignored the test reds. - libc's pthread stacks are a reservation of the size asked, in whole pages, above one guard page, committed before the thread starts and released when it is reaped, as std's are; they no longer come from the heap. - `private_prot` is `vm_prot`: it parses the word for object ranges too. `clear`'s contract was rewritten with the inventory stage's fix. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- kernel/src/syscall/vm.rs | 10 ++-- sdk/libc/src/pthread.rs | 48 ++++++++++++------- tests/guest/rust-tests/src/bin/mmap_stress.rs | 12 +++++ 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/kernel/src/syscall/vm.rs b/kernel/src/syscall/vm.rs index a2d0838fa02..510628d3c10 100644 --- a/kernel/src/syscall/vm.rs +++ b/kernel/src/syscall/vm.rs @@ -21,9 +21,9 @@ fn whole_pages(size: u64) -> Option { size.checked_next_multiple_of(crate::mm::PAGE_SIZE).filter(|&bytes| bytes != 0) } -/// A protection a program may give its own private memory. Executable memory -/// is not yet one: `ReadExec` is `NotSupported`. -fn private_prot(word: u64) -> Result { +/// The protection a `VmProt` word names. Executable memory is not yet one a +/// program may ask for: `ReadExec` is `NotSupported`. +fn vm_prot(word: u64) -> Result { match VmProt::from_u64(word).ok_or(SyscallError::InvalidArgument)? { VmProt::None => Ok(Prot::None), VmProt::Read => Ok(Prot::Read), @@ -49,7 +49,7 @@ fn answer(done: Result<(), SyscallError>) -> u64 { /// exactly at `place` in place of whatever those pages held. pub(super) fn sys_vm_reserve(len: u64, prot: u64, place: u64, kind: u64) -> u64 { let reserved = (|| { - let prot = private_prot(prot)?; + let prot = vm_prot(prot)?; let len = whole_pages(len).ok_or(SyscallError::InvalidArgument)?; let at = match kind { VM_AT_ANYWHERE => At::Anywhere(place), @@ -74,7 +74,7 @@ pub(super) fn sys_vm_release(addr: u64, len: u64) -> u64 { pub(super) fn sys_vm_protect(addr: u64, len: u64, prot: u64) -> u64 { answer((|| { - let prot = private_prot(prot)?; + let prot = vm_prot(prot)?; let (start, len) = range(addr, len)?; process::current_address_space().lock().protect(start, len, prot) })()) diff --git a/sdk/libc/src/pthread.rs b/sdk/libc/src/pthread.rs index 81f75f4a138..d82a638ce67 100644 --- a/sdk/libc/src/pthread.rs +++ b/sdk/libc/src/pthread.rs @@ -14,10 +14,8 @@ //! `pthread_create` joins it and frees its stack, which no thread can free //! while standing on it. -use alloc::alloc::{alloc as heap_alloc, dealloc as heap_dealloc}; use alloc::boxed::Box; use alloc::vec::Vec; -use core::alloc::Layout; use core::cell::Cell; use core::ptr; use core::sync::atomic::{AtomicPtr, AtomicU32, AtomicU64, Ordering}; @@ -42,8 +40,9 @@ struct Thread { /// Nonzero once `tid` is written; the thread waits for it before it runs. published: AtomicU32, state: AtomicU32, - stack: *mut u8, - stack_size: usize, + /// The stack's reservation: [`GUARD`] reserved, then the stack committed. + reservation: *mut u8, + reserved: usize, start: StartRoutine, arg: *mut u8, result: AtomicPtr, @@ -69,8 +68,26 @@ static REAPED: Lock> = Lock::new(Vec::new()); pub(crate) use toyos::sync::Lock; -fn stack_layout(size: usize) -> Layout { - Layout::from_size_align(size, STACK_ALIGN).expect("a thread stack's size is a multiple of its alignment") +/// Reserved pages below a thread's stack, so that running off it faults: +/// one page, which C's frames reach through only by skipping it, as a guard +/// page on every POSIX system can be skipped. +const GUARD: usize = syscall::PAGE_SIZE; + +/// A stack of `size` bytes in whole pages above a [`GUARD`], committed before +/// the thread starts: the reservation, its length, and the stack's base. +fn reserve_stack(size: usize) -> Option<(*mut u8, usize, *mut u8)> { + let pages = size.checked_next_multiple_of(syscall::PAGE_SIZE)?; + let reserved = pages.checked_add(GUARD)?; + // SAFETY: placed anywhere, so it replaces nothing. + let reservation = unsafe { syscall::vm_reserve(reserved, syscall::VmProt::None, syscall::VmAt::ANYWHERE) }.ok()?; + let base = reservation.wrapping_add(GUARD); + // SAFETY: the stack's pages, inside the reservation just made. + if unsafe { syscall::vm_protect(base, pages, syscall::VmProt::ReadWrite) }.is_err() { + // SAFETY: the reservation just made, which nothing reaches. + let _ = unsafe { syscall::vm_release(reservation, reserved) }; + return None; + } + Some((reservation, reserved, base)) } /// Wait for `thread` to be gone, free its stack and block, and answer what it @@ -80,7 +97,7 @@ unsafe fn reap(thread: *mut Thread) -> *mut u8 { syscall::thread_join(t.tid.load(Ordering::Relaxed)); let result = t.result.load(Ordering::Acquire); unsafe { - heap_dealloc(t.stack, stack_layout(t.stack_size)); + let _ = syscall::vm_release(t.reservation, t.reserved); drop(Box::from_raw(thread)); } result @@ -158,29 +175,28 @@ unsafe fn create(attr: *const u64, start_routine: StartRoutine, arg: *mut u8, co let attr = if attr.is_null() { attr_default() } else { unsafe { *attr } }; let stack_size = (attr & !ATTR_DETACHED) as usize; - let stack = unsafe { heap_alloc(stack_layout(stack_size)) }; - if stack.is_null() { + let Some((reservation, reserved, stack)) = reserve_stack(stack_size) else { return Err(EAGAIN); - } + }; let state = if attr & ATTR_DETACHED != 0 { DETACHED } else { JOINABLE }; let block = Box::into_raw(Box::new(Thread { tid: AtomicU64::new(0), published: AtomicU32::new(0), state: AtomicU32::new(state), - stack, - stack_size, + reservation, + reserved, start: start_routine, arg, result: AtomicPtr::new(ptr::null_mut()), counted, })); - let top = (stack as usize + stack_size) & !(STACK_ALIGN - 1); - // SAFETY: the entry is a function of this library, and the stack is a - // fresh allocation of `stack_size` bytes. + let top = (reservation as usize + reserved) & !(STACK_ALIGN - 1); + // SAFETY: the entry is a function of this library, and the stack is the + // committed top of a fresh reservation. let tid = unsafe { syscall::thread_spawn(thread_entry as *const () as u64, top as u64, block as u64, stack as u64) }; if syscall::SyscallError::from_u64(tid).is_some() { unsafe { - heap_dealloc(stack, stack_layout(stack_size)); + let _ = syscall::vm_release(reservation, reserved); drop(Box::from_raw(block)); } return Err(EAGAIN); diff --git a/tests/guest/rust-tests/src/bin/mmap_stress.rs b/tests/guest/rust-tests/src/bin/mmap_stress.rs index 646b1d19ff5..01105e0e040 100644 --- a/tests/guest/rust-tests/src/bin/mmap_stress.rs +++ b/tests/guest/rust-tests/src/bin/mmap_stress.rs @@ -129,6 +129,18 @@ fn protection() { set(three, 1, 0x32); assert_eq!(byte(three, 1), 0x32, "a refused protect changed the committed page"); unsafe { syscall::vm_release(three, 2 * PAGE) }.expect("release the rest"); + + // A shared object's pages take no protection past what they were mapped + // with: the clock page, one object mapped read-only into every process, + // is not made writable for all of them by one. + let clock = toyos_abi::clock::CLOCK_PAGE as *mut u8; + assert_eq!( + unsafe { syscall::vm_protect(clock, PAGE, VmProt::ReadWrite) }, + Err(SyscallError::PermissionDenied), + "the clock page made writable", + ); + // And the page still reads as the clock: `page` asserts its magic. + let _ = toyos_abi::clock::page(); println!(" PASS: protect commits reservations and changes protection with contents kept, all or nothing"); } From db6eeef115e301cd037bfd400b00929ac5322ea4 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 22 Oct 2026 14:55:45 +0200 Subject: [PATCH 03/10] x86: a change to a user space's leaves reaches only the CPUs running it Every change to a present user leaf interrupted every other CPU and waited for all of them, so `vm_protect`, a `vm_release` of touched pages and V8-style protection toggles cost an IPI round on the whole machine even for a single-threaded process. Now `kernel::tlb_tags` keeps, per PCID, the CPUs running the space and when it last changed, and per CPU when it last flushed every user translation: - A change stamps the space and reaches the CPUs running it; the initiator flushes its own entries by page for up to 64 pages, and a change no other CPU runs sends no IPI and waits for nothing. - A CPU that ran the space before keeps its entries under PCID, so instead of being reached it flushes every user translation before it next enters a space that changed since its own last flush. - The two sides pair through a `SeqCst` fence each: either the entering CPU sees the stamp, or the change sees the CPU. `kernel-loom`'s `tlb_tags` checks it, and reds with the entering side's fence removed (a CI control). Kernel-half changes and PCID reclaims still reach every CPU. AArch64 broadcasts its invalidations in hardware and is unchanged. `tlb_shootdown_waits` now times its waits with a thread of its own spinning on each other CPU, touches the page a fixed mapping replaces (an untouched page has no translation to flush), and adds the other half: with no other thread running, a munmap reaches no other CPU, counted by the test kernel's `SPACE_REACHED` rather than timed. Each half reds under its mutation: a change that reaches no runner, and one that reaches every CPU. A CPU counts as running a space from before its root is loaded until after it is replaced, and one past the 256 the targets hold reaches every CPU, as before. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- build/ci.rs | 3 + ...eaches-only-the-cpus-that-ran-the-space.md | 46 +++-- ...aths-are-unmeasured-against-2-mib-pages.md | 6 +- kernel/Cargo.toml | 2 + kernel/loom/Cargo.toml | 3 + kernel/loom/src/lib.rs | 3 + kernel/loom/tests/tlb_tags.rs | 63 ++++++ kernel/src/arch/aarch64/paging.rs | 5 +- kernel/src/arch/x86_64/apic.rs | 7 + kernel/src/arch/x86_64/paging.rs | 29 ++- kernel/src/arch/x86_64/tlb.rs | 92 ++++++++- kernel/src/main.rs | 1 + kernel/src/mm/space.rs | 22 ++- kernel/src/syscall/dispatch.rs | 1 + kernel/src/tlb_tags.rs | 183 ++++++++++++++++++ sdk/abi/src/syscall.rs | 4 + .../rust-tests/src/bin/tlb_shootdown_waits.rs | 104 ++++++++-- 17 files changed, 519 insertions(+), 55 deletions(-) create mode 100644 kernel/loom/tests/tlb_tags.rs create mode 100644 kernel/src/tlb_tags.rs diff --git a/build/ci.rs b/build/ci.rs index 4db5e76e7f9..657704c3037 100644 --- a/build/ci.rs +++ b/build/ci.rs @@ -511,6 +511,9 @@ pub(crate) const CONTROLS: &[Control] = &[ Fails("an_acknowledged_flush_postdates_the_page_table_write"), Fails("one_serve_answers_two_concurrent_shootdowns"), ]), + red(KERNEL_LOOM, "tlb-tags-enter-relaxed", Some("tlb_tags"), &[ + Fails("a_change_reaches_a_runner_or_the_entering_cpu_flushes"), + ]), red(KERNEL_LOOM, "shootdown-served-relaxed", Some("counters_round"), &[ Fails("a_cpu_read_as_answered_is_read_with_its_answer"), ]), diff --git a/issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md b/issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md index cbf2b4ae769..e75186e5b49 100644 --- a/issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md +++ b/issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md @@ -6,24 +6,30 @@ opened: 2026-09-29 # A TLB shootdown reaches only the CPUs that ran the space -A shootdown blocks on every other CPU (`kernel/src/arch/x86_64/tlb.rs`, -`shootdown`), whether or not it ever ran the address space. It waits on -`issues/no-test-can-hold-a-thread-on-a-named-cpu.md`. A CPU that -switches away keeps the space's entries: under PCID every CR3 load sets -NOFLUSH (`kernel/src/arch/x86_64/paging.rs:295-306`), so a CPU leaves the -target set only when those entries are gone, not when it stops running the -space. The TCG model has no PCID (`build/arch.rs:169`), so only a machine with -PCID can show a switched-out CPU still holding them. +On x86 a change to one user space's leaves now reaches only the CPUs running +that space (`kernel/src/arch/x86_64/tlb.rs`, `user_change`, over +`kernel/src/tlb_tags.rs`). A CPU that switched away keeps the space's entries +under PCID, since every CR3 load sets NOFLUSH, so it is not reached; it +flushes every user translation before it next enters a space that changed +since its own last such flush. The initiator flushes its own entries by page +for up to 64 pages, and a change no other CPU runs sends no IPI. AArch64 +broadcasts its invalidations in hardware and needs none of this. Every other +shootdown, a kernel-half change and a PCID reclaim among them, still reaches +every CPU, and a reached CPU still answers with a flush of every user +translation, not of the range. -**Exit**: on every proving machine and under TCG, a sibling pinned on another -CPU touches a page the initiator unmaps and then faults; on each proving -machine with PCID, the sibling touches the page and blocks off its pinned CPU, -the initiator unmaps it while the sibling is blocked and a thread of another -space runs there, and the sibling faults on touching it again after it wakes -there; cases in `kernel/loom/tests/tlb_shootdown.rs` switch a CPU into the -space, and out of it, while the initiator reads the target set; the T14's -munmap figure. **Mutation**, each red: a target set that omits a CPU the space -ran on; a CPU that joins the set after loading CR3; a CPU cleared from the set -when it switches away from the space, its red recorded over repeated runs, -since it rests on the TLB keeping the entry across the block. **Oracle**: loom, -and each proving machine's TLB. +What holds it: `kernel/loom/tests/tlb_tags.rs` checks the pairing a CPU +entering a space and a change reading who runs it rest on, and reds with the +pairing's fence removed; `tlb_shootdown_waits` shows on the test kernel that +a change waits for a CPU running a thread of the space and for none when no +other CPU runs one, each half red under its mutation. + +What is owed, since it waits on +`issues/no-test-can-hold-a-thread-on-a-named-cpu.md`: on a machine with PCID, +a sibling that touches a page, blocks off its pinned CPU while a thread of +another space runs there, and faults touching the page again after the +initiator unmapped it meanwhile. The TCG model has no PCID +(`build/arch.rs`), so only metal can show a switched-out CPU's entries. + +**Exit**: that case, on each proving machine with PCID, red with the flush on +entry removed; and the T14's munmap and `vm_protect` figures against Linux. diff --git a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md index 2473f0923cf..09d711e742c 100644 --- a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md +++ b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md @@ -23,8 +23,10 @@ did, and none is measured on the T14 against the base or against Linux: inventory lookup and a table walk per page. - **Spawn.** The whole image is copied when the process is made. -The x86 shootdown that reaches every CPU for any change to a present leaf has -its own record (`issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md`). +- **Shootdowns.** A change to a present leaf reaches the CPUs running the + space, each of which flushes every user translation, and a CPU entering a + space that changed flushes them all too + (`issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md`). osbench measures faults (`mem.first_touch`, `mem.anon`), small maps (`mem.map_unmap_4k`), partial unmaps and protection changes against Linux on diff --git a/kernel/Cargo.toml b/kernel/Cargo.toml index 5b6c60b7bb3..5b3796d9d5d 100644 --- a/kernel/Cargo.toml +++ b/kernel/Cargo.toml @@ -55,6 +55,8 @@ reap-raise-relaxed = [] shootdown-serve-relaxed = [] # `src/shootdown.rs`'s `served` reads `Relaxed`, and `counters_round` reds. shootdown-served-relaxed = [] +# `src/tlb_tags.rs`'s `enter` loses its fence, and `tlb_tags` reds. +tlb-tags-enter-relaxed = [] # `src/log/shard.rs`'s publishing store goes `Relaxed`, and `log_record` reds. log-commit-release-off = [] # `src/log/registry.rs`'s pointer store and load go `Relaxed`, and diff --git a/kernel/loom/Cargo.toml b/kernel/loom/Cargo.toml index 2b4df3d483f..d2ba72efec6 100644 --- a/kernel/loom/Cargo.toml +++ b/kernel/loom/Cargo.toml @@ -182,6 +182,9 @@ log-ring-publish-relaxed = [] # room that is not there. log-ring-tail-relaxed = [] log-ring-loads-swapped = [] +# The negative control of `tests/tlb_tags.rs`: `Tags::enter` with its pairing +# relaxed, which must red the model. +tlb-tags-enter-relaxed = [] # The negative control for the panic console's published descriptor. It drops # the publisher's `Release` fence between its odd mark and its words, so a diff --git a/kernel/loom/src/lib.rs b/kernel/loom/src/lib.rs index 3a6e874eecc..5d89f9b14b4 100644 --- a/kernel/loom/src/lib.rs +++ b/kernel/loom/src/lib.rs @@ -171,6 +171,9 @@ pub mod cpu_slice; #[path = "../../src/shootdown.rs"] pub mod shootdown; +#[path = "../../src/tlb_tags.rs"] +pub mod tlb_tags; + /// The CPU roster and the release/answer word, driven by `tests/smp_bringup.rs`; /// under `loom` alone, as is the [`arch::cpu`] shim it names. #[cfg(feature = "loom")] diff --git a/kernel/loom/tests/tlb_tags.rs b/kernel/loom/tests/tlb_tags.rs new file mode 100644 index 00000000000..11800cbf742 --- /dev/null +++ b/kernel/loom/tests/tlb_tags.rs @@ -0,0 +1,63 @@ +//! Loom: a change to a space's leaves reaches every CPU that could still use +//! the old translation, or that CPU flushes before it uses one. +//! +//! cpu 1 ran the space before (its TLB holds the mapped entry) and enters it +//! again while cpu 0 unmaps the page and asks who runs the space. Either +//! cpu 0's [`Tags::changed`] names cpu 1, which then owes a flush before the +//! page is freed, or cpu 1's [`Tags::enter`] answers that it must flush. The +//! model fails if neither holds and cpu 1 reads the page through the stale +//! entry. +//! +//! ```text +//! cargo test --manifest-path kernel/loom/Cargo.toml --features tlb-tags-enter-relaxed \ +//! --test tlb_tags +//! ``` +//! +//! relaxes `enter`'s pairing, and this file must red. + +#![cfg(feature = "loom")] + +use kernel_loom::tlb_tags::Tags; +use loom::sync::atomic::{AtomicBool, AtomicU64, Ordering::SeqCst}; +use loom::sync::Arc; + +const TAG: usize = 1; + +#[test] +fn a_change_reaches_a_runner_or_the_entering_cpu_flushes() { + loom::model(|| { + let tags = Arc::new(Tags::new()); + tags.size(2, 2); + let pte = Arc::new(AtomicU64::new(1)); + let reached = Arc::new(AtomicBool::new(false)); + + let initiator = { + let (tags, pte, reached) = (Arc::clone(&tags), Arc::clone(&pte), Arc::clone(&reached)); + loom::thread::spawn(move || { + pte.store(0, SeqCst); + tags.changed(TAG, |cpu| { + if cpu == 1 { + reached.store(true, SeqCst); + } + }); + }) + }; + + // cpu 1's TLB still holds the mapped entry from its last run. + let mut tlb = 1; + if tags.switch(1, TAG).flush { + let stamp = tags.flushing(); + tlb = pte.load(SeqCst); + tags.flushed(1, stamp); + } + let used = tlb; + initiator.join().unwrap(); + + // A reached cpu flushes before the frame is freed, which is the + // shootdown protocol's own property (`tlb_shootdown.rs`). + assert!( + reached.load(SeqCst) || used == pte.load(SeqCst), + "cpu 1 entered the space unreached and used a translation the change took away" + ); + }); +} diff --git a/kernel/src/arch/aarch64/paging.rs b/kernel/src/arch/aarch64/paging.rs index 5c95a886045..a4f854ab707 100644 --- a/kernel/src/arch/aarch64/paging.rs +++ b/kernel/src/arch/aarch64/paging.rs @@ -695,7 +695,9 @@ impl UserTables { /// Every CPU has dropped what it cached of `[start, end)` before this /// returns, and of the tables `unlinked` took out: a walk cache entry for /// another address under one of them is dropped only by the whole ASID. - pub fn shoot_down(&self, start: u64, end: u64, unlinked: &Unlinked) { + /// Answers whether it reached another CPU by interrupt: never, since + /// the invalidations broadcast in hardware. + pub fn shoot_down(&self, start: u64, end: u64, unlinked: &Unlinked) -> bool { const BY_PAGE: u64 = 64; let asid = self.asid.value(); if !unlinked.is_empty() || (end - start) / PAGE_4K > BY_PAGE { @@ -703,6 +705,7 @@ impl UserTables { } else { (start..end).step_by(PAGE_4K as usize).for_each(|va| tlb::page(asid, va)); } + false } } diff --git a/kernel/src/arch/x86_64/apic.rs b/kernel/src/arch/x86_64/apic.rs index ba3227be116..a6b8e5b6ca4 100644 --- a/kernel/src/arch/x86_64/apic.rs +++ b/kernel/src/arch/x86_64/apic.rs @@ -251,6 +251,13 @@ pub(super) fn tlb_ipi() { } } +/// Ask one CPU to flush its TLB. +pub(super) fn tlb_ipi_to(cpu_id: u32) { + if ENABLED.load(Ordering::Relaxed) { + send(Some(crate::smp::hardware_id(cpu_id)), 0x4000 | 0xFE); + } +} + /// Send the kick IPI to one CPU, waking it if halted. // Targeted, not broadcast: a broadcast kick would preempt every sibling per wake and cannot scale. pub fn kick_cpu(cpu_id: u32) { diff --git a/kernel/src/arch/x86_64/paging.rs b/kernel/src/arch/x86_64/paging.rs index c6b20b97316..3bcd9001470 100644 --- a/kernel/src/arch/x86_64/paging.rs +++ b/kernel/src/arch/x86_64/paging.rs @@ -337,6 +337,15 @@ pub fn flush_tlb_all() { } } +/// Flush this CPU's translation of the user page `va` in the space tagged +/// `tag`, which this CPU runs. +pub fn flush_user_page(tag: u16, va: u64) { + match pcid_active() { + Some(have) => crate::arch::cpu::invpcid(have, Invpcid::Address, tag, va), + None => crate::arch::cpu::invlpg(va), + } +} + /// Flush every TLB entry on this CPU but the global ones, all PCIDs: what a /// user-half change owes, since no user leaf is global. pub fn flush_tlb_user() { @@ -374,11 +383,13 @@ impl Cr3 { /// # Safety /// The underlying page tables must be valid and live. pub unsafe fn activate(self) { - if pcid_active().is_some() { - crate::arch::cpu::write_cr3(self.0 | CR3_NOFLUSH); - } else { - crate::arch::cpu::write_cr3(self.0); - } + crate::arch::tlb::switching(self.pcid(), || { + if pcid_active().is_some() { + crate::arch::cpu::write_cr3(self.0 | CR3_NOFLUSH); + } else { + crate::arch::cpu::write_cr3(self.0); + } + }); } /// Load CR3 with a flush; used during boot before PCID is enabled. @@ -930,8 +941,12 @@ impl UserTables { /// Every CPU has dropped what it cached of `[start, end)` and of the /// tables `unlinked` took out before this returns: every CPU's user /// translations go, paging-structure caches with them. - pub fn shoot_down(&self, _start: u64, _end: u64, _unlinked: &Unlinked) { - crate::arch::tlb::shootdown(crate::invalidation::Origin::Unmap, crate::invalidation::Half::User); + /// Answers whether it reached another CPU. + pub fn shoot_down(&self, start: u64, end: u64, unlinked: &Unlinked) -> bool { + // A table taken away is cached apart from the leaves: by page is not + // enough for it. + let (start, end) = if unlinked.is_empty() { (start, end) } else { (0, u64::MAX) }; + crate::arch::tlb::user_change(crate::invalidation::Origin::Unmap, self.pcid.value(), start, end) } } diff --git a/kernel/src/arch/x86_64/tlb.rs b/kernel/src/arch/x86_64/tlb.rs index 8665c65943f..1c94e8c6fae 100644 --- a/kernel/src/arch/x86_64/tlb.rs +++ b/kernel/src/arch/x86_64/tlb.rs @@ -6,9 +6,10 @@ //! memory-type translation surviving an early return is undefined per SDM //! Vol. 3A §11.12.4. //! -//! **The target set is every other CPU the machine brought up**, not the CPUs -//! sharing the initiator's address space: nothing a process does puts a CPU -//! into it or takes one out. +//! **A change to one user space's leaves reaches only the CPUs running it** +//! ([`user_change`], over `crate::tlb_tags`): a CPU that ran it before flushes +//! when it next enters it, if it changed since. Every other shootdown targets +//! every other CPU the machine brought up. //! //! **A user-half change keeps global translations**: its flush is `INVPCID` //! type 3 or a `CR3` reload, and only a kernel-half change, whose leaves are @@ -24,9 +25,90 @@ use crate::smp; static SHOOTDOWN: Shootdown = Shootdown::new(); +/// Which CPUs run each user space, by its PCID, and when each changed. +static USER: crate::tlb_tags::Tags = crate::tlb_tags::Tags::new(); + /// An answer for each of the `cpus` CPUs the firmware names. pub fn size(cpus: usize) { SHOOTDOWN.size(cpus); + USER.size(usize::from(kernel::pcid::MAX_USER_PCID) + 1, cpus); +} + +/// This CPU loads the root of the space tagged `to` with `load`: what +/// `paging::Cr3::activate` owes the targeting. It runs `to` from before the +/// load, flushing every user translation first where `to` changed since its +/// last, and leaves the space it ran only after it. +pub fn switching(to: u16, load: impl FnOnce()) { + const _: () = assert!(kernel::pcid::KERNEL_PCID == 0, "the targeting's tag 0 is no space"); + if !USER.sized() { + return load(); + } + let cpu = percpu::cpu_id() as usize; + let switched = USER.switch(cpu, usize::from(to)); + if switched.flush { + flush(Reach::Narrow); + } + load(); + USER.left(cpu, switched.from); +} + +/// Pages a change flushes one at a time on the initiator; past this, its +/// whole space. +const BY_PAGE: u64 = 64; + +/// The user space tagged `tag` changed the leaves of `[start, end)`, already +/// written: every CPU running it has flushed them before this returns, and +/// every other flushes before it next enters the space. So the change's +/// frames may go as soon as it returns. +/// Answers whether it reached another CPU. +pub fn user_change(origin: Origin, tag: u16, start: u64, end: u64) -> bool { + let me = percpu::cpu_id() as usize; + let mut targets = [0u64; 4]; + let mut beyond = false; + USER.changed(usize::from(tag), |cpu| match targets.get_mut(cpu / 64) { + Some(word) => *word |= 1 << (cpu % 64), + None => beyond = true, + }); + // A CPU past what the targets hold reaches every other, as before. + let Some(mine) = targets.get(me / 64).copied() else { + shootdown(origin, Half::User); + return true; + }; + // This CPU first: it may be running the space. + if mine & (1 << (me % 64)) != 0 { + if (end - start) / crate::mm::PAGE_SIZE <= BY_PAGE { + (start..end).step_by(crate::mm::PAGE_SIZE as usize).for_each(|va| crate::mm::paging::flush_user_page(tag, va)); + } else { + flush(Reach::Narrow); + } + targets[me / 64] &= !(1 << (me % 64)); + } + if beyond { + // More CPUs than the targets hold: every one of them, as before. + shootdown(origin, Half::User); + return true; + } + if targets.iter().all(|&word| word == 0) || !smp::answering() { + return false; + } + ISSUED[origin as usize].fetch_add(1, Ordering::Relaxed); + let began = crate::clock::nanos_since_boot(); + let generation = SHOOTDOWN.issue(Reach::Narrow); + let each = |f: &mut dyn FnMut(u32)| { + for (w, &word) in targets.iter().enumerate() { + let mut bits = word; + while bits != 0 { + f((w * 64 + bits.trailing_zeros() as usize) as u32); + bits &= bits - 1; + } + } + }; + each(&mut |cpu| apic::tlb_ipi_to(cpu)); + each(&mut |cpu| wait_for(me, cpu, generation)); + let took = crate::clock::nanos_since_boot().saturating_sub(began); + WAIT_NS.fetch_add(took, Ordering::Relaxed); + MAX_NS.fetch_max(took, Ordering::Relaxed); + true } use crate::invalidation::{Half, Origin}; @@ -160,10 +242,14 @@ fn wait_for(me: usize, cpu: u32, generation: Generation) { /// What `reach` asks of this CPU: every translation, or every one but the /// global ones a user-half change leaves. fn flush(reach: Reach) { + let stamp = USER.flushing(); match reach { Reach::Wide => crate::mm::paging::flush_tlb_all(), Reach::Narrow => crate::mm::paging::flush_tlb_user(), } + if USER.sized() { + USER.flushed(percpu::cpu_id() as usize, stamp); + } } /// Which side of a shootdown this CPU answers from: asked, by the interrupt or diff --git a/kernel/src/main.rs b/kernel/src/main.rs index 5d2065407d4..b7a26902b38 100644 --- a/kernel/src/main.rs +++ b/kernel/src/main.rs @@ -15,6 +15,7 @@ pub use mm::{UserAddr, DirectMap, PHYS_OFFSET}; mod invalidation; mod seqlock; mod shootdown; +mod tlb_tags; mod cpu_local; mod cpu_slice; mod sleeplock; diff --git a/kernel/src/mm/space.rs b/kernel/src/mm/space.rs index eb10025d0db..797c99b9cbe 100644 --- a/kernel/src/mm/space.rs +++ b/kernel/src/mm/space.rs @@ -64,9 +64,17 @@ pub struct Usage { /// Pages a fault populated, and the time faults took. faults: AtomicU64, fault_ns: AtomicU64, + /// Changes to the space's leaves that reached another CPU by interrupt. + reached: AtomicU64, } impl Usage { + /// Read only by the test kernel's `SYS_DEBUG`. + #[cfg(feature = "test-actuators")] + pub fn reached(&self) -> u64 { + self.reached.load(Relaxed) + } + pub fn resident_bytes(&self) -> u64 { self.resident.load(Relaxed) * PAGE_SIZE } @@ -339,7 +347,7 @@ impl AddressSpace { (Leaf::Held(_), None) | (Leaf::Empty, _) => None, }); if present { - self.tables.shoot_down(start, end, &Unlinked::none()); + self.shoot_down(start, end, &Unlinked::none()); } Ok(()) } @@ -378,6 +386,14 @@ impl AddressSpace { Ok((start.raw(), end)) } + /// Every CPU has dropped what it cached of `[start, end)` and of the + /// tables `unlinked` took out, counting the changes that reached another. + fn shoot_down(&self, start: u64, end: u64, unlinked: &Unlinked) { + if self.tables.shoot_down(start, end, unlinked) { + self.usage.reached.fetch_add(1, Relaxed); + } + } + /// [`release`](Self::release) of every page of `[start, start + len)` /// that still maps `object` where a mapping of it from its start at /// `start` would: a page the process unmapped or replaced since is left as @@ -424,7 +440,7 @@ impl AddressSpace { Leaf::Held(_) | Leaf::Empty => None, }); if present { - self.tables.shoot_down(start, end, &Unlinked::none()); + self.shoot_down(start, end, &Unlinked::none()); } let mut freed = 0; self.tables.take_owned(start, end, |frames| { @@ -438,7 +454,7 @@ impl AddressSpace { let inventory = &self.inventory; let unlinked = self.tables.unlink(start, end, |s, e| inventory.backed(s, e)); if !unlinked.is_empty() { - self.tables.shoot_down(start, end, &unlinked); + self.shoot_down(start, end, &unlinked); } unlinked.free(); self.usage.freed(freed); diff --git a/kernel/src/syscall/dispatch.rs b/kernel/src/syscall/dispatch.rs index 14816405016..6076eb83505 100644 --- a/kernel/src/syscall/dispatch.rs +++ b/kernel/src/syscall/dispatch.rs @@ -565,6 +565,7 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> DA::CANARY_CHANGED => canary::changed() as u64, DA::TLB_ACK_DELAY_ARM => crate::arch::tlb::debug_arm_ack_delay(a2), DA::TLB_ACK_DELAY_DISARM => crate::arch::tlb::debug_disarm_ack_delay(), + DA::SPACE_REACHED => crate::process::current_address_space().usage().reached(), // The two leaks this object model accepts — a stranded Arc, the cross-pair // connection cycle — are visible in no other way than this census. // Per kind, not total: a total would hide one kind's leak behind another's churn. diff --git a/kernel/src/tlb_tags.rs b/kernel/src/tlb_tags.rs new file mode 100644 index 00000000000..232ae890ed5 --- /dev/null +++ b/kernel/src/tlb_tags.rs @@ -0,0 +1,183 @@ +//! Which CPUs a change to one user space's leaves must reach: those running +//! the space, and no other. A CPU that ran the space before keeps its +//! translations under the space's tag (x86 keeps them across `CR3` loads +//! under PCID), so instead of being reached it flushes when it next enters the +//! space, if the space changed since its own last flush of every user +//! translation. +//! +//! **The pairing that makes it sound** is [`Tags::enter`] against +//! [`Tags::changed`]: a CPU marks itself running the tag, then reads when the +//! tag last changed; a change stamps the tag, then reads who runs it. A +//! `SeqCst` fence sits between each side's store and load, so either the +//! entering CPU sees the stamp and flushes, or the change sees the CPU and +//! reaches it. The stamp is released after the leaves the change wrote and +//! acquired by the entering CPU, so its flush follows them. +//! +//! Compiled a second time by `kernel-loom`, so it names nothing of the kernel +//! but [`CpuSlice`]. + +#[cfg(not(feature = "loom"))] +use core::sync::atomic::{fence, AtomicU64, AtomicUsize, Ordering}; + +#[cfg(feature = "loom")] +use loom::sync::atomic::{fence, AtomicU64, AtomicUsize, Ordering}; + +use crate::cpu_slice::CpuSlice; + +/// The fence each side of the pairing puts between its store and its load: +/// without it either load may be satisfied before the other side's store. +#[cfg(not(feature = "tlb-tags-enter-relaxed"))] +fn pair() { + fence(Ordering::SeqCst); +} + +// Kernel-loom's negative-control mutation, never selected by a kernel build. +#[cfg(feature = "tlb-tags-enter-relaxed")] +fn pair() {} + +/// What [`Tags::switch`] answers. +pub struct Switched { + /// The tag left, 0 for none, to give [`Tags::left`] after the load. + pub from: usize, + /// Every user translation must be flushed before the load. + pub flush: bool, +} + +pub struct Tags { + /// Monotonic: one more for every change to any user space's leaves. + stamp: AtomicU64, + /// Per tag, the stamp of its last change. + changed: CpuSlice, + /// Per tag, a bit per CPU running it: `words` words a tag. + running: CpuSlice, + words: AtomicUsize, + /// Per CPU, the stamp read before its last flush of every user + /// translation: a tag changed since needs another. + flushed: CpuSlice, + /// Per CPU, the tag it runs, 0 for none. + on: CpuSlice, +} + +impl Tags { + /// Must stay `const`: the kernel's instance is a `static`, [`Self::size`]d + /// before an AP is attempted. + #[cfg(not(feature = "loom"))] + pub const fn new() -> Self { + Self { + stamp: AtomicU64::new(0), + changed: CpuSlice::new(), + running: CpuSlice::new(), + words: AtomicUsize::new(0), + flushed: CpuSlice::new(), + on: CpuSlice::new(), + } + } + + // Not `const`: loom's atomics have no const constructor. + #[allow(clippy::new_without_default)] + #[cfg(feature = "loom")] + pub fn new() -> Self { + Self { + stamp: AtomicU64::new(0), + changed: CpuSlice::new(), + running: CpuSlice::new(), + words: AtomicUsize::new(0), + flushed: CpuSlice::new(), + on: CpuSlice::new(), + } + } + + /// Room for `tags` tags on `cpus` CPUs. Once. + pub fn size(&self, tags: usize, cpus: usize) { + let words = cpus.div_ceil(64); + self.words.store(words, Ordering::Relaxed); + self.changed.size(tags, |_| AtomicU64::new(0)); + self.running.size(tags * words, |_| AtomicU64::new(0)); + self.flushed.size(cpus, |_| AtomicU64::new(0)); + self.on.size(cpus, |_| AtomicUsize::new(0)); + } + + /// Whether [`Self::size`] has run: before it the boot CPU runs alone, no + /// user space exists, and a CPU may not yet know its own number. + pub fn sized(&self) -> bool { + !self.on.is_empty() + } + + /// `cpu` is about to run the space tagged `to`, 0 for none, in place of + /// the one it runs: called before it loads `to`'s root. Answers the tag it + /// leaves, for [`Self::left`] once that root is replaced, and whether it + /// must flush every user translation first, since `to` changed after its + /// last. A CPU counts as running a space from before its root is loaded + /// until after it is replaced, so for that instant it runs both: the + /// walker may still use the old root's tables until the load. + pub fn switch(&self, cpu: usize, to: usize) -> Switched { + // Before sizing the boot CPU runs alone and no user space exists. + let Some(on) = self.on.get(cpu) else { return Switched { from: 0, flush: false } }; + let from = on.swap(to, Ordering::Relaxed); + if from == to { + return Switched { from: 0, flush: false }; + } + Switched { from, flush: to != 0 && self.enter(cpu, to) } + } + + /// `cpu` has replaced the root of the space tagged `from`, which + /// [`Self::switch`] answered. + pub fn left(&self, cpu: usize, from: usize) { + if from != 0 { + self.leave(cpu, from); + } + } + + fn bit(&self, tag: usize, cpu: usize) -> (&AtomicU64, u64) { + let words = self.words.load(Ordering::Relaxed); + (&self.running[tag * words + cpu / 64], 1 << (cpu % 64)) + } + + /// `cpu` runs `tag` from here; `true` where it must flush every user + /// translation before it does, since the tag changed after its last. + fn enter(&self, cpu: usize, tag: usize) -> bool { + let (word, bit) = self.bit(tag, cpu); + word.fetch_or(bit, Ordering::Relaxed); + pair(); + // Acquire: a flush this answers for follows the leaves the change wrote. + self.changed[tag].load(Ordering::Acquire) > self.flushed[cpu].load(Ordering::Relaxed) + } + + /// `cpu` runs `tag` no more: it touches none of its memory from here. + fn leave(&self, cpu: usize, tag: usize) { + let (word, bit) = self.bit(tag, cpu); + word.fetch_and(!bit, Ordering::Release); + } + + /// The stamp a flush of every user translation is about to cover, read + /// before it, and given to [`Self::flushed`] after it. + pub fn flushing(&self) -> u64 { + self.stamp.load(Ordering::SeqCst) + } + + /// `cpu` flushed every user translation, covering `stamp`. + pub fn flushed(&self, cpu: usize, stamp: u64) { + if let Some(own) = self.flushed.get(cpu) { + own.fetch_max(stamp, Ordering::Relaxed); + } + } + + /// `tag`'s leaves changed: call `reach` for each CPU running it now, + /// which must flush before the change's frames go. Every other CPU flushes + /// when it next enters the tag. + pub fn changed(&self, tag: usize, mut reach: impl FnMut(usize)) { + let Some(changed) = self.changed.get(tag) else { return }; + let stamp = self.stamp.fetch_add(1, Ordering::Relaxed) + 1; + // Release: the leaves written before are visible to who reads the stamp. + changed.fetch_max(stamp, Ordering::Release); + fence(Ordering::SeqCst); + let words = self.words.load(Ordering::Relaxed); + for w in 0..words { + let mut bits = self.running[tag * words + w].load(Ordering::Relaxed); + while bits != 0 { + reach(w * 64 + bits.trailing_zeros() as usize); + bits &= bits - 1; + } + } + } +} diff --git a/sdk/abi/src/syscall.rs b/sdk/abi/src/syscall.rs index 72d46365c44..a3816ab6f03 100644 --- a/sdk/abi/src/syscall.rs +++ b/sdk/abi/src/syscall.rs @@ -909,6 +909,10 @@ pub mod debug_action { /// whichever comes first. pub const TLB_ACK_DELAY_ARM: u64 = 12; pub const TLB_ACK_DELAY_DISARM: u64 = 13; + /// How many changes to the caller's own address space have reached + /// another CPU by interrupt: what a change no other CPU runs the space on + /// must leave as it was. + pub const SPACE_REACHED: u64 = 29; /// How many kernel objects of one kind are alive right now. The argument is /// an [`ObjectKind`](crate::handle::ObjectKind) word. /// diff --git a/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs b/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs index 42653bad344..b0d33449983 100644 --- a/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs +++ b/tests/guest/rust-tests/src/bin/tlb_shootdown_waits.rs @@ -1,5 +1,5 @@ -//! A shootdown returns when every other CPU has flushed — and the paths that -//! free memory are behind it. +//! A shootdown returns when every CPU running the space has flushed — and the +//! paths that free memory are behind it; one no other CPU runs reaches none. //! //! **Why this needs an actuator at all.** A correct wait and no wait whatsoever //! measure the same zero on a machine where every CPU answers in microseconds, @@ -10,9 +10,12 @@ //! is a small number here rather than an invisible one, and the set's *width* //! is assertable and not only its depth. //! -//! **The precondition is the kernel's target set, and that set is every other -//! CPU on the machine.** Nothing this process does puts a CPU into it or takes -//! one out, so `SYS_CPU_COUNT` is the whole of the arrangement. +//! **The target set is the CPUs running the space.** So the waits are timed +//! while a spinning thread of this process runs on every other CPU the +//! scheduler gives one, and the absence of a wait while this thread runs +//! alone is counted (`SYS_DEBUG`'s `SPACE_REACHED`): no change reached +//! another CPU. It runs on the test kernel: the `shared-debug` metal boot, and +//! a q35 boot of the test kernel. //! //! **What is gated is the wait, not the harm.** A sibling reading through a //! stale translation is not constructible under TCG — the correct outcome is a @@ -43,7 +46,7 @@ const TRIALS: u32 = 5; const PAGE_2M: usize = 2 * 1024 * 1024; -use toyos_abi::syscall::debug_action::{TLB_ACK_DELAY_ARM as ARM, TLB_ACK_DELAY_DISARM as DISARM}; +use toyos_abi::syscall::debug_action::{SPACE_REACHED, TLB_ACK_DELAY_ARM as ARM, TLB_ACK_DELAY_DISARM as DISARM}; fn debug(action: u64, arg: u64) -> u64 { let ret: u64; @@ -98,6 +101,37 @@ fn map(size: usize) -> *mut u8 { p } +/// A thread of this process spinning on each other CPU it is given, until +/// told to stop: the CPUs a change to this space must reach. +struct Spinners { + stop: std::sync::Arc, + threads: Vec>, +} + +impl Spinners { + fn start(cpus: u32) -> Self { + let stop = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let threads = (1..cpus) + .map(|_| { + let stop = std::sync::Arc::clone(&stop); + std::thread::spawn(move || { + while !stop.load(std::sync::atomic::Ordering::Relaxed) { + core::hint::spin_loop(); + } + }) + }) + .collect(); + Self { stop, threads } + } + + fn stop(self) { + self.stop.store(true, std::sync::atomic::Ordering::Relaxed); + for t in self.threads { + t.join().expect("a spinner"); + } + } +} + fn main() { // The set the kernel computes, asked of the kernel. A machine with one CPU // takes `shootdown`'s local-flush return, which issues no IPI and waits for @@ -111,6 +145,7 @@ fn main() { // 1. `munmap`, which is the syscall the stage exists for: the pages go back // to the PMM behind the flush. + let spinners = Spinners::start(cpus); let mut judged = false; for trial in 1..=TRIALS { let region = map(PAGE_2M); @@ -136,22 +171,53 @@ fn main() { // free: the address keeps its meaning and changes what it names, so a // sibling holding the old translation writes into the wrong physical // page with nothing ever faulting. - let placed = map(PAGE_2M); - armed(cpus); - let elapsed = timed(|| { - let p = unsafe { - syscall::vm_reserve(PAGE_2M, VmProt::ReadWrite, VmAt::Fixed(placed)).unwrap_or(core::ptr::null_mut()) - }; - assert_eq!(p, placed, "MAP_FIXED did not honour the address"); - }); + // The page is touched first: a translation of a page never touched is + // held by no CPU, so replacing it owes no flush. + // Tried as `munmap` is: a spinner the scheduler has just moved off its + // CPU is a target the change rightly does not wait for. + let mut judged = false; + for trial in 1..=TRIALS { + let placed = map(PAGE_2M); + unsafe { placed.write_volatile(1) }; + armed(cpus); + let elapsed = timed(|| { + let p = unsafe { + syscall::vm_reserve(PAGE_2M, VmProt::ReadWrite, VmAt::Fixed(placed)).unwrap_or(core::ptr::null_mut()) + }; + assert_eq!(p, placed, "MAP_FIXED did not honour the address"); + }); + unsafe { syscall::vm_release(placed, PAGE_2M) }.expect("munmap the fixed mapping"); + if elapsed >= FLOOR_NANOS { + judged = true; + break; + } + println!("trial {trial}: a fixed mmap returned in {elapsed}ns, under the {FLOOR_NANOS}ns floor"); + } assert!( - elapsed >= FLOOR_NANOS, - "a fixed mmap returned in {elapsed}ns with every other CPU answering {DELAY_NANOS}ns \ - late — it replaced the mapping without waiting for the flush", + judged, + "every one of {TRIALS} fixed mmaps returned in under {FLOOR_NANOS}ns with every other CPU \ + answering {DELAY_NANOS}ns late — it replaced the mapping without waiting for the flush", + ); + spinners.stop(); + + // 3. With no other thread of this process left, a change reaches no + // other CPU: counted, not timed, since a guest's clock bounds nothing + // from above. + let alone = map(PAGE_2M); + unsafe { alone.write_volatile(1) }; + let before = debug(SPACE_REACHED, 0); + unsafe { syscall::vm_release(alone, PAGE_2M) }.expect("munmap alone"); + let after = debug(SPACE_REACHED, 0); + assert_eq!( + after, before, + "a munmap no other CPU's thread could reach interrupted another CPU {} time(s)", + after - before, ); - unsafe { syscall::vm_release(placed, PAGE_2M) }.expect("munmap the fixed mapping"); disarm(); - println!("a shootdown waits for every other CPU, and munmap and a fixed mmap wait for it"); + println!( + "a shootdown waits for every CPU running the space, munmap and a fixed mmap wait for it, \ + and a space no other CPU runs waits for none" + ); } From 0f51175d05cd1248c3cc27797a132422a8c8a7c7 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 22 Oct 2026 15:00:49 +0200 Subject: [PATCH 04/10] x86: a Ring 3 page fault does not seal the black box Every exception entry sealed its registers into the black box before its handler ran and wrote the whole 16 KiB box back out of the caches: 256 `CLFLUSH`es. With user memory populated a 4 KiB page at a time, that ran on every first touch of every page, from every CPU at once, into one box. On the T14 at 3446e9c33 a first touch cost 6.1 us a page (osbench `mem.first_touch.4k`), against Linux's 0.93 us on the same machine; that the write-back is most of the difference is inferred, not yet measured. A Ring 3 page fault no longer seals. It is served, or it ends its process through the crash path, which reports; a fault inside its handler is a Ring 0 fault, which still seals before anything else. AArch64 records only a fault it does not serve, and is unchanged. The black box's single-writer claim is corrected: two CPUs taking a kernel fault or an exception at once still write the one box, recorded in `issues/two-cpus-faulting-at-once-write-one-black-box.md`. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...us-faulting-at-once-write-one-black-box.md | 23 +++++++++++++++++++ kernel/src/arch/x86_64/idt/mod.rs | 11 ++++++++- kernel/src/blackbox.rs | 7 ++++-- 3 files changed, 38 insertions(+), 3 deletions(-) create mode 100644 issues/two-cpus-faulting-at-once-write-one-black-box.md diff --git a/issues/two-cpus-faulting-at-once-write-one-black-box.md b/issues/two-cpus-faulting-at-once-write-one-black-box.md new file mode 100644 index 00000000000..ae62ae538d9 --- /dev/null +++ b/issues/two-cpus-faulting-at-once-write-one-black-box.md @@ -0,0 +1,23 @@ +--- +status: open +kind: defect +opened: 2026-10-22 +--- + +# Two CPUs faulting at once write one black box + +Every exception entry but a Ring 3 page fault seals its registers into the +black box before its handler runs (`kernel/src/arch/x86_64/idt/mod.rs`, +`trap_dispatch`, into `kernel/src/blackbox.rs`'s `record_fault`). The box is +one page set for the machine, and `with_page` hands each writer a `&mut` over +it with no exclusion between CPUs: two CPUs taking a kernel-mode fault or an +exception at the same moment write it at once. That is an aliasing +`&mut` and a data race, and one record can tear the other's under a checksum +the second writer computed, so the next boot reads a record no CPU wrote. + +Rare: a kernel-mode fault is a kernel bug, and the Ring 3 page fault, the one +ordinary work takes, no longer seals. Found in review of #1008, which stopped +user page faults from sealing. + +Owner: the black box. Exit: an exception record per CPU, or one exclusion +that cannot deadlock an exception entry, with a host test of two writers. diff --git a/kernel/src/arch/x86_64/idt/mod.rs b/kernel/src/arch/x86_64/idt/mod.rs index c04a42b7d42..1bc00b3eb22 100644 --- a/kernel/src/arch/x86_64/idt/mod.rs +++ b/kernel/src/arch/x86_64/idt/mod.rs @@ -338,7 +338,16 @@ extern "sysv64" fn trap_dispatch(frame: *mut TrapFrame) { // vector, the page fault included: a diagnostic that skips the commonest // fault class is one that misses the answer, and a returning fault's record // is simply overwritten by the next. - crate::blackbox::record_fault(&fault_of(frame)); + // + // **But not a Ring 3 page fault**, the one fault ordinary work takes on + // every first touch of every page: it is served or it ends its process + // through the crash path, which reports, and a fault inside its handler is + // a Ring 0 fault that records itself here. Sealing it would write back the + // whole box on every first touch, from every CPU at once. + let user_page_fault = Vector::from_raw(frame.vector) == Vector::PageFault && toyos_userbound::Ring::of_cs(frame.cs).is_user(); + if !user_page_fault { + crate::blackbox::record_fault(&fault_of(frame)); + } match Vector::from_raw(frame.vector) { Vector::DoubleFault => exceptions::double_fault_handler(frame), Vector::MachineCheck => exceptions::machine_check_handler(frame), diff --git a/kernel/src/blackbox.rs b/kernel/src/blackbox.rs index f37b1955ec4..1b251f43ba3 100644 --- a/kernel/src/blackbox.rs +++ b/kernel/src/blackbox.rs @@ -202,8 +202,11 @@ fn with_page(write: impl FnOnce(&mut [u8; BYTES], u64, toyos_blackbox::Identity) // SAFETY: `at` is non-zero only where `arm` found the address on this // boot's parameter line, which is the box the loader allocated and which // `mm::init` was told to keep out of the allocator. The panic path holds - // `PAINTING` and the quiesce path has stopped every other CPU, so the one - // CPU still running is the only writer. + // `PAINTING` and the quiesce path has stopped every other CPU, so there + // one CPU writes. **An exception entry does not exclude another CPU's**: + // two CPUs taking a kernel-mode fault or an exception at once both write + // the box, and one record may tear the other's + // (`issues/two-cpus-faulting-at-once-write-one-black-box.md`). let page = unsafe { &mut *(at as *mut [u8; BYTES]) }; // Both carried forward and neither verified: this runs where nothing may be // checked, and the checksum the write puts down is what covers them. A box From 01a82698575078d898bdacbc87f0e81fdbec83ae Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 22 Oct 2026 15:09:58 +0200 Subject: [PATCH 05/10] mm: a spawn copies each page of its program at the page's first touch A spawn copied every byte of its program's segments into the new process when it made it. On the T14 at 3446e9c33 osbench's large program, 32 MiB of data it never reads, took 6.6 ms to spawn and wait, against 0.72 ms on main, which paged the image in at faults, and 0.054 ms on Linux. A segment out of an image object is now a copied range (`kernel::vm`'s `Kind::Copy`): private, committed memory whose page starts as the object's bytes at its offset, as far as the segment's file bytes go, and zeros past them, copied when the page is first touched. That is the promise `SharedImage` already makes, a change after the hand-off reaching only pages not yet read. A discarded page is copied again at its next touch. A program from any other backing is copied in whole, as before. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...aths-are-unmeasured-against-2-mib-pages.md | 5 ++- kernel/pure/vm.rs | 37 +++++++++++++++- kernel/src/file_backing.rs | 11 +++++ kernel/src/loader/mod.rs | 15 ++++++- kernel/src/mm/space.rs | 44 ++++++++++++++++--- 5 files changed, 103 insertions(+), 9 deletions(-) diff --git a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md index 09d711e742c..831487d5a89 100644 --- a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md +++ b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md @@ -21,7 +21,10 @@ did, and none is measured on the T14 against the base or against Linux: space's lock: 1/512 of the range. - **Copies.** One round trip of the space's lock per 16 pages, plus an inventory lookup and a table walk per page. -- **Spawn.** The whole image is copied when the process is made. +- **Spawn.** A segment out of an image object is copied a page at a time at + first touch, so while a child lives the image is held twice: once by the + spawner's object, which the child's ranges keep alive, and once promised + to the child, until shared text removes one. - **Shootdowns.** A change to a present leaf reaches the CPUs running the space, each of which flushes every user translation, and a CPU entering a diff --git a/kernel/pure/vm.rs b/kernel/pure/vm.rs index 12c7f152a7f..1d53aacdcd2 100644 --- a/kernel/pure/vm.rs +++ b/kernel/pure/vm.rs @@ -75,6 +75,11 @@ pub enum Kind { Private { committed: bool }, /// `object`'s bytes from `offset` on, never given more than `max` allows. Object { object: O, offset: u64, max: Prot }, + /// Memory of this space's own, committed, whose every page starts as a + /// copy of `source` from `offset` on as far as the range's first `len` + /// bytes go, and zeros past them: taken at the page's first touch, as a + /// program's segments are. + Copy { source: O, offset: u64, len: u64 }, } #[derive(Clone, Debug)] @@ -85,12 +90,18 @@ pub struct Entry { } impl Entry { + /// Whether its pages are this space's own, committed: each owns a frame + /// once touched, and a promise of one before. + pub fn committed(&self) -> bool { + matches!(self.kind, Kind::Private { committed: true } | Kind::Copy { .. }) + } + /// Whether the range needs page tables: memory a fault or a copy can /// populate or reach. pub fn backed(&self) -> bool { match self.kind { Kind::Private { committed } => committed, - Kind::Object { .. } => true, + Kind::Object { .. } | Kind::Copy { .. } => true, } } @@ -101,6 +112,9 @@ impl Entry { Kind::Object { object, offset, max } => { Kind::Object { object: object.clone(), offset: offset + (at - start), max: *max } } + Kind::Copy { source, offset, len } => { + Kind::Copy { source: source.clone(), offset: offset + (at - start), len: len.saturating_sub(at - start) } + } }; Self { end: self.end, prot: self.prot, kind } } @@ -113,6 +127,9 @@ impl Entry { (Kind::Object { object: a, offset: x, max: m }, Kind::Object { object: b, offset: y, max: n }) => { a.same(b) && x + (self.end - start) == *y && m == n } + (Kind::Copy { source: a, offset: x, len: l }, Kind::Copy { source: b, offset: y, len: k }) => { + a.same(b) && x + (self.end - start) == *y && l.saturating_sub(self.end - start) == *k + } _ => false, } } @@ -396,6 +413,24 @@ mod tests { assert_eq!(starts(&v), vec![(FLOOR, FLOOR + PAGE), (FLOOR + 6 * PAGE, FLOOR + 8 * PAGE)]); } + /// A copied range cut in two keeps, in its tail, where the copy goes on + /// and how much of it is left; a tail past the copied bytes is all zeros; + /// and the halves made one again are the range they were. + #[test] + fn a_copied_range_cut_keeps_its_source_offset_and_length() { + let mut v = Inventory::new(FLOOR, TOP); + let copy = |end, offset, len| Entry { end, prot: Prot::ReadWrite, kind: Kind::Copy { source: Obj(7), offset, len } }; + v.insert(FLOOR, copy(FLOOR + 4 * PAGE, 3 * PAGE, PAGE + 100)); + v.update(FLOOR + PAGE, FLOOR + 2 * PAGE, |_, e| e.prot = Prot::Read); + let kind = |at| v.find(at).unwrap().1.kind.clone(); + assert!(matches!(kind(FLOOR), Kind::Copy { offset, len, .. } if offset == 3 * PAGE && len == PAGE + 100)); + assert!(matches!(kind(FLOOR + PAGE), Kind::Copy { offset, len, .. } if offset == 4 * PAGE && len == 100)); + assert!(matches!(kind(FLOOR + 2 * PAGE), Kind::Copy { offset, len, .. } if offset == 5 * PAGE && len == 0)); + assert!(v.find(FLOOR).unwrap().1.committed(), "a copied range is committed"); + v.update(FLOOR + PAGE, FLOOR + 2 * PAGE, |_, e| e.prot = Prot::ReadWrite); + assert_eq!(starts(&v), vec![(FLOOR, FLOOR + 4 * PAGE)], "the cut halves did not join again"); + } + #[test] fn neighbours_that_continue_each_other_join() { let mut v = Inventory::new(FLOOR, TOP); diff --git a/kernel/src/file_backing.rs b/kernel/src/file_backing.rs index 33fdd641540..44165e9b8fa 100644 --- a/kernel/src/file_backing.rs +++ b/kernel/src/file_backing.rs @@ -24,6 +24,13 @@ pub trait FileBacking: Send + Sync { /// Total file size in bytes. fn file_size(&self) -> u64; + + /// The memory object the file's bytes are, from its start, where they + /// are one: what a spawn copies a segment out of page by page as each is + /// first touched, rather than whole when the process is made. + fn object(&self) -> Option<&Arc> { + None + } } /// An executable in a shared memory object handed over by handle: one a @@ -70,4 +77,8 @@ impl FileBacking for SharedImage { fn file_size(&self) -> u64 { self.size } + + fn object(&self) -> Option<&Arc> { + Some(&self.object) + } } diff --git a/kernel/src/loader/mod.rs b/kernel/src/loader/mod.rs index 14d7960aca0..eb5170c2f67 100644 --- a/kernel/src/loader/mod.rs +++ b/kernel/src/loader/mod.rs @@ -85,7 +85,10 @@ pub(crate) fn read_file_range( /// Commit one range of private memory per `PT_LOAD` segment at `image_start` -/// plus its offset, its file bytes copied in and the rest of it zero. +/// plus its offset, its file bytes and the rest of it zero: out of an image +/// object, each page copied at its first touch, as `file_backing::SharedImage` +/// promises a change after the hand-off reaches only pages not yet read; out +/// of any other backing, copied in whole now. /// /// `Err` when two segments would share a page, which their ranges cannot, or /// when the file's bytes cannot be read: checked before the first range, and @@ -109,11 +112,19 @@ fn load_segments( if seg_end == seg_start { continue; } - space.reserve(seg_end - seg_start, At::Fixed(UserAddr::new(seg_start)), segment_prot(seg))?; // The file's bytes from the segment's first page: the bytes of that // page before the segment are the file's too, as a linker lays them. let file_start = seg.file_offset() & !4095; let file_bytes = seg.filesz() + seg.file_offset() % 4096; + let at = At::Fixed(UserAddr::new(seg_start)); + if let Some(object) = backing.object() { + // Each page copied out of the object at its first touch, the file + // past its end and the segment past its file size reading zeros. + let copied = file_bytes.min(backing.file_size().saturating_sub(file_start)); + space.copy(seg_end - seg_start, at, segment_prot(seg), Arc::clone(object), file_start, copied)?; + continue; + } + space.reserve(seg_end - seg_start, at, segment_prot(seg))?; let mut done = 0; while done < file_bytes { if backing.read_page(file_start + done, &mut page).is_err() { diff --git a/kernel/src/mm/space.rs b/kernel/src/mm/space.rs index 797c99b9cbe..a44941646b9 100644 --- a/kernel/src/mm/space.rs +++ b/kernel/src/mm/space.rs @@ -257,11 +257,23 @@ impl AddressSpace { /// `len` bytes of private memory, committed unless `prot` is `None`, which /// reserves the range and nothing more. pub fn reserve(&mut self, len: u64, at: At, prot: Prot) -> Result { + self.private(len, at, prot, Kind::Private { committed: prot != Prot::None }) + } + + /// `len` bytes of private memory, committed, whose pages start as a copy + /// of `source` from `offset` as far as their first `copied` bytes go and + /// zeros past them, each taken at its first touch: a program's segment + /// read lazily out of the object it was handed in. + pub fn copy(&mut self, len: u64, at: At, prot: Prot, source: Arc, offset: u64, copied: u64) -> Result { + self.private(len, at, prot, Kind::Copy { source, offset, len: copied }) + } + + fn private(&mut self, len: u64, at: At, prot: Prot, kind: Kind>) -> Result { self.room_at(len, at)?; let start = self.locate(len, at)?; - let committed = prot != Prot::None; - self.inventory.insert(start, Entry { end: start + len, prot, kind: Kind::Private { committed } }); - if committed { + let backed = !matches!(kind, Kind::Private { committed: false }); + self.inventory.insert(start, Entry { end: start + len, prot, kind }); + if backed { self.tables.build(start, start + len); } self.usage.made.fetch_add(1, Relaxed); @@ -313,7 +325,7 @@ impl AddressSpace { self.room(start, end, 0)?; let allowed = self.inventory.meeting(start, end).all(|(_, entry)| match entry.kind { Kind::Object { max, .. } => prot.within(max), - Kind::Private { .. } => true, + Kind::Private { .. } | Kind::Copy { .. } => true, }); if !allowed { return Err(SyscallError::PermissionDenied); @@ -366,7 +378,7 @@ impl AddressSpace { pub fn decommit(&mut self, start: UserAddr, len: u64) -> Result<(), SyscallError> { let (start, end) = self.held(start, len)?; self.room(start, end, 0)?; - if !self.inventory.meeting(start, end).all(|(_, entry)| matches!(entry.kind, Kind::Private { .. })) { + if !self.inventory.meeting(start, end).all(|(_, entry)| matches!(entry.kind, Kind::Private { .. } | Kind::Copy { .. })) { return Err(SyscallError::InvalidArgument); } self.inventory.update(start, end, |_, entry| { @@ -572,9 +584,31 @@ fn page_leaf(start: u64, entry: &Entry>, page: u64) -> Opti Kind::Object { object, offset, .. } => { Leaf::Mapped(Mapping { phys: object.page(offset + (page - start)), prot, owned: false, cache: object.cache() }) } + Kind::Copy { source, offset, len } => { + let Some(frame) = super::pmm::alloc_frame() else { + crate::log!("space: no frame to populate {page:#x}"); + return None; + }; + let phys = frame.into_raw(); + copy_in(source, offset + (page - start), len.saturating_sub(page - start), phys); + Leaf::Mapped(Mapping { phys, prot, owned: true, cache: CachePolicy::Normal }) + } }) } +/// Write `source`'s bytes from `offset` into the zeroed frame at `phys`, as +/// many of the page's as `len` and the object both hold. +fn copy_in(source: &SharedMemObject, offset: u64, len: u64, phys: u64) { + let n = len.min(PAGE_SIZE).min(source.size().saturating_sub(offset)) as usize; + if n == 0 { + return; + } + // SAFETY: a frame just allocated to this page, which no leaf maps yet, so + // nothing else reaches it; `n` bytes stay inside it. + let frame = unsafe { core::slice::from_raw_parts_mut(super::DirectMap::from_phys(phys).as_mut_ptr::(), n) }; + source.read(offset, frame); +} + /// Every frame the space owns goes back; its tables go with `UserTables`. /// /// Not through [`AddressSpace::clear`]: no thread runs in a space that drops, From bed72c5935c9e3a935a85cc99471ed4e260e580f Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 23 Oct 2026 00:33:39 +0200 Subject: [PATCH 06/10] mm: a spawn from /tmp copies each page at its first touch, from a snapshot A spawn copied a program's segments page by page at first touch only when they came out of an image object; a program in the kernel's /tmp was copied in whole when the process was made. On the T14 at dadeebe0a osbench's large program, 32 MiB of data in /tmp it never reads, took 6.4 ms to spawn and wait against 0.74 ms at the base, which paged it in at faults. A copied range's source is now any `CopySource`: an image object, or a snapshot of a /tmp file (`file_cache::Snapshot`). A snapshot holds a reference to each of the file's pages as they are at the spawn, and a write to a shared page writes a copy of it, so a later write, truncation or delete of the file never reaches the child: it runs the bytes it was spawned from. Its reads take no lock, which matters because a first touch copies under the space's lock and a read(2) holds the file cache's lock while it copies into user memory. The snapshot itself is taken before the child's space is locked. Every backing now has a source, so the whole-copy path is gone. `spawn_from_tmp` overwrites and deletes the file the moment its child is spawned, then has the child read 256 KiB of its data; with the source reading the file live it reds. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...b-pages-still-copies-each-programs-text.md | 6 +- ...aths-are-unmeasured-against-2-mib-pages.md | 10 ++-- kernel/pure/vm.rs | 30 +++++----- kernel/src/file_backing.rs | 14 ++--- kernel/src/file_cache.rs | 57 +++++++++++++++++-- kernel/src/loader/mod.rs | 47 +++++++-------- kernel/src/mm/space.rs | 37 +++++++++--- kernel/src/tmpfs.rs | 8 +++ .../rust-tests/src/bin/spawn_from_tmp.rs | 43 ++++++++++++++ 9 files changed, 182 insertions(+), 70 deletions(-) create mode 100644 tests/guest/rust-tests/src/bin/spawn_from_tmp.rs diff --git a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md index ed6e0c0f882..7061ecde0d0 100644 --- a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md +++ b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md @@ -83,8 +83,10 @@ live split of a user leaf. remains of the stage's exit — `mmap` past what memory can hold refused at `mmap` — is the strict charge of the allocator track's migration (`issues/toyos-has-its-own-allocator.md`). -5. **Shared text.** A spawn copies its program's segments into the process - whole; no fault reads a file. ROOT's read-only text shared zero-copy, from +5. **Shared text.** A spawn copies each page of its program's segments into + the process at the page's first touch, out of the image object or out of + a snapshot of its `/tmp` file; no fault reads a file. ROOT's read-only + text shared zero-copy, from immutable sealed objects, is the next step. Exit: two processes of one binary hold its text once. 6. **`/apps` image sharing.** The design is the owner's decision, owed when diff --git a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md index 831487d5a89..0a5a6050233 100644 --- a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md +++ b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md @@ -21,10 +21,12 @@ did, and none is measured on the T14 against the base or against Linux: space's lock: 1/512 of the range. - **Copies.** One round trip of the space's lock per 16 pages, plus an inventory lookup and a table walk per page. -- **Spawn.** A segment out of an image object is copied a page at a time at - first touch, so while a child lives the image is held twice: once by the - spawner's object, which the child's ranges keep alive, and once promised - to the child, until shared text removes one. +- **Spawn.** A segment is copied a page at a time at first touch, so while a + child lives the image is held twice: once by the spawner's object or the + `/tmp` file's snapshot, which the child's ranges keep alive, and once + promised to the child, until shared text removes one. A snapshot takes a + reference to each resident page of the file under the file cache's lock, + and 16 bytes a page, at the spawn. - **Shootdowns.** A change to a present leaf reaches the CPUs running the space, each of which flushes every user translation, and a CPU entering a diff --git a/kernel/pure/vm.rs b/kernel/pure/vm.rs index 1d53aacdcd2..295d8eca11e 100644 --- a/kernel/pure/vm.rs +++ b/kernel/pure/vm.rs @@ -69,7 +69,7 @@ impl Object for alloc::sync::Arc { /// What backs a range. #[derive(Clone, Debug)] -pub enum Kind { +pub enum Kind { /// Memory of this space's own, its frames recorded in the leaves that map /// them. Uncommitted, it is a reservation: no page of it can be touched. Private { committed: bool }, @@ -79,17 +79,17 @@ pub enum Kind { /// copy of `source` from `offset` on as far as the range's first `len` /// bytes go, and zeros past them: taken at the page's first touch, as a /// program's segments are. - Copy { source: O, offset: u64, len: u64 }, + Copy { source: S, offset: u64, len: u64 }, } #[derive(Clone, Debug)] -pub struct Entry { +pub struct Entry { pub end: u64, pub prot: Prot, - pub kind: Kind, + pub kind: Kind, } -impl Entry { +impl Entry { /// Whether its pages are this space's own, committed: each owns a frame /// once touched, and a promise of one before. pub fn committed(&self) -> bool { @@ -135,17 +135,17 @@ impl Entry { } } -pub struct Inventory { +pub struct Inventory { floor: u64, top: u64, - entries: BTreeMap>, + entries: BTreeMap>, /// Free ranges by start, each to its end. free: BTreeMap, /// The same ranges by `(length, start)`. by_len: BTreeSet<(u64, u64)>, } -impl Inventory { +impl Inventory { /// Nothing held in `[floor, top)`, both on pages. pub fn new(floor: u64, top: u64) -> Self { assert!(floor.is_multiple_of(PAGE) && top.is_multiple_of(PAGE) && floor < top, "vm: {floor:#x}..{top:#x}"); @@ -177,14 +177,14 @@ impl Inventory { } /// The entry holding `at`, and where it starts. - pub fn find(&self, at: u64) -> Option<(u64, &Entry)> { + pub fn find(&self, at: u64) -> Option<(u64, &Entry)> { let (&start, entry) = self.entries.range(..=at).next_back()?; (at < entry.end).then_some((start, entry)) } /// Every entry `[start, end)` meets, in address order, each with where it /// starts: the first may start below `start` and the last end past `end`. - pub fn meeting(&self, start: u64, end: u64) -> impl Iterator)> { + pub fn meeting(&self, start: u64, end: u64) -> impl Iterator)> { let first = self.find(start).map(|(s, _)| s).unwrap_or(start); self.entries.range(first..end).map(|(&s, e)| (s, e)) } @@ -222,7 +222,7 @@ impl Inventory { } /// Hold `[start, entry.end)`, which is free. - pub fn insert(&mut self, start: u64, entry: Entry) { + pub fn insert(&mut self, start: u64, entry: Entry) { assert!(self.holds(start, entry.end), "vm: {start:#x}..{:#x} is not pages of this space", entry.end); let (free_start, free_end) = self.free.range(..=start).next_back().map(|(&s, &e)| (s, e)).expect("vm: inserted over held memory"); assert!(entry.end <= free_end, "vm: {start:#x}..{:#x} inserted over held memory", entry.end); @@ -240,11 +240,11 @@ impl Inventory { /// Take every entry out of `[start, end)`, cutting any that crosses its /// ends, and answer what was there, in address order. - pub fn remove(&mut self, start: u64, end: u64) -> Vec<(u64, Entry)> { + pub fn remove(&mut self, start: u64, end: u64) -> Vec<(u64, Entry)> { self.split_at(start); self.split_at(end); let starts: Vec = self.entries.range(start..end).map(|(&s, _)| s).collect(); - let removed: Vec<(u64, Entry)> = + let removed: Vec<(u64, Entry)> = starts.into_iter().map(|s| (s, self.entries.remove(&s).expect("vm: a start just listed"))).collect(); for (s, entry) in &removed { self.release(*s, entry.end); @@ -254,7 +254,7 @@ impl Inventory { /// Change every entry inside `[start, end)`, cutting any that crosses its /// ends, with `change`; neighbours that become one range are joined. - pub fn update(&mut self, start: u64, end: u64, mut change: impl FnMut(u64, &mut Entry)) { + pub fn update(&mut self, start: u64, end: u64, mut change: impl FnMut(u64, &mut Entry)) { self.split_at(start); self.split_at(end); for (&s, entry) in self.entries.range_mut(start..end) { @@ -264,7 +264,7 @@ impl Inventory { } /// Every entry, in address order. - pub fn entries(&self) -> impl Iterator)> { + pub fn entries(&self) -> impl Iterator)> { self.entries.iter().map(|(&s, e)| (s, e)) } diff --git a/kernel/src/file_backing.rs b/kernel/src/file_backing.rs index 44165e9b8fa..bf160bc68d4 100644 --- a/kernel/src/file_backing.rs +++ b/kernel/src/file_backing.rs @@ -1,5 +1,6 @@ use alloc::sync::Arc; +use crate::mm::space::CopySource; use crate::object::shm::SharedMemObject; use toyos_abi::syscall::SyscallError; @@ -25,12 +26,9 @@ pub trait FileBacking: Send + Sync { /// Total file size in bytes. fn file_size(&self) -> u64; - /// The memory object the file's bytes are, from its start, where they - /// are one: what a spawn copies a segment out of page by page as each is - /// first touched, rather than whole when the process is made. - fn object(&self) -> Option<&Arc> { - None - } + /// The file's bytes as a spawn's segments copy them, each page at its + /// first touch: `None` once the file is gone. + fn source(&self) -> Option>; } /// An executable in a shared memory object handed over by handle: one a @@ -78,7 +76,7 @@ impl FileBacking for SharedImage { self.size } - fn object(&self) -> Option<&Arc> { - Some(&self.object) + fn source(&self) -> Option> { + Some(self.object.clone()) } } diff --git a/kernel/src/file_cache.rs b/kernel/src/file_cache.rs index cc54a7f8718..005509e08a0 100644 --- a/kernel/src/file_cache.rs +++ b/kernel/src/file_cache.rs @@ -3,7 +3,10 @@ use alloc::collections::btree_map::Entry; use alloc::collections::BTreeMap; use alloc::sync::Arc; +use alloc::vec::Vec; + use crate::file_backing::{BlockError, FileBacking}; +use crate::mm::space::CopySource; use crate::sync::Lock; use crate::user_ptr::{ByteSource, UserBytesMut}; @@ -16,8 +19,12 @@ const PAGE_SIZE: usize = crate::mm::PAGE_SIZE as usize; /// this it would wrap and alias a low page — seek, write and truncate refuse it. pub const MAX_FILE_SIZE: u64 = (u32::MAX as u64 + 1) * PAGE_SIZE as u64; +/// A page's bytes, shared with each [`Snapshot`] taken while it was the +/// file's: a write to a shared page writes a copy of it. +type PageData = Arc>; + struct CachedPage { - data: Box<[u8; PAGE_SIZE]>, + data: PageData, /// CLOCK's second-chance bit: set on every hit, cleared when the sweep passes it over. referenced: bool, } @@ -278,7 +285,7 @@ fn apply_write( ) { let page = file.pages.get_mut(&page_idx).expect("write_page: page not resident"); let end = (offset + data.len()).min(PAGE_SIZE); - data.read_at(0, &mut page.data[offset..end]); + data.read_at(0, &mut Arc::make_mut(&mut page.data)[offset..end]); page.referenced = true; let write_end = page_idx as u64 * PAGE_SIZE as u64 + end as u64; @@ -293,10 +300,50 @@ pub fn copy_page_out(file_id: FileId, page_idx: u32, buf: &mut [u8; PAGE_SIZE]) let Some(page) = cache.files.get(&file_id).and_then(|file| file.pages.get(&page_idx)) else { return false; }; - *buf = *page.data; + *buf = **page.data; true } +/// A file's bytes as they were when it was taken, which no later write, +/// truncation or delete changes: a spawn's program, read without the cache's +/// lock. +pub struct Snapshot { + size: u64, + /// The resident pages by index; an absent one is a hole and reads zeros. + pages: Vec<(u32, PageData)>, +} + +/// `file_id`'s bytes now, or `None` for a file the cache no longer holds. Only +/// a file with no backing: an absent page of one with a backing was evicted, +/// not a hole. +pub fn snapshot(file_id: FileId) -> Option { + let cache = FILE_CACHE.lock(); + let file = cache.files.get(&file_id)?; + assert!(file.backing.is_none(), "file cache: a snapshot of file {file_id}, whose absent pages are on its backing"); + Some(Snapshot { size: file.size, pages: file.pages.iter().map(|(&idx, page)| (idx, Arc::clone(&page.data))).collect() }) +} + +impl CopySource for Snapshot { + fn size(&self) -> u64 { + self.size + } + + fn read(&self, offset: u64, out: &mut [u8]) { + assert!(offset.checked_add(out.len() as u64).is_some_and(|end| end <= self.size), "file cache: a snapshot read past its end"); + let mut done = 0; + while done < out.len() { + let at = offset + done as u64; + let (idx, within) = ((at / PAGE_SIZE as u64) as u32, (at % PAGE_SIZE as u64) as usize); + let n = (PAGE_SIZE - within).min(out.len() - done); + match self.pages.binary_search_by_key(&idx, |&(i, _)| i) { + Ok(i) => out[done..done + n].copy_from_slice(&self.pages[i].1[within..within + n]), + Err(_) => out[done..done + n].fill(0), + } + done += n; + } + } +} + /// Get the authoritative file size. pub fn size(file_id: FileId) -> u64 { FILE_CACHE.lock().files.get(&file_id).map_or(0, |f| f.size) @@ -327,7 +374,7 @@ fn set_size_locked(cache: &mut FileCache, file_id: FileId, new_size: u64) { if tail != 0 { let straddled = (new_size / PAGE_SIZE as u64) as u32; if let Some(page) = file.pages.get_mut(&straddled) { - page.data[tail..].fill(0); + Arc::make_mut(&mut page.data)[tail..].fill(0); } } if is_cache { removed.len() } else { 0 } @@ -352,7 +399,7 @@ pub fn mark_deleted(file_id: FileId) { impl CachedPage { fn new(data: Box<[u8; PAGE_SIZE]>) -> Self { - Self { data, referenced: false } + Self { data: Arc::new(data), referenced: false } } } diff --git a/kernel/src/loader/mod.rs b/kernel/src/loader/mod.rs index eb5170c2f67..8bcb3bb216c 100644 --- a/kernel/src/loader/mod.rs +++ b/kernel/src/loader/mod.rs @@ -30,6 +30,7 @@ use alloc::vec::Vec; use crate::elf; use crate::object::{ops, HandleTable, KObjectRef}; +use crate::mm::space::CopySource; use crate::mm::{PAGE_2M, PAGE_BYTES}; use crate::process::{ Admission, ElfInfo, Endowments, OwnedAlloc, Parent, Pid, ProcessAccounting, ProcessData, @@ -85,19 +86,17 @@ pub(crate) fn read_file_range( /// Commit one range of private memory per `PT_LOAD` segment at `image_start` -/// plus its offset, its file bytes and the rest of it zero: out of an image -/// object, each page copied at its first touch, as `file_backing::SharedImage` -/// promises a change after the hand-off reaches only pages not yet read; out -/// of any other backing, copied in whole now. +/// plus its offset, its file bytes and the rest of it zero, each page copied +/// out of `source` at its first touch. /// -/// `Err` when two segments would share a page, which their ranges cannot, or -/// when the file's bytes cannot be read: checked before the first range, and -/// a refusal after it leaves ranges in a space nothing runs in yet. +/// `Err` when two segments would share a page, which their ranges cannot: +/// checked before the first range, and a refusal after it leaves ranges in a +/// space nothing runs in yet. fn load_segments( space: &mut crate::mm::space::AddressSpace, layout: &Layout, image_start: UserAddr, - backing: &dyn crate::file_backing::FileBacking, + source: &Arc, ) -> Result<(), SyscallError> { use crate::mm::space::At; @@ -105,7 +104,6 @@ fn load_segments( log!("spawn: PT_LOAD segments {} and {} contend for a page", a, b); return Err(SyscallError::InvalidArgument); } - let mut page = [0u8; PAGE_BYTES]; for seg in layout.segments() { let (lo, hi) = seg.page_range(4096); let (seg_start, seg_end) = ((image_start + lo).raw(), (image_start + hi).raw()); @@ -114,27 +112,13 @@ fn load_segments( } // The file's bytes from the segment's first page: the bytes of that // page before the segment are the file's too, as a linker lays them. + // Past the file's end and the segment's file size, the range reads + // zeros. let file_start = seg.file_offset() & !4095; let file_bytes = seg.filesz() + seg.file_offset() % 4096; + let copied = file_bytes.min(source.size().saturating_sub(file_start)); let at = At::Fixed(UserAddr::new(seg_start)); - if let Some(object) = backing.object() { - // Each page copied out of the object at its first touch, the file - // past its end and the segment past its file size reading zeros. - let copied = file_bytes.min(backing.file_size().saturating_sub(file_start)); - space.copy(seg_end - seg_start, at, segment_prot(seg), Arc::clone(object), file_start, copied)?; - continue; - } - space.reserve(seg_end - seg_start, at, segment_prot(seg))?; - let mut done = 0; - while done < file_bytes { - if backing.read_page(file_start + done, &mut page).is_err() { - log!("spawn: the file's byte {} would not read", file_start + done); - return Err(SyscallError::NotFound); - } - let valid = (file_bytes - done).min(4096) as usize; - space.fill(seg_start + done, &page[..valid]); - done += 4096; - } + space.copy(seg_end - seg_start, at, segment_prot(seg), Arc::clone(source), file_start, copied)?; } Ok(()) } @@ -243,6 +227,13 @@ pub fn spawn( return Err(SyscallError::InvalidArgument.into()); }; + // Taken before the child's space is locked: a file's snapshot takes the + // file cache's lock, which a user copy may hold while it takes a space's. + let Some(source) = backing.source() else { + log!("spawn: {}: the file is gone", path); + return Err(SyscallError::NotFound.into()); + }; + let t1 = crate::clock::nanos_since_boot(); let Some(child_pt) = crate::mm::space::Space::new_user() else { @@ -270,7 +261,7 @@ pub fn spawn( path, extent.min(), layout.span()); return Err(SyscallError::InvalidArgument.into()); }; - load_segments(&mut space, &layout, image_start, backing.as_ref())?; + load_segments(&mut space, &layout, image_start, &source)?; // Never executable, and a reserved guard below it, so running off the // stack's bottom faults instead of writing what lies there. let stack = space.reserve(STACK_GUARD + USER_STACK_SIZE as u64, At::ANYWHERE, Prot::None)?; diff --git a/kernel/src/mm/space.rs b/kernel/src/mm/space.rs index a44941646b9..78b201cc835 100644 --- a/kernel/src/mm/space.rs +++ b/kernel/src/mm/space.rs @@ -27,7 +27,28 @@ use crate::object::shm::SharedMemObject; use crate::sync::{Lock, LockGuard}; use crate::UserAddr; -type Inventory = vm::Inventory>; +type Inventory = vm::Inventory, Arc>; +type SpaceEntry = Entry, Arc>; +type SpaceKind = Kind, Arc>; + +/// The bytes a copied range's pages start as. A read runs under the space +/// lock, from a fault or a user copy, so it may take no lock a user copy can +/// hold, and it cannot fail: what it reads was fixed when the range was made. +pub trait CopySource: Send + Sync { + fn size(&self) -> u64; + /// `out.len()` bytes from `offset`, all inside [`size`](Self::size). + fn read(&self, offset: u64, out: &mut [u8]); +} + +impl CopySource for SharedMemObject { + fn size(&self) -> u64 { + SharedMemObject::size(self) + } + + fn read(&self, offset: u64, out: &mut [u8]) { + SharedMemObject::read(self, offset, out) + } +} /// Nothing is placed below this, so a null pointer and the pages above it /// fault. @@ -263,12 +284,12 @@ impl AddressSpace { /// `len` bytes of private memory, committed, whose pages start as a copy /// of `source` from `offset` as far as their first `copied` bytes go and /// zeros past them, each taken at its first touch: a program's segment - /// read lazily out of the object it was handed in. - pub fn copy(&mut self, len: u64, at: At, prot: Prot, source: Arc, offset: u64, copied: u64) -> Result { + /// read lazily out of its file. + pub fn copy(&mut self, len: u64, at: At, prot: Prot, source: Arc, offset: u64, copied: u64) -> Result { self.private(len, at, prot, Kind::Copy { source, offset, len: copied }) } - fn private(&mut self, len: u64, at: At, prot: Prot, kind: Kind>) -> Result { + fn private(&mut self, len: u64, at: At, prot: Prot, kind: SpaceKind) -> Result { self.room_at(len, at)?; let start = self.locate(len, at)?; let backed = !matches!(kind, Kind::Private { committed: false }); @@ -570,7 +591,7 @@ impl AddressSpace { /// The leaf a fault puts at `page`, inside the range `entry` starting at /// `start`: a fresh frame of private memory, or the object's own page. -fn page_leaf(start: u64, entry: &Entry>, page: u64) -> Option { +fn page_leaf(start: u64, entry: &SpaceEntry, page: u64) -> Option { let prot = leaf_prot(entry.prot)?; Some(match &entry.kind { Kind::Private { committed: false } => return None, @@ -590,15 +611,15 @@ fn page_leaf(start: u64, entry: &Entry>, page: u64) -> Opti return None; }; let phys = frame.into_raw(); - copy_in(source, offset + (page - start), len.saturating_sub(page - start), phys); + copy_in(&**source, offset + (page - start), len.saturating_sub(page - start), phys); Leaf::Mapped(Mapping { phys, prot, owned: true, cache: CachePolicy::Normal }) } }) } /// Write `source`'s bytes from `offset` into the zeroed frame at `phys`, as -/// many of the page's as `len` and the object both hold. -fn copy_in(source: &SharedMemObject, offset: u64, len: u64, phys: u64) { +/// many of the page's as `len` and the source both hold. +fn copy_in(source: &dyn CopySource, offset: u64, len: u64, phys: u64) { let n = len.min(PAGE_SIZE).min(source.size().saturating_sub(offset)) as usize; if n == 0 { return; diff --git a/kernel/src/tmpfs.rs b/kernel/src/tmpfs.rs index c73c7c3cc93..599e83831e1 100644 --- a/kernel/src/tmpfs.rs +++ b/kernel/src/tmpfs.rs @@ -7,6 +7,7 @@ use core::sync::atomic::{AtomicBool, Ordering}; use crate::file_backing::FileBacking; use crate::file_cache::{self, FileId}; use crate::fs_rename::{self, Committed, ReplaceRename}; +use crate::mm::space::CopySource; use crate::mm::PAGE_BYTES; use toyos_abi::syscall::SyscallError; @@ -39,6 +40,13 @@ impl FileBacking for TmpfsBacking { fn file_size(&self) -> u64 { file_cache::size(self.file_id) } + + fn source(&self) -> Option> { + if !self.alive.load(Ordering::Acquire) { + return None; + } + Some(Arc::new(file_cache::snapshot(self.file_id)?)) + } } /// Ends a file entry's life: cleared first (so every backing fails), pages after. diff --git a/tests/guest/rust-tests/src/bin/spawn_from_tmp.rs b/tests/guest/rust-tests/src/bin/spawn_from_tmp.rs new file mode 100644 index 00000000000..5df01c3d2d8 --- /dev/null +++ b/tests/guest/rust-tests/src/bin/spawn_from_tmp.rs @@ -0,0 +1,43 @@ +//! A program spawned from a file in the kernel's `/tmp` runs the file's bytes +//! as they were at the spawn: the file overwritten with `hlt` and deleted the +//! moment the spawn returns, the child reads every page of its data after +//! that, at each page's first touch, and finds the bytes it was built with. + +use std::io::{Read, Write}; +use std::process::{Command, Stdio}; + +const SELF: &str = "/system/bin/test_rs_spawn_from_tmp"; +const COPY: &str = "/tmp/spawn_from_tmp"; +const CHILD: &str = "child"; + +/// Read-only data no page of which the child touches before it is told to. +const BYTE: u8 = 7; +static DATA: [u8; 256 * 1024] = [BYTE; 256 * 1024]; + +fn child() { + let mut go = [0u8]; + std::io::stdin().read_exact(&mut go).expect("the parent's go-ahead"); + let data = std::hint::black_box(&DATA); + // SAFETY: each index is inside `DATA`, a static that lives for the program. + let wrong = (0..data.len()).filter(|&i| unsafe { core::ptr::read_volatile(&data[i]) } != BYTE).count(); + if wrong != 0 { + eprintln!("spawn_from_tmp: {wrong} of {} data bytes are not the ones the program was built with", data.len()); + std::process::exit(1); + } +} + +fn main() { + if std::env::args().nth(1).as_deref() == Some(CHILD) { + return child(); + } + let bytes = std::fs::read(SELF).expect("read this program"); + std::fs::write(COPY, &bytes).expect("copy this program into /tmp"); + let mut child = Command::new(COPY).arg(CHILD).stdin(Stdio::piped()).spawn().expect("spawn the copy in /tmp"); + std::fs::write(COPY, vec![0xF4; bytes.len()]).expect("overwrite the copy with hlt"); + std::fs::remove_file(COPY).expect("delete the copy"); + child.stdin.take().expect("the child's stdin").write_all(&[1]).expect("tell the child to read its data"); + let status = child.wait().expect("wait for the child"); + assert!(status.success(), "the child spawned from /tmp, its file overwritten and deleted: {status}"); + println!("spawn_from_tmp: a child spawned from /tmp read its data as it was at the spawn"); + println!("spawn_from_tmp: PASS"); +} From 94838adbb10ac72ce6bd37bdcda8ba8af1430c4c Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 23 Oct 2026 02:53:47 +0200 Subject: [PATCH 07/10] mm: answer review round 4 of the lifecycle stage - spawn_from_tmp overwrites its child's file in place, then cuts it to nothing and deletes it. A create-and-truncate write replaced the file with a new one, so the test had only shown the child surviving its file's delete. The bytes written are named by what they are not, since 0xF4 is hlt only on x86. - A space counts the changes that reached another CPU only in the test kernel, which is the one reader of the count. - The black box's seal is one rule: every vector but a Ring 3 page fault. - A swapped binary's second window is closed: a /tmp spawn runs a snapshot. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...nary-lives-where-any-process-can-rewrite-it.md | 10 ++++++---- kernel/src/arch/x86_64/idt/mod.rs | 12 +++--------- kernel/src/mm/space.rs | 11 +++++++---- tests/guest/rust-tests/src/bin/spawn_from_tmp.rs | 15 ++++++++++----- 4 files changed, 26 insertions(+), 22 deletions(-) diff --git a/issues/a-swapped-binary-lives-where-any-process-can-rewrite-it.md b/issues/a-swapped-binary-lives-where-any-process-can-rewrite-it.md index 4f45faacd0b..4bd6004aa08 100644 --- a/issues/a-swapped-binary-lives-where-any-process-can-rewrite-it.md +++ b/issues/a-swapped-binary-lives-where-any-process-can-rewrite-it.md @@ -18,8 +18,10 @@ guarantee — only by nobody having rewritten it in between. Two windows: requester has hung up, milliseconds to two seconds later; a process that rewrites the file in that window runs its own bytes holding the service's manifest row — its device claims included. -- **For the service's life.** The kernel pages a binary in on demand, and a - restore after a failed swap spawns the previous installed path again. +- **For the service's life**, closed: a spawn from `/tmp` runs a snapshot of + the file taken at the spawn (`file_cache::Snapshot`), which no later write + reaches. A restore after a failed swap spawns the previous installed path + again, so it is the first window again. Nothing in the tree does either; the authority to ask for a swap is `/system/bin/swap`'s alone, and this is the one step of it the capability @@ -37,8 +39,8 @@ track would have to rule on. The supervisor now hashes an installed binary again at every start — the swap's and a restore's — and refuses to start one that no longer holds what its path names (`toyos_swap::installed_digest`). That moves the first window from "until the -requester hangs up" to the spawn syscall itself. The second is unchanged: the -kernel still pages the running binary in from the file. A directory only the supervisor +requester hangs up" to the spawn syscall itself, and the snapshot closed the +second. A directory only the supervisor may write does not exist in this kernel: every writable mount is ambient, and gating one on the writer's identity would be pid-as-authority. So the exit condition above stands, and choosing between its three shapes is the owner's. diff --git a/kernel/src/arch/x86_64/idt/mod.rs b/kernel/src/arch/x86_64/idt/mod.rs index 1bc00b3eb22..59557caaa9b 100644 --- a/kernel/src/arch/x86_64/idt/mod.rs +++ b/kernel/src/arch/x86_64/idt/mod.rs @@ -335,15 +335,9 @@ extern "sysv64" fn trap_dispatch(frame: *mut TrapFrame) { // is a double and then a triple fault, and a triple fault is a reset that // wrote nothing; the registers are what this entry can say without risking // that, and the report overwrites them if the machine gets that far. Every - // vector, the page fault included: a diagnostic that skips the commonest - // fault class is one that misses the answer, and a returning fault's record - // is simply overwritten by the next. - // - // **But not a Ring 3 page fault**, the one fault ordinary work takes on - // every first touch of every page: it is served or it ends its process - // through the crash path, which reports, and a fault inside its handler is - // a Ring 0 fault that records itself here. Sealing it would write back the - // whole box on every first touch, from every CPU at once. + // vector seals but a Ring 3 page fault, which every first touch of a page + // takes: it is served or ends its process through the crash path, which + // reports, and a fault inside its handler is a Ring 0 fault and seals. let user_page_fault = Vector::from_raw(frame.vector) == Vector::PageFault && toyos_userbound::Ring::of_cs(frame.cs).is_user(); if !user_page_fault { crate::blackbox::record_fault(&fault_of(frame)); diff --git a/kernel/src/mm/space.rs b/kernel/src/mm/space.rs index 78b201cc835..56cd7db07dc 100644 --- a/kernel/src/mm/space.rs +++ b/kernel/src/mm/space.rs @@ -85,12 +85,13 @@ pub struct Usage { /// Pages a fault populated, and the time faults took. faults: AtomicU64, fault_ns: AtomicU64, - /// Changes to the space's leaves that reached another CPU by interrupt. + /// Changes to the space's leaves that reached another CPU by interrupt, + /// for the test kernel's `SYS_DEBUG`. + #[cfg(feature = "test-actuators")] reached: AtomicU64, } impl Usage { - /// Read only by the test kernel's `SYS_DEBUG`. #[cfg(feature = "test-actuators")] pub fn reached(&self) -> u64 { self.reached.load(Relaxed) @@ -420,9 +421,11 @@ impl AddressSpace { } /// Every CPU has dropped what it cached of `[start, end)` and of the - /// tables `unlinked` took out, counting the changes that reached another. + /// tables `unlinked` took out. fn shoot_down(&self, start: u64, end: u64, unlinked: &Unlinked) { - if self.tables.shoot_down(start, end, unlinked) { + let _reached = self.tables.shoot_down(start, end, unlinked); + #[cfg(feature = "test-actuators")] + if _reached { self.usage.reached.fetch_add(1, Relaxed); } } diff --git a/tests/guest/rust-tests/src/bin/spawn_from_tmp.rs b/tests/guest/rust-tests/src/bin/spawn_from_tmp.rs index 5df01c3d2d8..1a520337a53 100644 --- a/tests/guest/rust-tests/src/bin/spawn_from_tmp.rs +++ b/tests/guest/rust-tests/src/bin/spawn_from_tmp.rs @@ -1,8 +1,10 @@ //! A program spawned from a file in the kernel's `/tmp` runs the file's bytes -//! as they were at the spawn: the file overwritten with `hlt` and deleted the -//! moment the spawn returns, the child reads every page of its data after -//! that, at each page's first touch, and finds the bytes it was built with. +//! as they were at the spawn: the file overwritten in place with bytes other +//! than its own, cut to nothing and deleted the moment the spawn returns, the +//! child reads every page of its data after that, at each page's first touch, +//! and finds the bytes it was built with. +use std::fs::OpenOptions; use std::io::{Read, Write}; use std::process::{Command, Stdio}; @@ -33,11 +35,14 @@ fn main() { let bytes = std::fs::read(SELF).expect("read this program"); std::fs::write(COPY, &bytes).expect("copy this program into /tmp"); let mut child = Command::new(COPY).arg(CHILD).stdin(Stdio::piped()).spawn().expect("spawn the copy in /tmp"); - std::fs::write(COPY, vec![0xF4; bytes.len()]).expect("overwrite the copy with hlt"); + let mut file = OpenOptions::new().write(true).open(COPY).expect("open the copy to overwrite it"); + file.write_all(&vec![0xF4; bytes.len()]).expect("overwrite the copy in place"); + file.set_len(0).expect("cut the copy to nothing"); + drop(file); std::fs::remove_file(COPY).expect("delete the copy"); child.stdin.take().expect("the child's stdin").write_all(&[1]).expect("tell the child to read its data"); let status = child.wait().expect("wait for the child"); - assert!(status.success(), "the child spawned from /tmp, its file overwritten and deleted: {status}"); + assert!(status.success(), "the child spawned from /tmp, its file overwritten, cut and deleted: {status}"); println!("spawn_from_tmp: a child spawned from /tmp read its data as it was at the spawn"); println!("spawn_from_tmp: PASS"); } From 361fe17815665f69fea8708ddead1984b412a0be Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 22 Oct 2026 11:15:49 +0200 Subject: [PATCH 08/10] mm: a commitment is promised its frames when it is made Strict commitment, as one count in the frame allocator rather than a ledger beside it. A commitment promises its frames (`Frames::promise`) when it is made: a committed `vm_reserve`, a `vm_protect` that commits reserved pages, and a shared object's frames at `SYS_SHM_CREATE`. A first touch draws one with `alloc_promised`, which cannot be refused. Every other allocation, the kernel heap's growth among them, is refused rather than take a promised frame, so the frames a commitment was promised are there when it touches them; and a promise must leave the kernel 1/256 of managed memory (16 MiB at least) that no commitment may take. `vm_discard` promises again the frames it frees, which stay committed; `vm_decommit`, `vm_release` and a space's end take back the promises of pages never touched. So a commitment past what memory holds is refused at the call, `ResourceExhausted`, where it used to succeed and end a thread at a later fault: `try_reserve(16 GB)` fails again (`allocator_stress`), and a 1 TiB reservation is placed while a 1 TiB commitment, made at once or by protecting the reservation, is refused (`mmap_stress`). A length no placement could hold is still `InvalidArgument`, checked first. Page tables are taken when the range is committed, and their allocation is fallible now: a commitment whose tables cannot be built is refused, with the tables it built for itself unlinked and freed. A user space's root is fallible too. `SYS_SYSINFO`'s used memory counts promised frames, which no new commitment can have. Not yet: the per-job account and `account_info`, and charging kernel objects a process makes, which still draw on the frames the kernel keeps uncharged (`issues/nothing-charges-kernel-memory-to-a-process.md`). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...b-pages-still-copies-each-programs-text.md | 17 +- kernel/pure/frames.rs | 164 ++++++++++++- kernel/src/arch/aarch64/paging.rs | 65 ++++- kernel/src/arch/x86_64/paging.rs | 58 ++++- kernel/src/mm/pmm.rs | 68 +++++- kernel/src/mm/space.rs | 231 ++++++++++++++---- kernel/src/object/shm.rs | 6 +- sdk/abi/src/syscall.rs | 2 + tests/guest/rust-tests/src/bin/mmap_stress.rs | 20 +- 9 files changed, 545 insertions(+), 86 deletions(-) diff --git a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md index 7061ecde0d0..935c92cade0 100644 --- a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md +++ b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md @@ -63,7 +63,8 @@ live split of a user leaf. 2. **Superframe frame allocator.** Built (`kernel::frames`): 4 KiB frames carved from 2 MiB superframes, pins per frame, a frame freed while pinned deferred to its last unpin, page tables in frames of their own, and every - 2 MiB caller of the PMM unchanged. + 2 MiB caller of the PMM unchanged. Promises, below, refuse a commitment + before the machine runs dry. 3. **4 KiB user leaves.** Built: one inventory per space (`kernel::vm`, host-tested) is the only record of its ranges, and a software owned bit in the leaf the only record of a private frame; stacks are populated as they @@ -79,14 +80,16 @@ live split of a user leaf. `vm_reserve`, `vm_protect`, `vm_discard`, `vm_decommit` and `vm_release` are the calls, behind libc's `mmap`, `mprotect`, `madvise` and `munmap` and std's allocator, which trims, and its thread stacks, each reserved - at the size asked with a guard page below. What - remains of the stage's exit — `mmap` past what memory can hold refused at - `mmap` — is the strict charge of the allocator track's migration - (`issues/toyos-has-its-own-allocator.md`). + at the size asked with a guard page below. A commitment is promised its + frames when it is made, and every other allocation is refused rather than + take a promised frame (`kernel::frames`), so a commitment past what memory + holds is refused at the call, not at a fault; its page tables are taken + then too. Kernel objects a process makes are still drawn from the frames + no commitment may promise, uncharged + (`issues/nothing-charges-kernel-memory-to-a-process.md`). 5. **Shared text.** A spawn copies each page of its program's segments into the process at the page's first touch, out of the image object or out of - a snapshot of its `/tmp` file; no fault reads a file. ROOT's read-only - text shared zero-copy, from + a snapshot of its `/tmp` file; no fault reads a file. ROOT's read-only text shared zero-copy, from immutable sealed objects, is the next step. Exit: two processes of one binary hold its text once. 6. **`/apps` image sharing.** The design is the owner's decision, owed when diff --git a/kernel/pure/frames.rs b/kernel/pure/frames.rs index c5eb85b4835..90efbb7c0e6 100644 --- a/kernel/pure/frames.rs +++ b/kernel/pure/frames.rs @@ -13,6 +13,13 @@ //! not take it back until its last pin goes, and [`Pins::unpin`] says when //! that is. //! +//! **A promise is a frame no other allocation may take.** A commitment of +//! memory that is populated later [`promise`](Frames::promise)s its frames +//! when it is made, and each is drawn with +//! [`alloc_promised`](Frames::alloc_promised) when it is first touched; every +//! other allocation is refused rather than take a frame promised. So a +//! commitment is refused when it is made, never at its first touch. +//! //! The storage is the caller's, sized by [`supers_for`] and [`pins_for`] for //! the span of physical memory it manages, because the kernel lays it out of //! that memory before it has a heap. @@ -87,6 +94,15 @@ enum Which { Partial, } +/// What [`Frames::discard`] did with a frame. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Discarded { + /// Freed, and its page promised a frame again. + Freed, + /// Pinned by a copy, so kept for its page. + Kept, +} + /// The allocator. Every method but the constructors takes the caller's lock. pub struct Frames<'m> { base: u64, @@ -95,6 +111,8 @@ pub struct Frames<'m> { partial: List, free: u64, usable: u64, + /// Frames promised and not yet drawn: never more than `free`. + promised: u64, } impl<'m> Frames<'m> { @@ -105,7 +123,7 @@ impl<'m> Frames<'m> { assert!(base.is_multiple_of(SUPER), "frames: base {base:#x} is not on a superframe"); assert!(supers.len() < NIL as usize, "frames: {} superframes do not fit the list links", supers.len()); supers.fill(Super::ABSENT); - Self { base, supers, whole: List::EMPTY, partial: List::EMPTY, free: 0, usable: 0 } + Self { base, supers, whole: List::EMPTY, partial: List::EMPTY, free: 0, usable: 0, promised: 0 } } /// One past the last byte the records cover. @@ -191,9 +209,77 @@ impl<'m> Frames<'m> { } } - /// One frame, from a split superframe if one has a frame free and from a - /// whole one otherwise. + /// Frames promised and not yet drawn. + pub fn promised(&self) -> u64 { + self.promised + } + + /// Promise `frames` if that leaves `keep` free frames unpromised. + pub fn promise(&mut self, frames: u64, keep: u64) -> bool { + let room = self.free.saturating_sub(self.promised).saturating_sub(keep); + if frames > room { + return false; + } + self.promised += frames; + true + } + + /// Free the frame at `phys` of a page that stays committed, promising + /// the page a frame again — or, where a user copy pins the frame, keep it + /// for the page: the caller zeroes it in place and leaves it in the + /// page's leaf, not present, for the next touch to map again. + /// + /// Sound only where no pin can be taken meanwhile: a copy pins a frame + /// under the lock of the space that maps it, which the caller holds, so a + /// frame read unpinned here stays unpinned. Then every promise is backed + /// by a frame that is free now, never by one deferred to a later unpin. + pub fn discard(&mut self, phys: u64, pins: &Pins) -> Discarded { + if pins.pinned(phys) { + return Discarded::Kept; + } + self.free_frame(phys, pins); + self.repromise(1); + Discarded::Freed + } + + /// Promise `frames` again, which a commitment still holds and whose + /// frames just came back free. + fn repromise(&mut self, frames: u64) { + self.promised += frames; + assert!(self.promised <= self.free, "frames: {} promised of {} free", self.promised, self.free); + } + + /// Take back promises of `frames` no commitment holds any more. + pub fn unpromise(&mut self, frames: u64) { + assert!(frames <= self.promised, "frames: {frames} unpromised of {} promised", self.promised); + self.promised -= frames; + } + + /// One frame a promise held, which a promise guarantees is free. + pub fn alloc_promised(&mut self) -> u64 { + assert!(self.promised > 0, "frames: a promised frame drawn and none was promised"); + let phys = self.take_frame().unwrap_or_else(|| panic!("frames: {} promised and none free", self.promised)); + self.promised -= 1; + phys + } + + /// One frame no promise holds, from a split superframe if one has a frame + /// free and from a whole one otherwise. pub fn alloc_frame(&mut self) -> Option { + self.alloc_frame_keeping(0) + } + + /// [`alloc_frame`](Self::alloc_frame), refused unless it leaves `keep` + /// frames neither promised nor taken: what a frame a process causes + /// without a promise, such as its page tables, may not take. + pub fn alloc_frame_keeping(&mut self, keep: u64) -> Option { + if self.free <= self.promised + keep { + return None; + } + self.take_frame() + } + + fn take_frame(&mut self) -> Option { let s = match self.partial.head { NIL => { let s = self.pop(Which::Whole)?; @@ -253,6 +339,9 @@ impl<'m> Frames<'m> { /// One whole superframe. pub fn alloc_super(&mut self) -> Option { + if self.free < self.promised + PER_SUPER as u64 { + return None; + } let s = self.pop(Which::Whole)?; self.supers[s].state = State::Taken; self.free -= PER_SUPER as u64; @@ -278,7 +367,7 @@ impl<'m> Frames<'m> { /// `count` superframes in a row, lowest address first; `None` for none. pub fn alloc_run(&mut self, count: usize) -> Option { - if count == 0 || count as u64 > u64::from(self.whole.len) { + if count == 0 || count as u64 > u64::from(self.whole.len) || self.free < self.promised + (count * PER_SUPER) as u64 { return None; } let mut run = 0; @@ -389,6 +478,11 @@ impl<'m> Pins<'m> { self.words.get(usize::try_from(frame).ok()?) } + /// Whether a copy holds the frame at `phys` pinned. + pub fn pinned(&self, phys: u64) -> bool { + self.word(phys).is_some_and(|word| word.load(Ordering::Acquire) & COUNT != 0) + } + /// Pin the frame holding `phys`, which is handed out; `false`, with /// nothing pinned, at the count's limit. Memory outside the records — a /// device's — has nothing to reissue and pins as `true`. @@ -491,6 +585,68 @@ mod tests { assert_eq!(m.frames.alloc_frame(), Some(first[7])); } + /// What is promised is refused to every other allocation and drawn by + /// its holder alone; a promise past what is free, less what must be kept, + /// is refused whole. + #[test] + fn a_promise_holds_its_frames_against_every_other_allocation() { + let mut m = whole(2); + let all = 2 * PER_SUPER as u64; + assert!(!m.frames.promise(all - 9, 10), "a promise into the kept frames"); + assert!(m.frames.promise(all - 10, 10)); + assert_eq!(m.frames.promised(), all - 10); + assert_eq!(m.frames.alloc_super(), None, "a superframe out of promised frames"); + assert_eq!(m.frames.alloc_run(1), None, "a run out of promised frames"); + let free: Vec = core::iter::from_fn(|| m.frames.alloc_frame()).collect(); + assert_eq!(free.len(), 10, "only the unpromised frames go to anyone else"); + assert!(!m.frames.promise(1, 0), "nothing is left to promise"); + + let drawn: Vec = (0..all - 10).map(|_| m.frames.alloc_promised()).collect(); + assert_eq!((m.frames.promised(), m.frames.free_frames()), (0, 0)); + + // A frame a commitment still holds, given back, is promised to it + // again; one given back by none is anyone's. + assert_eq!(m.frames.discard(drawn[0], &m.pins), Discarded::Freed); + assert_eq!((m.frames.promised(), m.frames.free_frames()), (1, 1)); + assert_eq!(m.frames.alloc_frame(), None, "a frame promised again went to another"); + m.frames.free_frame(drawn[1], &m.pins); + assert!(m.frames.alloc_frame().is_some(), "a frame no promise holds was refused"); + assert_eq!(m.frames.alloc_frame(), None, "the promised one went too"); + m.frames.unpromise(1); + assert!(m.frames.alloc_frame().is_some(), "an unpromised frame stayed held"); + } + + /// What a process's page tables may take leaves the kept frames to the + /// kernel, as a promise does. + #[test] + fn a_frame_kept_for_the_kernel_goes_to_the_kernel_alone() { + let mut m = whole(1); + let all = PER_SUPER as u64; + let keep = 10; + let taken: Vec = core::iter::from_fn(|| m.frames.alloc_frame_keeping(keep)).collect(); + assert_eq!(taken.len() as u64, all - keep, "a table took a kept frame"); + assert!(m.frames.alloc_frame().is_some(), "the kernel was refused a kept frame"); + } + + /// A frame a copy pins is kept by a discard rather than freed: freed, it + /// would come back only at its last unpin, and the promise made for it + /// meanwhile would be backed by no free frame, so a touch of the page, or + /// any other promised page, could find none. + #[test] + fn a_discard_keeps_a_pinned_frame_and_promises_only_what_came_back() { + let mut m = whole(1); + let all = PER_SUPER as u64; + assert!(m.frames.promise(all, 0)); + let drawn: Vec = (0..all).map(|_| m.frames.alloc_promised()).collect(); + assert!(m.pins.pin(drawn[0])); + assert_eq!(m.frames.discard(drawn[0], &m.pins), Discarded::Kept, "a pinned frame was freed"); + assert_eq!(m.frames.discard(drawn[1], &m.pins), Discarded::Freed); + assert_eq!((m.frames.promised(), m.frames.free_frames()), (1, 1)); + // The touch of the page whose frame went finds its frame. + assert_eq!(m.frames.alloc_promised(), drawn[1]); + assert_eq!(m.pins.unpin(drawn[0]), Unpinned::Kept, "a kept frame was deferred"); + } + #[test] fn exhaustion_answers_none_and_a_return_answers_again() { let mut m = whole(1); diff --git a/kernel/src/arch/aarch64/paging.rs b/kernel/src/arch/aarch64/paging.rs index a4f854ab707..ce157e6fc7a 100644 --- a/kernel/src/arch/aarch64/paging.rs +++ b/kernel/src/arch/aarch64/paging.rs @@ -142,6 +142,15 @@ impl Table { unsafe { Table::at_mut(frame.into_raw()) } } + /// A table of a user space, or `None` where taking a frame would cut into + /// what the kernel keeps (`pmm::alloc_table_frame`). + fn try_new() -> Option<&'static mut Self> { + let frame = crate::mm::pmm::alloc_table_frame()?; + // SAFETY: a zeroed frame just allocated: a table of invalid + // descriptors nothing else names. + Some(unsafe { Table::at_mut(frame.into_raw()) }) + } + fn phys(&self) -> u64 { DirectMap::phys_of(self) } @@ -521,10 +530,15 @@ pub struct UserTables { } impl UserTables { - /// An empty user half, or `None` when every user ASID is held. + /// An empty user half, or `None` when every user ASID is held or no + /// frame is free for its root. pub fn new() -> Option { - let asid = alloc_asid()?; - Some(Self { tables: Tables::new(), asid: Asid::User(asid) }) + let root = Table::try_new()?.phys(); + let Some(asid) = alloc_asid() else { + free_table(root); + return None; + }; + Some(Self { tables: Tables { root }, asid: Asid::User(asid) }) } /// The kernel's own space, whose user half is empty: what a kernel thread @@ -545,8 +559,10 @@ impl UserTables { } /// Build every table `[start, end)`'s leaves need; answers how many - /// frames that took. Only an invalid entry is written, which owes nothing. - pub fn build(&mut self, start: u64, end: u64) -> u64 { + /// frames that took, or `None` where a frame ran out, leaving what it + /// built linked for an [`unlink`](Self::unlink). Only an invalid entry is + /// written, which owes nothing. + pub fn build(&mut self, start: u64, end: u64) -> Option { let mut built = 0; let mut va = start & !(PAGE_2M - 1); while va < end { @@ -556,7 +572,11 @@ impl UserTables { let table = unsafe { Table::at_mut(level) }; let i = index(va, depth); if table.0[i] & VALID == 0 { - table.set(i, Table::new().phys() | TABLE | VALID); + let Some(child) = Table::try_new() else { + published(); + return None; + }; + table.set(i, child.phys() | TABLE | VALID); built += 1; } level = table.0[i] & ADDR; @@ -564,7 +584,7 @@ impl UserTables { va += PAGE_2M; } published(); - built + Some(built) } /// The leaf over `va`, empty where no table holds one. @@ -603,6 +623,37 @@ impl UserTables { published(); } + /// [`take_owned`](Self::take_owned), where `decide` says of each frame of + /// a table's batch, with the address its leaf maps, whether the leaf + /// keeps it: a kept frame's leaf stays, every other is emptied. + pub fn discard_owned(&mut self, start: u64, end: u64, mut decide: impl FnMut(&[u64], &[u64], &mut [bool])) { + self.each_table(start, end, |base, table| { + let mut frames = [0u64; 512]; + let mut vas = [0u64; 512]; + let mut slot = [0usize; 512]; + let mut n = 0; + for i in slots(start, end, base, 12) { + if let Leaf::Held(phys) | Leaf::Mapped(Mapping { phys, owned: true, .. }) = Leaf::decode(table.0[i]) { + frames[n] = phys; + vas[n] = base + ((i as u64) << 12); + slot[n] = i; + n += 1; + } + } + if n == 0 { + return; + } + let mut kept = [false; 512]; + decide(&frames[..n], &vas[..n], &mut kept[..n]); + for k in 0..n { + if !kept[k] { + table.set(slot[k], Leaf::Empty.encode()); + } + } + }); + published(); + } + /// Empty every leaf of `[start, end)` holding a frame the space owns, one /// table of leaves at a time, handing each table's frames to `taken`: at /// most 512 at once, in a buffer of its own, so nothing is allocated. diff --git a/kernel/src/arch/x86_64/paging.rs b/kernel/src/arch/x86_64/paging.rs index 3bcd9001470..dea8371a38d 100644 --- a/kernel/src/arch/x86_64/paging.rs +++ b/kernel/src/arch/x86_64/paging.rs @@ -148,6 +148,15 @@ impl PageTablePage { unsafe { Self::from_phys_mut(frame.into_raw()) } } + /// A table of a user space, or `None` where taking a frame would cut into + /// what the kernel keeps (`pmm::alloc_table_frame`). + fn try_new() -> Option<&'static mut PageTablePage> { + let frame = crate::mm::pmm::alloc_table_frame()?; + // SAFETY: a zeroed frame just allocated, so a table of empty entries + // nothing else names. + Some(unsafe { Self::from_phys_mut(frame.into_raw()) }) + } + fn phys(&self) -> u64 { crate::mm::DirectMap::phys_of(self) } @@ -762,11 +771,14 @@ pub struct UserTables { impl UserTables { /// A new space with the kernel half copied in, or `None` when every user - /// PCID is held by a live space. + /// PCID is held by a live space or no frame is free for its root. pub fn new() -> Option { - let pcid = alloc_pcid()?; + let pml4 = PageTablePage::try_new()?; + let Some(pcid) = alloc_pcid() else { + free_table(pml4.phys()); + return None; + }; let kernel = KERNEL.lock(); - let pml4 = PageTablePage::new(); for slot in ROOT_HIGH_HALF..512 { let entry = kernel.root_table()[slot]; assert!( @@ -809,8 +821,10 @@ impl UserTables { } /// Build every table `[start, end)`'s leaves need; answers how many - /// frames that took. Only an empty entry is written, which owes nothing. - pub fn build(&mut self, start: u64, end: u64) -> u64 { + /// frames that took, or `None` where a frame ran out, leaving what it + /// built linked for an [`unlink`](Self::unlink). Only an empty entry is + /// written, which owes nothing. + pub fn build(&mut self, start: u64, end: u64) -> Option { let mut built = 0; let mut va = start & !(PAGE_2M - 1); while va < end { @@ -818,7 +832,7 @@ impl UserTables { let mut level = self.root_table_mut(); for index in [a, b, c] { if level[index] & PAGE_PRESENT == 0 { - let child = PageTablePage::new(); + let child = PageTablePage::try_new()?; level.write(index, va, child.phys() | TABLE_FLAGS).expect_install("build"); built += 1; } @@ -827,7 +841,7 @@ impl UserTables { } va += PAGE_2M; } - built + Some(built) } /// The leaf over `va`, empty where no table holds one. @@ -859,6 +873,36 @@ impl UserTables { }); } + /// [`take_owned`](Self::take_owned), where `decide` says of each frame of + /// a table's batch, with the address its leaf maps, whether the leaf + /// keeps it: a kept frame's leaf stays, every other is emptied. + pub fn discard_owned(&mut self, start: u64, end: u64, mut decide: impl FnMut(&[u64], &[u64], &mut [bool])) { + self.each_table(start, end, |base, table| { + let mut frames = [0u64; 512]; + let mut vas = [0u64; 512]; + let mut slot = [0usize; 512]; + let mut n = 0; + for i in slots(start, end, base, 12) { + if let Leaf::Held(phys) | Leaf::Mapped(Mapping { phys, owned: true, .. }) = Leaf::decode(table.0[i]) { + frames[n] = phys; + vas[n] = base + ((i as u64) << 12); + slot[n] = i; + n += 1; + } + } + if n == 0 { + return; + } + let mut kept = [false; 512]; + decide(&frames[..n], &vas[..n], &mut kept[..n]); + for k in 0..n { + if !kept[k] { + table.store(slot[k], Leaf::Empty.encode()); + } + } + }); + } + /// Empty every leaf of `[start, end)` holding a frame the space owns, one /// table of leaves at a time, handing each table's frames to `taken`: at /// most 512 at once, in a buffer of its own, so nothing is allocated. diff --git a/kernel/src/mm/pmm.rs b/kernel/src/mm/pmm.rs index a44587d7e82..de9c7e83983 100644 --- a/kernel/src/mm/pmm.rs +++ b/kernel/src/mm/pmm.rs @@ -8,6 +8,7 @@ use core::sync::atomic::{AtomicPtr, AtomicU32, AtomicU64, AtomicUsize, Ordering}; use kernel::frames::{self, Frames, Pins, Super, Unpinned, FRAME, SUPER}; +pub use kernel::frames::Discarded; use super::{DirectMap, PAGE_2M}; use crate::sync::Lock; @@ -205,6 +206,63 @@ fn place(entries: &[MemoryMapEntry], reserved: &[Region], bytes: u64) -> Option< entries.iter().filter(|e| is_usable(e)).filter_map(lowest_in).min() } +/// The free frames no commitment may promise, so that the kernel's own +/// allocations meet room when commitments have taken the rest: 1/256 of +/// managed memory, and 16 MiB at least. +fn kept(frames: &Frames) -> u64 { + (frames.usable_frames() / 256).max((16 << 20) / FRAME) +} + +/// Promise `frames` to a commitment, which draws each with +/// [`alloc_promised`] when it is first touched; `false`, and nothing +/// promised, where that would leave less than the kernel keeps. +pub fn promise(frames: u64) -> bool { + FRAMES.lock().as_mut().is_some_and(|f| { + let keep = kept(f); + f.promise(frames, keep) + }) +} + +/// Take back promises no commitment holds any more. +pub fn unpromise(frames: u64) { + if frames > 0 { + FRAMES.lock().as_mut().expect("pmm: a promise outlived no allocator").unpromise(frames); + } +} + +/// [`Frames::discard`] of `phys`, a committed page's frame the caller holds +/// the lock of the mapping space for. +/// +/// # Safety +/// `phys` is an owned leaf's frame, which no leaf on any CPU maps any more, +/// unless this answers [`Discarded::Kept`]. +pub unsafe fn discard(phys: u64) -> Discarded { + FRAMES.lock().as_mut().expect("pmm: a frame outlived no allocator").discard(phys, &pins()) +} + +/// A zeroed frame a promise held, which the promise guarantees is free. +pub fn alloc_promised() -> Frame { + let phys = FRAMES.lock().as_mut().expect("pmm: a promise outlived no allocator").alloc_promised(); + let frame = Frame { phys }; + // SAFETY: just taken off the allocator, so unaliased, and inside the direct map. + unsafe { core::ptr::write_bytes(frame.direct_map().as_mut_ptr::(), 0, FRAME as usize) }; + frame +} + +/// A zeroed frame for a page table a process causes, refused unless it +/// leaves the frames the kernel keeps: a process's tables are not promised, +/// so this is what keeps them out of the kernel's room. +pub fn alloc_table_frame() -> Option { + let phys = FRAMES.lock().as_mut().and_then(|f| { + let keep = kept(f); + f.alloc_frame_keeping(keep) + })?; + let frame = Frame { phys }; + // SAFETY: just taken off the allocator, so unaliased, and inside the direct map. + unsafe { core::ptr::write_bytes(frame.direct_map().as_mut_ptr::(), 0, FRAME as usize) }; + Some(frame) +} + /// Allocate one 2MB physical page. pub fn alloc_page() -> Option { let page = claim()?; @@ -284,16 +342,12 @@ pub fn top() -> u64 { FRAMES.lock().as_ref().map_or(0, Frames::top) } -/// Frames free now: an answer that may be stale by the time it is read. -pub fn free_frames() -> u64 { - FRAMES.lock().as_ref().map_or(0, Frames::free_frames) -} - -/// Return (total_bytes, used_bytes). +/// Managed bytes, and those in use or promised. pub fn stats() -> (u64, u64) { let guard = FRAMES.lock(); let Some(frames) = guard.as_ref() else { return (0, 0) }; - (frames.usable_frames() * FRAME, (frames.usable_frames() - frames.free_frames()) * FRAME) + let used = frames.usable_frames() - frames.free_frames() + frames.promised(); + (frames.usable_frames() * FRAME, used * FRAME) } fn is_usable(entry: &MemoryMapEntry) -> bool { diff --git a/kernel/src/mm/space.rs b/kernel/src/mm/space.rs index 56cd7db07dc..56aaad6bd25 100644 --- a/kernel/src/mm/space.rs +++ b/kernel/src/mm/space.rs @@ -231,16 +231,16 @@ impl AddressSpace { self.tables.root() } - /// Where a range of `len` bytes goes; a `Fixed` one takes away whatever - /// was there first. `len` is whole pages. - fn locate(&mut self, len: u64, at: At) -> Result { + /// Where a range of `len` bytes goes, changing nothing: a `Fixed` one is + /// put in place of what is there by [`put`](Self::put). `len` is whole + /// pages. + fn place(&self, len: u64, at: At) -> Result { match at { At::Fixed(start) => { let end = pages(start.raw(), len).ok_or(SyscallError::InvalidArgument)?; if !self.inventory.holds(start.raw(), end) { return Err(SyscallError::InvalidArgument); } - self.release(start, len); Ok(start.raw()) } At::Anywhere(align) => { @@ -255,6 +255,30 @@ impl AddressSpace { } } + /// Hold `entry` from `start`, in place of whatever those pages held: the + /// new range is in the inventory before the old leaves are cleared, so the + /// clear keeps the tables it needs. It cannot be refused. + fn put(&mut self, start: u64, entry: SpaceEntry) { + let end = entry.end; + if self.inventory.is_free(start, end) { + self.inventory.insert(start, entry); + } else { + let removed = self.inventory.remove(start, end); + let committed: u64 = removed + .iter() + .filter(|(_, old)| old.committed()) + .map(|(s, old)| (old.end - s) / PAGE_SIZE) + .sum(); + self.inventory.insert(start, entry); + let freed = self.clear(start, end); + pmm::unpromise(committed - freed); + self.usage.taken.fetch_add(1, Relaxed); + // The objects go here, after every CPU has stopped reaching them. + drop(removed); + } + self.usage.made.fetch_add(1, Relaxed); + } + /// `ResourceExhausted` unless the space may still hold `added` ranges /// more and those a change over `[start, end)` adds by cutting a range at /// either end. The bound keeps what a space's ranges take of the kernel @@ -277,7 +301,11 @@ impl AddressSpace { } /// `len` bytes of private memory, committed unless `prot` is `None`, which - /// reserves the range and nothing more. + /// reserves the range and nothing more. A commitment is promised its + /// frames here, so it is refused here, `ResourceExhausted`, and never at a + /// later touch. All or nothing: a refused `Fixed` range leaves what the + /// pages held; one that is not refused needs its frames promised beside + /// those of what it replaces. pub fn reserve(&mut self, len: u64, at: At, prot: Prot) -> Result { self.private(len, at, prot, Kind::Private { committed: prot != Prot::None }) } @@ -285,35 +313,78 @@ impl AddressSpace { /// `len` bytes of private memory, committed, whose pages start as a copy /// of `source` from `offset` as far as their first `copied` bytes go and /// zeros past them, each taken at its first touch: a program's segment - /// read lazily out of its file. + /// read lazily out of its file. Promised and all or nothing, as + /// [`reserve`](Self::reserve). pub fn copy(&mut self, len: u64, at: At, prot: Prot, source: Arc, offset: u64, copied: u64) -> Result { self.private(len, at, prot, Kind::Copy { source, offset, len: copied }) } fn private(&mut self, len: u64, at: At, prot: Prot, kind: SpaceKind) -> Result { + // A length no placement could hold is the caller's mistake, whatever + // memory is left to promise. + if len > self.inventory.top() - self.inventory.floor() { + return Err(SyscallError::InvalidArgument); + } self.room_at(len, at)?; - let start = self.locate(len, at)?; - let backed = !matches!(kind, Kind::Private { committed: false }); - self.inventory.insert(start, Entry { end: start + len, prot, kind }); - if backed { - self.tables.build(start, start + len); + let start = self.place(len, at)?; + let entry = Entry { end: start + len, prot, kind }; + let promised = if entry.committed() { len / PAGE_SIZE } else { 0 }; + if !pmm::promise(promised) { + return Err(SyscallError::ResourceExhausted); } - self.usage.made.fetch_add(1, Relaxed); + if entry.backed() { + self.build(start, start + len).inspect_err(|_| pmm::unpromise(promised))?; + } + self.put(start, entry); Ok(UserAddr::new(start)) } + /// Build the tables `[start, end)` needs, or none of those it needs alone: + /// `ResourceExhausted` where taking a frame would cut into what the + /// kernel keeps. + fn build(&mut self, start: u64, end: u64) -> Result<(), SyscallError> { + if self.tables.build(start, end).is_some() { + return Ok(()); + } + let inventory = &self.inventory; + let unlinked = self.tables.unlink(start, end, |s, e| inventory.backed(s, e)); + if !unlinked.is_empty() { + self.tables.shoot_down(start, end, &unlinked); + } + unlinked.free(); + Err(SyscallError::ResourceExhausted) + } + + /// Committed private pages of `[start, end)`. + fn committed_pages(&self, start: u64, end: u64) -> u64 { + self.inventory + .meeting(start, end) + .filter(|(_, entry)| entry.committed()) + .map(|(s, entry)| (entry.end.min(end) - s.max(start)) / PAGE_SIZE) + .sum() + } + + /// Pages of `[start, end)` whose range's kind `which` answers `true` for. + fn pages_where(&self, start: u64, end: u64, which: impl Fn(&SpaceKind) -> bool) -> u64 { + self.inventory + .meeting(start, end) + .filter(|(_, entry)| which(&entry.kind)) + .map(|(s, entry)| (entry.end.min(end) - s.max(start)) / PAGE_SIZE) + .sum() + } + /// `len` bytes of `object` from `offset`, as `prot` allows; the caller - /// bounds `prot` by what its handle grants. + /// bounds `prot` by what its handle grants. All or nothing, as + /// [`reserve`](Self::reserve). pub fn map(&mut self, object: Arc, offset: u64, len: u64, prot: Prot, at: At) -> Result { let within = pages(offset, len).is_some_and(|end| end <= object.size()); if !within { return Err(SyscallError::InvalidArgument); } self.room_at(len, at)?; - let start = self.locate(len, at)?; - self.inventory.insert(start, Entry { end: start + len, prot, kind: Kind::Object { object, offset, max: prot } }); - self.tables.build(start, start + len); - self.usage.made.fetch_add(1, Relaxed); + let start = self.place(len, at)?; + self.build(start, start + len)?; + self.put(start, Entry { end: start + len, prot, kind: Kind::Object { object, offset, max: prot } }); Ok(UserAddr::new(start)) } @@ -329,19 +400,26 @@ impl AddressSpace { if removed.is_empty() { return; } - self.clear(start, end); + let committed: u64 = removed + .iter() + .filter(|(_, entry)| entry.committed()) + .map(|(s, entry)| (entry.end - s) / PAGE_SIZE) + .sum(); + let freed = self.clear(start, end); + pmm::unpromise(committed - freed); self.usage.taken.fetch_add(1, Relaxed); // The objects go here, after every CPU has stopped reaching them. drop(removed); } /// Give every page of `[start, start + len)` protection `prot`: a reserved - /// private page is committed unless `prot` is `None`, and a committed one - /// keeps its frame and commitment whatever `prot` is. An object's page - /// takes nothing past what it was mapped with. All or nothing: every page - /// is held (`BadAddress`) and allowed `prot` (`PermissionDenied`) first. - /// Every CPU stops using a present leaf's old protection before this - /// returns, a widening included. + /// private page is committed unless `prot` is `None`, promised its frame + /// here, and a committed one keeps its frame and commitment whatever + /// `prot` is. An object's page takes nothing past what it was mapped + /// with. All or nothing: every page is held (`BadAddress`), allowed `prot` + /// (`PermissionDenied`) and promised (`ResourceExhausted`) first. Every + /// CPU stops using a present leaf's old protection before this returns, a + /// widening included. pub fn protect(&mut self, start: UserAddr, len: u64, prot: Prot) -> Result<(), SyscallError> { let (start, end) = self.held(start, len)?; self.room(start, end, 0)?; @@ -352,19 +430,22 @@ impl AddressSpace { if !allowed { return Err(SyscallError::PermissionDenied); } - let mut commits = false; + let commits = match prot { + Prot::None => 0, + _ => self.pages_where(start, end, |kind| matches!(kind, Kind::Private { committed: false })), + }; + if !pmm::promise(commits) { + return Err(SyscallError::ResourceExhausted); + } + if commits > 0 { + self.build(start, end).inspect_err(|_| pmm::unpromise(commits))?; + } self.inventory.update(start, end, |_, entry| { entry.prot = prot; if let Kind::Private { committed } = &mut entry.kind { - if !*committed && prot != Prot::None { - *committed = true; - commits = true; - } + *committed |= prot != Prot::None; } }); - if commits { - self.tables.build(start, end); - } let leaf = leaf_prot(prot); let mut present = false; self.tables.edit(start, end, |_, old| match (old, leaf) { @@ -387,11 +468,62 @@ impl AddressSpace { } /// Free every frame behind `[start, start + len)` and keep its ranges as - /// they are: a private page reads zeros at its next touch, an object's - /// page is mapped again from the object. + /// they are: a private page reads zeros at its next touch, a copied one + /// its source again, an object's page is mapped again from the object. + /// + /// A private page's frame is freed and the page promised a frame again; + /// one a user copy pins is zeroed in place instead and kept, since freed + /// it would come back only at its last unpin and the promise would be + /// backed by nothing meanwhile (`Frames::discard`). No pin is taken + /// meanwhile: a copy pins under this space's lock, which this holds. pub fn discard(&mut self, start: UserAddr, len: u64) -> Result<(), SyscallError> { let (start, end) = self.held(start, len)?; - self.clear(start, end); + let mut present = false; + self.tables.edit(start, end, |_, leaf| match leaf { + Leaf::Mapped(Mapping { phys, owned: true, .. }) => { + present = true; + Some(Leaf::Held(phys)) + } + Leaf::Mapped(_) => { + present = true; + Some(Leaf::Empty) + } + Leaf::Held(_) | Leaf::Empty => None, + }); + if present { + self.tables.shoot_down(start, end, &Unlinked::none()); + } + let mut freed = 0; + let inventory = &self.inventory; + self.tables.discard_owned(start, end, |frames, vas, kept| { + let mut sorted = [0u64; 512]; + let sorted = &mut sorted[..frames.len()]; + sorted.copy_from_slice(frames); + sorted.sort_unstable(); + crate::sched::futex::revoke_frames(sorted); + for ((&phys, &va), kept) in frames.iter().zip(vas).zip(kept.iter_mut()) { + // SAFETY: a frame an owned leaf held, which no leaf on any CPU + // maps now and no waiter is keyed on; one kept is the page's. + *kept = match unsafe { pmm::discard(phys) } { + pmm::Discarded::Freed => { + freed += 1; + false + } + pmm::Discarded::Kept => { + // SAFETY: the page's own frame, which a copy may be + // reading or writing as the page's contents; zeroing + // it is what the discard makes those contents. + unsafe { core::ptr::write_bytes(super::DirectMap::from_phys(phys).as_mut_ptr::(), 0, PAGE_SIZE as usize) }; + // A copied page's contents start as its source's again. + if let Some((s, Entry { kind: Kind::Copy { source, offset, len }, .. })) = inventory.find(va) { + copy_in(&**source, offset + (va - s), len.saturating_sub(va - s), phys); + } + true + } + }; + } + }); + self.usage.freed(freed); Ok(()) } @@ -403,11 +535,13 @@ impl AddressSpace { if !self.inventory.meeting(start, end).all(|(_, entry)| matches!(entry.kind, Kind::Private { .. } | Kind::Copy { .. })) { return Err(SyscallError::InvalidArgument); } + let committed = self.committed_pages(start, end); self.inventory.update(start, end, |_, entry| { entry.prot = Prot::None; entry.kind = Kind::Private { committed: false }; }); - self.clear(start, end); + let freed = self.clear(start, end); + pmm::unpromise(committed - freed); Ok(()) } @@ -514,7 +648,13 @@ impl AddressSpace { let page = va & !(PAGE_SIZE - 1); let phys = match self.tables.get(page) { Leaf::Mapped(m) => m.phys, - Leaf::Held(_) => return None, + // A frame the page keeps while no leaf maps it: a discard that + // zeroed a pinned frame in place. + Leaf::Held(phys) => { + let prot = leaf_prot(entry.prot)?; + self.tables.set(page, Leaf::Mapped(Mapping { phys, prot, owned: true, cache: CachePolicy::Normal })); + phys + } Leaf::Empty => { let leaf = page_leaf(start, entry, page)?; let Leaf::Mapped(m) = leaf else { return None }; @@ -577,8 +717,7 @@ impl AddressSpace { Leaf::Mapped(m) => m.phys, Leaf::Held(phys) => phys, Leaf::Empty => { - let frame = super::pmm::alloc_frame().unwrap_or_else(|| panic!("space: no frame for {va:#x}")); - let phys = frame.into_raw(); + let phys = pmm::alloc_promised().into_raw(); let leaf = match leaf_prot(entry.prot) { Some(prot) => Leaf::Mapped(Mapping { phys, prot, owned: true, cache: CachePolicy::Normal }), None => Leaf::Held(phys), @@ -599,21 +738,13 @@ fn page_leaf(start: u64, entry: &SpaceEntry, page: u64) -> Option { Some(match &entry.kind { Kind::Private { committed: false } => return None, Kind::Private { committed: true } => { - let Some(frame) = super::pmm::alloc_frame() else { - crate::log!("space: no frame to populate {page:#x}"); - return None; - }; - Leaf::Mapped(Mapping { phys: frame.into_raw(), prot, owned: true, cache: CachePolicy::Normal }) + Leaf::Mapped(Mapping { phys: pmm::alloc_promised().into_raw(), prot, owned: true, cache: CachePolicy::Normal }) } Kind::Object { object, offset, .. } => { Leaf::Mapped(Mapping { phys: object.page(offset + (page - start)), prot, owned: false, cache: object.cache() }) } Kind::Copy { source, offset, len } => { - let Some(frame) = super::pmm::alloc_frame() else { - crate::log!("space: no frame to populate {page:#x}"); - return None; - }; - let phys = frame.into_raw(); + let phys = pmm::alloc_promised().into_raw(); copy_in(&**source, offset + (page - start), len.saturating_sub(page - start), phys); Leaf::Mapped(Mapping { phys, prot, owned: true, cache: CachePolicy::Normal }) } @@ -641,6 +772,7 @@ fn copy_in(source: &dyn CopySource, offset: u64, len: u64, phys: u64) { impl Drop for AddressSpace { fn drop(&mut self) { assert!(!self.kernel, "space: the kernel's dropped"); + let committed = self.committed_pages(FLOOR, TOP); self.tables.edit(FLOOR, TOP, |_, leaf| matches!(leaf, Leaf::Mapped(Mapping { owned: false, .. })).then_some(Leaf::Empty)); let mut freed = 0; self.tables.take_owned(FLOOR, TOP, |frames| { @@ -649,5 +781,6 @@ impl Drop for AddressSpace { freed += frames.len() as u64; }); self.usage.freed(freed); + pmm::unpromise(committed - freed); } } diff --git a/kernel/src/object/shm.rs b/kernel/src/object/shm.rs index 977b9ef5f21..35ac6a1288e 100644 --- a/kernel/src/object/shm.rs +++ b/kernel/src/object/shm.rs @@ -67,9 +67,9 @@ impl SharedMemObject { /// `ResourceExhausted` where frames run out. pub fn create(size: u64) -> Result, SyscallError> { let pages = pages(size)?; - // Refused whole before any is taken: frames one at a time until none + // Promised whole before any is taken: frames one at a time until none // is left would starve every other allocation meanwhile. - if pages > pmm::free_frames() { + if !pmm::promise(pages) { return Err(SyscallError::ResourceExhausted); } let mut pieces = Vec::new(); @@ -78,7 +78,7 @@ impl SharedMemObject { let n = left.min(CHUNK); let mut piece = Vec::with_capacity(n); for _ in 0..n { - piece.push(pmm::alloc_frame().ok_or(SyscallError::ResourceExhausted)?); + piece.push(pmm::alloc_promised()); } pieces.push(piece.into_boxed_slice()); left -= n; diff --git a/sdk/abi/src/syscall.rs b/sdk/abi/src/syscall.rs index a3816ab6f03..16e6e0c5327 100644 --- a/sdk/abi/src/syscall.rs +++ b/sdk/abi/src/syscall.rs @@ -48,6 +48,8 @@ pub const SYSINFO_HEADER_SIZE: usize = 48; #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub struct SysinfoHeader { pub memory_total: u64, + /// Bytes in use or promised to commitments not yet touched: what no + /// new commitment can have. pub memory_used: u64, pub cpus: u32, /// Live threads in the roster this same call would answer with diff --git a/tests/guest/rust-tests/src/bin/mmap_stress.rs b/tests/guest/rust-tests/src/bin/mmap_stress.rs index 01105e0e040..3a1f9fe05ac 100644 --- a/tests/guest/rust-tests/src/bin/mmap_stress.rs +++ b/tests/guest/rust-tests/src/bin/mmap_stress.rs @@ -4,7 +4,8 @@ //! pages it names held; `vm_protect` commits a reservation and changes //! protection in place, contents kept; `vm_discard` keeps the pages and reads //! back zeros; `vm_decommit` makes them a reservation again. The ranges one -//! space holds are bounded. +//! space holds are bounded, and a commitment past what memory holds is +//! refused when it is asked for. //! //! A page left unreachable must fault, which ends the process, so each such //! case runs in a child that makes the page itself and then loads from it. @@ -74,8 +75,23 @@ fn placement_and_release() { unsafe { syscall::vm_release(ptr, pages * PAGE) }.expect("release a whole range"); } - // A reservation costs no memory, so a tebibyte of one is placed. + // A reservation costs no memory, so a tebibyte of one is placed; a + // commitment is promised its frames, so a tebibyte of one is refused + // when it is asked for, whether it is reserved committed or committed + // later, and the reservation is as it was. let tib = reserve(1 << 40, VmProt::None); + assert_eq!( + unsafe { syscall::vm_reserve(1 << 40, VmProt::ReadWrite, VmAt::ANYWHERE) }, + Err(SyscallError::ResourceExhausted), + "a committed tebibyte", + ); + assert_eq!( + unsafe { syscall::vm_protect(tib, 1 << 40, VmProt::ReadWrite) }, + Err(SyscallError::ResourceExhausted), + "a tebibyte reservation committed", + ); + unsafe { syscall::vm_protect(tib, PAGE, VmProt::ReadWrite) }.expect("commit one page of the reservation"); + set(tib, 0, 0x7E); unsafe { syscall::vm_release(tib, 1 << 40) }.expect("release the tebibyte"); // Part of a range goes, and the rest stays as it was. From 16839efdfc66a4e22911ef4a382d7b808a45674d Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 23 Oct 2026 01:09:32 +0200 Subject: [PATCH 09/10] issues: what the T14 measured slower than the base at dadeebe0a osbench on the T14 at the stack's head before the /tmp snapshot, against its base 028244feb: first touch is at Linux's cost and far above the 2 MiB pages it replaced, realloc's growth pays for having no mremap, and every row that copies through user memory (tmpfs and served file reads and writes, IPC and pipes) is 15 to 40 per cent slower, which points at the user-copy path the record already names. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...paths-are-unmeasured-against-2-mib-pages.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md index 0a5a6050233..ea6fbd23f9c 100644 --- a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md +++ b/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md @@ -37,6 +37,24 @@ osbench measures faults (`mem.first_touch`, `mem.anon`), small maps (`mem.map_unmap_4k`), partial unmaps and protection changes against Linux on one machine. +On the T14 at dadeebe0a against its base 028244feb (osbench p50, one run +each), what is still slower than the base: + +| row | head | base | +|---|---|---| +| `mem.first_touch.4k` ns/page | 882 | 11 (2 MiB pages; Linux 927) | +| `mem.anon.total` ns/MiB | 233,466 | 51,522 (Linux 258,177) | +| `mem.realloc_grow.1m_256m` ms | 116 | 30 (Linux 131 ns, by `mremap`) | +| `fs.tmp.seq_read.1048576` MB/s | 6,421 | 9,055 | +| `fs.tmp.seq_write.1048576` MB/s | 4,595 | 6,080 | +| `fs.tmp.readdir.1000` ms | 1.20 | 0.74 | +| `fs.served.read_4k_at` ns | 31,185 | 23,953 | +| `ipc.reqrep.65536` ns | 48,248 | 38,214 | +| `pipe.throughput.65536` MB/s | 7,214 | 10,420 | + +The copies (`fs.tmp`, `fs.served`, `ipc`, `pipe`) point at the user-copy +path above. + Owner: the memory track. Exit: each path above measured on the T14 at a change and at its base, and against Linux; a regression is fixed or its cost accepted by the owner. Cheap remedies when measurement asks: a per-CPU frame From adeea81d0704d163571e7526ec31c0a10c6dc525 Mon Sep 17 00:00:00 2001 From: japabu Date: Fri, 23 Oct 2026 02:54:19 +0200 Subject: [PATCH 10/10] issues: the hot paths' record names what it measures The record said its paths were unmeasured; it now holds the T14's numbers, so it is named for the defect they show, slower than 2 MiB pages, and its table is the landing head 53049d55c against its base 35e930136, with map-touch-unmap beside first touch. fs.served's 4 KiB reads are no longer slower there and leave the table. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PXEk8BBMDT8tXu4AnENB9Q --- ...b-pages-still-copies-each-programs-text.md | 5 ++-- ...-hot-paths-are-slower-than-2-mib-pages.md} | 23 +++++++++---------- 2 files changed, 14 insertions(+), 14 deletions(-) rename issues/{user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md => user-memory-hot-paths-are-slower-than-2-mib-pages.md} (78%) diff --git a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md index 935c92cade0..7f357eaf083 100644 --- a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md +++ b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md @@ -73,7 +73,7 @@ live split of a user leaf. space holds at most `MAX_RANGES` ranges, refused past it, until a range's kernel memory is charged to its process. Owed: a process's floor on the T14 against the 10 to 14 MB above, and the - hot paths' costs in `issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md`. + hot paths' costs in `issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md`. 4. **Demand-zero anonymous `mmap`.** Built, over 4 KiB pages only: a fault populates one page, unmapping a range no thread touched sends no IPI, and `munmap` and a `FIXED` `mmap` take any run of pages, as POSIX does. @@ -89,7 +89,8 @@ live split of a user leaf. (`issues/nothing-charges-kernel-memory-to-a-process.md`). 5. **Shared text.** A spawn copies each page of its program's segments into the process at the page's first touch, out of the image object or out of - a snapshot of its `/tmp` file; no fault reads a file. ROOT's read-only text shared zero-copy, from + a snapshot of its `/tmp` file; no fault reads a file. ROOT's read-only + text shared zero-copy, from immutable sealed objects, is the next step. Exit: two processes of one binary hold its text once. 6. **`/apps` image sharing.** The design is the owner's decision, owed when diff --git a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md b/issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md similarity index 78% rename from issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md rename to issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md index ea6fbd23f9c..d4496cc699c 100644 --- a/issues/user-memory-hot-paths-are-unmeasured-against-2-mib-pages.md +++ b/issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md @@ -4,11 +4,11 @@ kind: defect opened: 2026-10-22 --- -# User memory's hot paths are unmeasured against 2 MiB pages +# User memory's hot paths are slower than 2 MiB pages User memory moved from 2 MiB pages filled when mapped to 4 KiB pages filled at first touch (`kernel/src/mm/space.rs`). Each of these costs more than it -did, and none is measured on the T14 against the base or against Linux: +did; the T14's numbers are below: - **Faults.** About 512 times as many per byte touched. Each takes the space's lock, an inventory lookup, two table walks (`reach` reads the leaf @@ -37,23 +37,22 @@ osbench measures faults (`mem.first_touch`, `mem.anon`), small maps (`mem.map_unmap_4k`), partial unmaps and protection changes against Linux on one machine. -On the T14 at dadeebe0a against its base 028244feb (osbench p50, one run +On the T14 at 53049d55c against its base 35e930136 (osbench p50, one run each), what is still slower than the base: | row | head | base | |---|---|---| -| `mem.first_touch.4k` ns/page | 882 | 11 (2 MiB pages; Linux 927) | -| `mem.anon.total` ns/MiB | 233,466 | 51,522 (Linux 258,177) | +| `mem.first_touch.4k` ns/page | 887 | 11 (2 MiB pages; Linux 927) | +| `mem.map_touch_unmap.4k` ns/page | 909 | 207 | +| `mem.anon.total` ns/MiB | 228,677 | 51,612 (Linux 258,177) | | `mem.realloc_grow.1m_256m` ms | 116 | 30 (Linux 131 ns, by `mremap`) | -| `fs.tmp.seq_read.1048576` MB/s | 6,421 | 9,055 | -| `fs.tmp.seq_write.1048576` MB/s | 4,595 | 6,080 | +| `fs.tmp.seq_read.1048576` MB/s | 6,060 | 9,425 | +| `fs.tmp.seq_write.1048576` MB/s | 4,297 | 5,889 | | `fs.tmp.readdir.1000` ms | 1.20 | 0.74 | -| `fs.served.read_4k_at` ns | 31,185 | 23,953 | -| `ipc.reqrep.65536` ns | 48,248 | 38,214 | -| `pipe.throughput.65536` MB/s | 7,214 | 10,420 | +| `ipc.reqrep.65536` ns | 47,887 | 38,508 | +| `pipe.throughput.65536` MB/s | 7,021 | 10,902 | -The copies (`fs.tmp`, `fs.served`, `ipc`, `pipe`) point at the user-copy -path above. +The copies (`fs.tmp`, `ipc`, `pipe`) point at the user-copy path above. Owner: the memory track. Exit: each path above measured on the T14 at a change and at its base, and against Linux; a regression is fixed or its cost