blockring: the arena holds only what is on the wire, and a flush keeps many writes there

issues/anonymous-memory-swaps-to-an-encrypted-partition.md

@@ -134,18 +134,29 @@ The symbols:
     atomically, and is refused whole if any is not free. A referenced block is
     never written.
   - **Free after durable.** `clean_end` comes only after a flush answers durable
-    for every write of the batch. The block client keeps acknowledged writes and
-    reissues them after a loss (`lib/blockring/src/client.rs`), and durable rests
-    on that.
+    for every write of the batch. The block client keeps an acknowledged
+    write's blocks, which the service hands the session shared, until a flush
+    covers it, and reissues it from them after a loss
+    (`lib/blockring/src/client.rs`); durable rests on that.
   - **Install before freeing.** Swap-in installs the frame and frees the block in
     one hold of the space lock.
   - **Re-dirtying.** A store, a kernel write copy, or a `mem_write` into a page
     being cleaned keeps it dirty, and keeps its frame.
 - **Arena reserve.** A swap session's arena has 511 blocks and its ring has 64
-  entries.
-  - Writes hold at most 447 blocks and 48 entries, leaving 64 blocks and 16
-    entries for swap-in reads.
-  - The service flushes once 223 acknowledged writes are held.
+  entries, and both carry only requests on the wire: an acknowledged write
+  holds neither.
+  - Writes on the wire hold at most 447 blocks and 48 entries, leaving 64
+    blocks and 16 entries for swap-in reads.
+  - What the session keeps shared is bounded by the batch: `clean_end` waits
+    for a flush covering it. Past `diskserver::session::KEPT_BLOCKS` the
+    session flushes for itself before the next write. The service's batch size
+    is its own choice and needs no arena arithmetic.
+  - **A swap-in read can wait behind a write.** Requests go out in the order
+    asked, so a read asked after a write waits while that write has no
+    contiguous arena room, or waits at the session's kept limit for its flush.
+    The 64-block reserve bounds how many reads fit, not that wait. If swap-in
+    latency matters, reads go out ahead of waiting writes, which the client's
+    one outbox does not do today.
 - **`CLEANING` on private leaves.**
   - A store faults and clears it. So does a kernel write copy, which takes its
     pin under the space lock. A read copy leaves it.

issues/the-file-servers-clean-blocks-are-given-back-to-no-other-job.md

@@ -23,13 +23,15 @@ interim, not a sizing.
 
 A clean block the allocator refuses, or whose place in the eviction order it
 refuses, is not kept: a read is still answered, and a flushed block is already
-on the disk. Four allocations still have no refusal path, so a server whose job is out of memory can still
-abort at them:
+on the disk. These allocations still have no refusal path, so a server whose
+job is out of memory can still abort at them:
 - the block map's node, a small one every few inserts of a clean or a dirty
   block;
+- a block's share count, a small one per block kept;
 - a dirty block;
 - a fetch's staging buffer;
-- a flush's list of the dirty blocks and its buffer for one run.
+- a flush's list of runs, the writes it keeps in flight, and each run's list
+  of shares.
 
 **Exit**: the file servers drop clean blocks when the memory service reports
 pressure, and `CLEAN_SHARE` is deleted.

lib/blockdev/src/lib.rs

@@ -33,6 +33,13 @@ pub const BLOCK: usize = 4096;
 /// One block.
 pub type Block = [u8; BLOCK];
 
+/// A block's bytes shared between whoever wrote them and a device that may
+/// write them again: a served disk keeps a write's until a flush covers it.
+/// The block is boxed on its own so the file server's cache can allocate a
+/// block it reads in fallibly (`system/fileserver/src/cache.rs`, `boxed`),
+/// which `Rc::new` of the block itself cannot.
+pub type SharedBlock = Rc<alloc::boxed::Box<Block>>;
+
 /// Why a device did not do what it was asked.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
 pub enum Error {